Custom Models for Auto Labeling

Loading custom models will enable you to use your own models for auto labeling. This is useful if you have a custom model that you have trained on your own data and want to use it for auto labeling. You can also create a labeling - training loop for improving your model incrementally.

Presiquites

AnyLabeling ≥ 0.2.22.

Steps

1. Prepare your model(s)

Convert your model(s) into corresponding format(s) supported by AnyLabeling. The supported format for YOLOv8 and all Segment Anything variants (SAM, SAM2, SAM2.1, SAM3) is ONNX.

Convert YOLOv5 to ONNX: https://docs.ultralytics.com/yolov5/tutorials/model_export/#formats (opens in a new tab). Note: The sample models are converted with YOLOv5 v6.2. Other versions may not work.
Convert YOLOv8 to ONNX: https://github.com/ultralytics/ultralytics (opens in a new tab).
SegmentAnything: Please download ONNX files from https://github.com/vietanhdev/anylabeling-assets/releases/tag/v0.4.0 (opens in a new tab).

Recommendation: Put all the model files in a folder.

2. Create a model config file

Create a model config file config.yaml in YAML format in the same folder with models. The config file needs to follow the formats below:

Segment Anything (SAM / SAM2 / SAM2.1):

type: segment_anything
name: segment_anything_vit_b_quant-r20230416
display_name: Segment Anything (ViT-B Quant)
decoder_model_path: segment_anything_vit_b_decoder_quant.onnx
encoder_model_path: segment_anything_vit_b_encoder_quant.onnx
input_size: 1024
max_height: 682
max_width: 1024

Segment Anything 3 (SAM 3) — text-prompt support:

SAM 3 requires an additional language_encoder_path field pointing to the CLIP-based language encoder ONNX model. Use input_size: 1008 (SAM 3 uses a different resolution than SAM/SAM2).

type: segment_anything
name: sam3_vit_h_custom
display_name: Segment Anything 3 (ViT-H)
encoder_model_path: sam3_image_encoder.onnx
decoder_model_path: sam3_decoder.onnx
language_encoder_path: sam3_language_encoder.onnx
input_size: 1008
max_height: 1008
max_width: 1008

Note: SAM 3 ONNX models can be exported using samexporter (opens in a new tab) or downloaded from Hugging Face (opens in a new tab). When a language_encoder_path is present, the UI shows a text prompt field for open-vocabulary detection.

YOLOv5:

type: yolov5
name: yolov5l-r20230415
display_name: YOLOv5l Ultralytics
model_path: yolov5l.onnx
confidence_threshold: 0.45
input_height: 640
input_width: 640
nms_threshold: 0.45
score_threshold: 0.5
classes:
- person
- bicycle
- car
- motorcycle
- airplane
- bus
- train
- truck
- boat
- traffic light
- fire hydrant
- stop sign
- parking meter
- bench
- bird
- cat
- dog
- horse
- sheep
- cow
- elephant
- bear
- zebra
- giraffe
- backpack
- umbrella
- handbag
- tie
- suitcase
- frisbee
- skis
- snowboard
- sports ball
- kite
- baseball bat
- baseball glove
- skateboard
- surfboard
- tennis racket
- bottle
- wine glass
- cup
- fork
- knife
- spoon
- bowl
- banana
- apple
- sandwich
- orange
- broccoli
- carrot
- hot dog
- pizza
- donut
- cake
- chair
- couch
- potted plant
- bed
- dining table
- toilet
- tv
- laptop
- mouse
- remote
- keyboard
- cell phone
- microwave
- oven
- toaster
- sink
- refrigerator
- book
- clock
- vase
- scissors
- teddy bear
- hair drier
- toothbrush

YOLOv8:

type: yolov8
name: yolov8m-r20230415
display_name: YOLOv8m Ultralytics
model_path: yolov8m.onnx
confidence_threshold: 0.45
input_height: 640
input_width: 640
nms_threshold: 0.45
score_threshold: 0.5
classes:
- person
- bicycle
- car
- motorcycle
- airplane
- bus
- train
- truck
- boat
- traffic light
- fire hydrant
- stop sign
- parking meter
- bench
- bird
- cat
- dog
- horse
- sheep
- cow
- elephant
- bear
- zebra
- giraffe
- backpack
- umbrella
- handbag
- tie
- suitcase
- frisbee
- skis
- snowboard
- sports ball
- kite
- baseball bat
- baseball glove
- skateboard
- surfboard
- tennis racket
- bottle
- wine glass
- cup
- fork
- knife
- spoon
- bowl
- banana
- apple
- sandwich
- orange
- broccoli
- carrot
- hot dog
- pizza
- donut
- cake
- chair
- couch
- potted plant
- bed
- dining table
- toilet
- tv
- laptop
- mouse
- remote
- keyboard
- cell phone
- microwave
- oven
- toaster
- sink
- refrigerator
- book
- clock
- vase
- scissors
- teddy bear
- hair drier
- toothbrush

3. Select the model from AnyLabeling UI

In Auto Labeling mode, select ...Load Custom Model from the Auto dropdown. Then select the model config file (config.yaml) file.

Custom Models

If the model is loaded successfully, you will see the model name in the Auto dropdown.

Custom Models

Note: Please download sample models and configuration files here (opens in a new tab). After downloading, extract the zip file and load the models with above UI.

GPU Acceleration Text OCR Labeling