Object detection

Object detection trains models to identify and locate objects in images using bounding boxes. This problem type uses DETR-style (Detection Transformer) models. See Known limitations for what isn't supported yet.

Dataset format

Your dataset must include:

• Image Column: A column containing image data (base64-encoded or binary)
• Output Column: A column containing the object annotations for each image, in COCO format

Each annotation must be a dictionary (or JSON string) with the following fields:

{
  "bbox": [[x, y, width, height], [x, y, width, height], ...],
  "category": [0, 1, 2, ...]
}

• bbox: Bounding boxes in COCO format, [x, y, width, height], where x and y are the coordinates of the top-left corner.
• category: The class label for each bounding box, as an integer or a string. Labels are mapped to internal class IDs automatically.

An optional area field is also accepted.

Example annotation for an image with two objects:

{
  "bbox": [[100, 150, 200, 300], [400, 200, 150, 250]],
  "category": [0, 2]
}

The two bbox entries from the example above, drawn to scale: [100, 150, 200, 300] (category 0) and [400, 200, 150, 250] (category 2).

Supported models

Object detection uses DETR-style models from Hugging Face:

• facebook/detr-resnet-50 (default)
• facebook/detr-resnet-101
• SenseTime/deformable-detr
• hustvl/yolos-tiny and hustvl/yolos-small
• Other models compatible with AutoModelForObjectDetection

Training modes

Object detection supports all experiment training modes:

• Full Model Training
• LoRA
• QLoRA

Metrics

The following metrics are available for object detection and can be selected in the Training configuration section:

• mAP: Mean Average Precision, averaged across all IoU thresholds
• mAP@50: Mean Average Precision at an IoU threshold of 0.50
• mAP@75: Mean Average Precision at an IoU threshold of 0.75 (stricter)
• mAR: Mean Average Recall, averaged across all IoU thresholds

IoU (Intersection over Union) measures how much a predicted box overlaps the true box.

Known limitations

• Deployment: You can't deploy object detection models yet, because the vLLM inference engine used for deployments doesn't support them. The Deploy button is still shown, but the deployment fails. To get predictions, use Predict or download the Validation Results.
• Model architecture: Only DETR-style transformer models (DETR, Deformable DETR, and YOLOS) are supported. CNN-based detectors such as YOLOv5, YOLOv8, and Faster R-CNN aren't supported.

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O Cascade to cloud-feedback@h2o.ai