Visual Question Answering¶

The VQA tools generate deterministic questions from recorded images and evaluate them through the shared dimOS evaluation runner.

Generate a Dataset¶

Generate from one image in a recording containing color_image, camera_info, tf, and either pointlio_lidar or lidar:

dimos evals vqa generate /path/to/calibrated-recording.db --image-index 100

Generate from a range. stop is exclusive:

dimos evals vqa generate /path/to/calibrated-recording.db --start 0 --stop 100 --stride 10

Generated datasets default to:

~/.local/state/dimos/datasets/vqa/<recording-stem>-frames

Use --output <directory> to override that location. The destination must be empty. Generation requires non-empty color_image, camera_info, tf, and either pointlio_lidar or lidar streams; incomplete datasets are rejected at startup. For a valid dataset, point-cloud evidence is available when image and LiDAR observations are within 0.1 seconds. Individual frames with unmatched LiDAR observations or unresolvable TF retain only image-based question families.

Point-cloud frame preparation¶

Point-cloud question families use an explicit image-aligned frame boundary. After validating the required dataset streams, the loader in dimos/evals/vqa/pointcloud_frame.py reads the selected image and nearest point cloud, rectifies the image and camera intrinsics, resolves the point-cloud-to-camera transform from recorded TF, and packages the synchronized observations as a PointCloudFrame.

The range primitive in dimos/evals/vqa/primitives/range.py consumes that prepared frame. It projects valid points into the image, keeps the nearest camera-depth point per pixel, selects points inside each object mask, and derives robust range statistics. Keeping dataset access and calibration preparation outside the primitive lets range estimation operate only on explicit geometry.

Question Families¶

The image-only question author selects object names and applicable families. It does not produce answers. Every proposal uses object_names; single-object families require one entry, while closest_object requires two to five.

Family	Choices	Evidence rule
`presence`	yes, no	At least one Moondream detection
`horizontal_direction`	left, center, right	Exactly one detection; use its horizontal center
`object_count`	one, two, three, four or more	Count Moondream detections
`image_coverage`	Adaptive percentage buckets	Divide EdgeTAM mask pixels by image pixels
`largest_visible_area`	Two to five authored object references	Select a mask at least 20% larger than the runner-up
`object_distance`	under 1 m, 1 to under 2 m, 2 to under 3 m, 3 m or more	Median LiDAR range inside an EdgeTAM mask
`closest_object`	Two to five authored object references	Select the smallest unambiguous LiDAR range

Proposals without sufficient evidence are rejected and retained in the private audit. If no proposal can be answered, generation fails without publishing a dataset.

Distance questions require one Moondream detection, one EdgeTAM mask, and at least five projected LiDAR points. Evidence whose range quartiles cross an answer boundary is rejected. Closest-object questions accept references such as left person, right person, and chair, which lets Moondream distinguish repeated categories. They are rejected unless every reference has exactly one detection and the closest range interval does not overlap any other candidate.

Image-coverage questions choose between two bucket schemes so the measured mask coverage is as far as possible from the nearest answer boundary. Largest-visible-area questions are rejected unless the winning EdgeTAM mask contains at least 20% more pixels than the runner-up. Both families work without point-cloud data.

Dataset Layout¶

dataset/
  cases.jsonl
  labels.jsonl
  assets/
    frame-000100.png
  audit/
    run.json
    frame-000100/
      frame.json
      ground_truth.json
      cases.json
      labels.json

cases.jsonl and assets/ are public evaluation inputs. labels.jsonl and audit/ contain private answers, evidence, rejected proposals, source indices, and timestamps.

Run Evaluation¶

dimos evals vqa run ~/.local/state/dimos/datasets/vqa/calibrated-recording-frames \
  --model gpt-4o-mini

Evaluation exposes only each public image, question, and fixed choices to the model. The shared runner writes results under ~/.local/state/dimos/evals/run-*/.