Skip to content
View as markdown

Evals in Docker

One eval, one container. dimos evals run --docker ... starts a fresh worker container from the eval image, runs the identical command inside it, and returns at once. The container removes itself when the eval ends. Run the command again for another suite and you have two evals running side by side; each container has its own network namespace, so every dimos inside uses its default ports and multicast groups without touching its neighbours.

dimos evals run --docker dimos.evals.suites.dimsim_apartment_qa \
    --agent dimos.evals.agents.pi --set max_steps=12 --limit 4
# evals-20260918-133000-3f1a-dimsim_apartment_qa: started (detached)
#   follow:   docker logs -f evals-20260918-133000-3f1a-dimsim_apartment_qa
#   results:  docker/evals/eval-runs/dimos/evals/   recordings + rerun.rrd: docker/evals/eval-runs/dimos/recordings/

docker ps lists the evals still running; docker logs -f <name> follows one; docker stop <name> abandons one, keeping what it wrote so far but no results.jsonl/summary.json. The container runs the stock EvalRunner, which boots a dimos plus DimSim per case, runs the agent, records, and tears them down again.

Setup

Build the image once from the repo root, with the DimSim assets fetched (the image bakes them, and the sim refuses to load LFS pointer stubs), and export the API keys the evals use (OPENAI_API_KEY, ANTHROPIC_API_KEY, TYPESAFENAVIGATIONAGENT__API_KEY):

git lfs pull --include="misc/DimSim/**"
docker build -f docker/evals/Dockerfile -t dimensional/evals .

The image is a snapshot of the checkout, so rebuild after code changes; the dependency layer is cached unless the lock file moved, so that is minutes, not the first build's half hour. On a shared host, tag one image per branch so a rebuild never replaces the image someone else's eval is running on:

docker build -f docker/evals/Dockerfile -t dimensional/evals:my-branch .
export EVALS_IMAGE=dimensional/evals:my-branch

What a run leaves behind

Everything a container writes lands on the host under EVAL_RUNS_DIR (default docker/evals/eval-runs), which the compose file mounts on /state:

docker/evals/eval-runs/dimos/evals/run-<stamp>-<id>/   the EvalRunner run dir: manifest, results.jsonl, summary.json, per-case trajectory
docker/evals/eval-runs/dimos/recordings/<run-id>/      one per dimos boot, i.e. one per case:
    memory.db                                          the --record sensor recording the grades read
    rerun.rrd                                          the whole Rerun stream of that case (see below)

Each case's result names its recording folder, so its memory.db and rerun.rrd are one lookup away. Open a case's visualization with rerun docker/evals/eval-runs/dimos/recordings/<run-id>/rerun.rrd.

Rerun recording

Workers run with VIEWER=rerun RERUNBRIDGEMODULE__RERUN_OPEN=none RERUNBRIDGEMODULE__RERUN_SAVE=1: the Rerun bridge logs as usual but never opens a viewer, and a headless rerun --save client of the bridge's own gRPC server streams every event to rerun.rrd next to that run's memory.db. The same --rerunbridgemodule.rerun-save flag works on a bare dimos run, with a live viewer too if you want one. Files grow at roughly 20 MB per minute of a DimSim case with camera and lidar on, so budget disk accordingly and sync the runs directory to S3 after a batch. RERUNBRIDGEMODULE__RERUN_SAVE=0 in front of a --docker run skips the file for that run.

Habitat

The image also carries Habitat: habitat-sim's own python 3.9 conda environment under target/habitat, the annotated HM3D example house, and the habitat-native wrapper, all built by the module's installer at image build time so a run starts in seconds. Two things differ from DimSim runs:

  • Habitat renders headless through EGL from the host's NVIDIA driver, so it needs the GPU overlay from the EC2 runbook below. It cannot run on a CPU-only host.
  • Its native process speaks zenoh, while DimSim needs LCM. Export DIMOS_TRANSPORT=zenoh for Habitat runs:
DIMOS_TRANSPORT=zenoh dimos evals run --docker dimos.evals.suites.habitat_smoke --agent dimos.evals.agents.pi

Only the HM3D example house is in the image. HSSD, ReplicaCAD and licensed HM3D splits go on a host directory that docker/evals/compose.habitat-data.yaml mounts over the container's target/habitat/data, so suites find them at their default paths. Add it to COMPOSE_FILE next to the GPU overlay and set HABITAT_DATA_DIR; step 8 of the runbook fills it.

EC2 runbook

Verified on a g6.8xlarge (32 vCPU, 128 GB, one L4) with plain Ubuntu 26.04. A Deep Learning AMI already has the driver, Docker and the container toolkit; skip what you have.

  1. Driver, Docker with its compose and buildx plugins (Ubuntu's docker.io ships neither), git-lfs; then reboot for the driver:
sudo apt-get update && sudo apt-get install -y nvidia-driver-580-server docker.io docker-compose-v2 docker-buildx git git-lfs && sudo usermod -aG docker "$USER" && sudo reboot
  1. NVIDIA container toolkit, and BuildKit as Docker's default builder (the image uses cache mounts):
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list && sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
echo '{"features":{"buildkit":true},"runtimes":{"nvidia":{"path":"nvidia-container-runtime","runtimeArgs":[]}}}' | sudo tee /etc/docker/daemon.json && sudo systemctl restart docker
docker run --rm --gpus all ubuntu:24.04 nvidia-smi
  1. A headless Xorg on the GPU. DimSim's Chromium gets hardware WebGL on Linux only through an X server on the card (the self-hosted CI runner does the same; a virtual framebuffer is not enough). The datacenter GPU presents a virtual display of its own, so do not pass --use-display-device=None:
sudo apt-get install -y xserver-xorg xinit x11-xserver-utils
sudo nvidia-xconfig --allow-empty-initial-configuration --virtual=1280x720 --busid="PCI:$(nvidia-smi --query-gpu=pci.bus_id --format=csv,noheader | awk -F'[:.]' '{printf "%d:%d:%d", strtonum("0x"$2), strtonum("0x"$3), strtonum("0x"$4)}')"
sudo tee /etc/systemd/system/xorg.service >/dev/null <<'UNIT'
[Unit]
Description=Headless Xorg on the GPU
After=multi-user.target
[Service]
ExecStart=/usr/bin/Xorg :0 -noreset
Restart=always
[Install]
WantedBy=multi-user.target
UNIT
sudo systemctl enable --now xorg && sleep 3 && sudo DISPLAY=:0 xhost +local:root

xhost printing "non-network local connections being added" means Xorg is up; sudo journalctl -u xorg -n 30 otherwise.

Only do this on a dedicated eval host. xhost +local:root lets every container (they all run as root) connect to that display, and X11 does not isolate its clients: any of them can read the screen and keystrokes and inject input. Workers seeing each other's Chromium is harmless; pointing docker/evals/compose.gpu.yaml at your own desktop session (a laptop's DISPLAY) hands it to whatever code runs in the eval.

  1. The socket buffers dimos's LCM setup asks for; host-wide, so they cannot be set per container:
echo -e "net.core.rmem_max=67108864\nnet.core.rmem_default=67108864" | sudo tee /etc/sysctl.d/90-dimos.conf && sudo sysctl --system
  1. The repo, its DimSim assets, and the host venv the --docker flag runs from (a few packages build from source, hence the compilers):
sudo apt-get install -y build-essential pkg-config portaudio19-dev libturbojpeg0-dev libgl1
curl -LsSf https://astral.sh/uv/install.sh | sh && source ~/.bashrc
git clone -b <branch> https://github.com/dimensionalOS/dimos.git ~/dimos && cd ~/dimos && git lfs pull --include="misc/DimSim/**" && uv sync
  1. The image, 30 to 40 minutes the first time:
cd ~/dimos && docker build -f docker/evals/Dockerfile -t dimensional/evals .
  1. Environment, in the shell profile so it survives reconnects. Habitat runs need DIMOS_TRANSPORT=zenoh; DimSim runs need it unset:
export OPENAI_API_KEY=... TYPESAFENAVIGATIONAGENT__API_KEY=... DISPLAY=:0 HABITAT_DATA_DIR=/data/habitat
export COMPOSE_FILE=docker/evals/compose.yaml:docker/evals/compose.gpu.yaml:docker/evals/compose.habitat-data.yaml
  1. Datasets into HABITAT_DATA_DIR (the overlay replaces the container's whole data folder, so the HM3D example goes there too):
sudo mkdir -p /data/habitat && sudo chown "$USER" /data/habitat
cd ~/dimos && docker compose run --rm worker /app/target/habitat/env/bin/python -m habitat_sim.utils.datasets_download --uids hm3d_example --data-path /app/target/habitat/data --no-replace
uv tool install huggingface_hub && huggingface-cli login   # after accepting the terms on the hssd/hssd-hab dataset page
huggingface-cli download hssd/hssd-hab --repo-type dataset --local-dir /data/habitat/hssd-hab
  1. Check a container sees everything, then run:
cd ~/dimos && docker compose run --rm worker bash -c 'nvidia-smi -L; echo DISPLAY=$DISPLAY; ls /tmp/.X11-unix'
uv run dimos evals run --docker <suite> --agent <agent> ...

Sizing: with hardware rendering plan on 3 to 4 vCPU, 4 to 5 GB of RAM and a slice of the one GPU per concurrent eval; start a g6.8xlarge at six and watch per-case duration, score and nvidia-smi as you add more. The sim runs on wall clock, so a starved GPU or CPU shows up as lower scores, not just slower runs.

Many at once

There is no scheduler: to run a suite N ways, start N --docker invocations that select disjoint parts of it with --tags, and let each container work through its part in order. A shell loop with xargs -P N over a list of such commands is all the parallelism there is; docker ps shows what is running and EVAL_RUNS_DIR collects every run. A benchmark that spans scenes and agent configurations maps one container to each (configuration, scene) pair.

Notes

  • Everything in a container runs as root; the entrypoint sets umask 0000 so whatever a run writes, even one stopped midway, is readable and deletable by the host user. It is so for every other local account too, prompts and trajectories included: on a shared multi-user machine, keep EVAL_RUNS_DIR somewhere only you can reach.
  • One Xorg display per host means one GPU per host. Multi-GPU instances need one X screen per GPU and a per-container DISPLAY, which the compose files do not do yet.
  • Every concurrent eval calls the model API, so the account's rate limit caps useful parallelism as much as hardware does.