xArm Grasping¶

Two blueprints, differing only in which grasp provider they compose. Both carry the control coordinator, the wrist camera, scene registration and pick-and-place; both run on the real arm by default and switch to the MuJoCo room scene with --simulation:

dimos run xarm-grasp-graspgenx --controlcoordinator.addresses '{"arm": "192.168.1.x"}'  # hardware
dimos run xarm-grasp-graspgenx --simulation mujoco                                       # the room scene

On hardware the arm needs its address, from the flag above or a saved config file (see Hardware configuration). With --simulation a saved address is ignored and everything runs in MuJoCo. xarm-grasp-agent and xarm-grasp-graspgenx-agent add an MCP agent over the top; drive those with dimos agent-send "...".

GraspGenX requires Linux x86_64, a CUDA 12.8-compatible GPU, and uv >=0.9.25. The first launch prepares its isolated Python 3.12 environment and downloads the checkpoints. Runtime sources come from the development checkout or the shared repository clone used by installed dimOS.

To show the MuJoCo window with Rerun disabled:

MUJOCO_GL=glfw dimos --viewer none run xarm-grasp-graspgenx \
  --simulation mujoco --headless false

In another terminal, use dimos shell and follow Driving it to scan objects and request grasps. See the isolated-runtime development guide for test and type-check commands.

What differs between the arm and the sim is decided at import time: the hardware adapter, the base pose, the camera (RealSense plus its mount edge, versus the MuJoCo wrist camera), the detector backends, and the home pose.

The manipulation viewer is on viser at http://127.0.0.1:8095. To watch the MuJoCo scene itself, add --headless false with MUJOCO_GL=glfw. On a host where /dev/dri must be hidden from Mesa, run inside the team's existing /dev/dri-masked mount namespace:

MUJOCO_GL=egl LIBGL_ALWAYS_SOFTWARE=true MESA_LOADER_DRIVER_OVERRIDE=llvmpipe \
  dimos --viewer none run xarm-grasp --simulation mujoco

Voxel map obstacles¶

The wrist camera feeds a live voxel map that the planner treats as one octree obstacle, so trajectories avoid whatever has actually been seen rather than only the registered objects:

camera pointcloud
  -> RoboPlanPointCloudSelfFilter   drops robot surface returns using RoboPlan
  -> RayTracingVoxelMap             accumulates occupied cells in the world frame
  -> PointCloudBoxFilter            leaves out the cells inside named boxes
  -> ManipulationModule.voxel_map   rebuilt as the "mapping/voxel-map" obstacle

The blueprint adds this chain with manipulation_mapping() from dimos/robot/manipulators/common/blueprints.py. XARM_GRASP_VOXEL_SIZE is shared by the mapper and planner octree so their cells line up. The blueprint enables the camera's pointcloud output, which is off by default on both the RealSense and MuJoCo camera. The self filter drops a whole cloud if its full joint state or sensor/base TF is missing at capture time.

Because the target object is itself mapped geometry, a collision-checked plan into it can only ever be rejected. The pregrasp-to-grasp leg and the retreat are therefore straight-line move_linear servos with collision checking off; only the approach to the pregrasp pose is a checked plan.

For that checked approach, and for every plan made while the object is held, PickAndPlaceModule asks PointCloudBoxFilter.set_box to leave out the cells in a box around the target's point cloud, grown by target_clearance (3 cm on each side, which also covers the table cells under it). Once the jaws have closed on the object, the same box is set again on the tool tip's TF frame, so it moves with the gripper: the wrist camera keeps mapping whatever the gripper holds. Each change re-sends the last map at once. The box is cleared when the pick ends without holding anything, or once the arm has backed away after a release. Neighbouring objects stay obstacles: a pregrasp that would sweep the gripper into the object next to the target is still refused, with IK failed: COLLISION as the reason.

Seeing the proposals¶

The viser scene draws the ranked proposals as pose glyphs: an approach axis with the closing axis across it, coloured best-green through worst-orange so the ordering reads at a glance, with the top three drawn thicker and labelled with their score. Only the leading twenty are drawn, because a hundred glyphs bury the ranking they exist to show. manipulation.grasp-proposals in the Scene panel toggles them.

The markers are pose indicators, not a gripper: what they promise is where a grasp points and in what order the generator ranked it. To see what the arm will actually do with one, watch the plan preview.

The scene¶

The scene is an enclosed 2.6 m by 3.0 m room. The xArm is bolted to the world origin. Unlike data/xarm7, this scene has no 12 cm pedestal, so the planning model overrides base_pose to match. The 38 cm by 60 cm desk is in front of the arm, with its work surface at z=0.13 m. Six scaled household targets sit on it:

The canonical positions and geometry live in data/xarm_grasp_sim/scene.xml. The visual meshes were cooked from the dimos_office scene package and scaled for the xArm gripper; primitive geometry is used only for contact. Every target has a grasp axis below 7 cm, and the gray can is the designated pick smoke target. The blueprint starts the arm at an elevated, collision-free top-down scan pose so all six visual meshes fit in the wrist-camera frame.

OWL-ViT labels these synthetic renders unreliably. A scan routinely returns the right six positions under swapped names, so match objects by position, not label.

Driving it¶

In a second terminal, connect to the running blueprint:

dimos shell

Then run this complete scan and obstacle-inspection sequence:

from dimos.robot.manipulators.xarm.blueprints.grasp import XARM_GRASP_PROMPTS

app.ManipulationSkills.go_init()
scan = app.PickAndPlaceModule.scan_objects(XARM_GRASP_PROMPTS)
print(scan)

print(app.ObjectSceneRegistrationModule.get_detected_objects())
print(app.ManipulationModule.refresh_obstacles())
print(app.ManipulationModule.get_obstacles())

CPU OWL-ViT inference takes about 11 seconds per prompt/frame on the validation host, so let a scan finish rather than issuing another concurrently.

To pick, hand pick_object an object_id from that scan. Choose the target by where its point cloud actually is rather than by name:

scene = app.ObjectSceneRegistrationModule

for obj in scan.metadata["objects"]:
    cloud = scene.get_object_pointcloud_by_object_id(obj["object_id"])
    print(obj, cloud.points_f32().mean(axis=0) if cloud else None)

# the bottle sits at roughly (0.58, 0.19); pick whichever id landed there
pick = app.PickAndPlaceModule.pick_object("<object_id>")
print(pick)

app.PickAndPlaceModule.place_at(0.45, -0.25, 0.25)

pick_object generates the grasps itself, so there is no separate grasp call. It opens the gripper, plans to the pregrasp, servos in, closes, verifies, and retreats; with a learned provider it walks the ranked candidates until one is reachable, and the result metadata carries the winning rank, its score and the candidate count. To inspect grasps without moving the arm, call propose_grasps on the provider directly:

cloud = scene.get_object_pointcloud_by_object_id("<object_id>")
candidates = app.GraspGenXModule.propose_grasps(cloud)   # HeuristicGraspModule in the base blueprint
print(len(candidates.candidates), [c.score for c in candidates.candidates[:5]])

The prompt set includes a green ring fallback because the tape loses its category silhouette in the wrist camera's top-down view.

A failed grasp knocks free-body targets out of place, and MujocoSimModule.reset() does not respawn them. Restart the blueprint between pick attempts that need a pristine scene.

The generic PointCloudSelfFilter handles streams and point fields; subclasses supply a capture-time keep mask. The xArm blueprint selects RoboPlanPointCloudSelfFilter, which delegates geometry loading, kinematics, mimic joints and Narrowphase surface classification to RoboPlan 0.7. dimOS matches joint state and TF to the capture timestamp, preserves point fields, and drops unaligned captures. The xArm blueprint filters only the arm using a model without the gripper. Gripper points may enter the map; separate gripper control and grasp evaluation do not remove them. Planning and control retain their existing models.

Filtering removes robot surface returns before voxel fusion. There is no solid volume sampling or historical robot clear mask; clearing is the mapper's ordinary ray tracing. This does not promise to erase every previously accumulated robot voxel, especially where later rays do not reach.