SDGE — Simulator Dataset Generation Engine

A backend-agnostic pipeline for generating layered ground-truth robot manipulation data by replaying/scripting trajectories in simulation. Unlike existing simulators/datasets (see docs/robot-sim-survey.md for the full survey), SDGE exports, per frame:

• composite / robot-only / object-only / background-only RGB video (true multi-pass layered rendering)
• robot & object visible masks and amodal masks (unoccluded silhouette)
• per-entity depth (robot depth, object depth — not just scene depth)
• 6-DoF pose per robot link and per object
• contact state / force between entities

See docs/robot-sim-survey.md and the RoboTwin/ManiSkill3 code-level deep-dive (§5) for why no existing platform ships this combination, and for exactly which SAPIEN/ManiSkill3 APIs this project reuses.

Both backends are real, working implementations, verified end-to-end on this machine (RTX 5070 Laptop GPU), including real YCB objects (not placeholders) on both:

• SAPIEN (sdge/backends/sapien_backend/) — CPU sim, runs from .venv. Six tasks registered: pick_ycb (hand-tuned baseline), pick_ycb_planned / pick_cube / stack_cube / pick_ycb_random (mplib-planned — see below), pick_ycb_multicam (3-camera demo — see below). All six verified to actually pick up / stack their objects (not just "runs without crashing") across multiple seeds.

• Isaac Lab (sdge/backends/isaac_backend/) — Isaac Sim 6.0.1.0 + Isaac Lab, runs from a separate .venv-isaac (kept isolated to avoid dependency conflicts — different torch/gymnasium pins than the SAPIEN stack). Every API call in isaac_backend/backend.py and assets.py was verified interactively before being committed, the same way the SAPIEN backend was built; the module docstrings document the real, non-obvious gotchas that were found and fixed along the way (segmentation ids are packed RGBA colors, not sequential integers; RGB frames need an explicit Kit app.update() pump per render pass or every layer comes out byte-identical; semantic tags are opt-in per prim; a converted mesh needs its physics schemas baked into the USD file itself — layering rigid_props/collision_props on at spawn time reliably failed). One task registered: pick_ycb. Current status: contact/contacts.parquet is no longer empty (real contact events, forces up to ~8N) — the actual root cause was IsaacScene.reset() having no settle loop before planning against the object's position, not just untuned waypoints (see pick_ycb_isaac.py's docstring). It does not yet reliably lift the object; a cuRobo-based fix was attempted and is a documented dead end for now (pick_ycb_isaac_curobo.py — cuRobo's bundled Franka kinematic model measurably disagrees with Isaac Lab's actual spawned robot; proven not to be a fixable-by-calibration frame offset, see that file's docstring).

• MuJoCo (sdge/backends/mujoco_backend/) — raw mujoco package (not dm_control), runs from the same .venv as SAPIEN (no dependency conflicts found). Robot is the official google-deepmind/mujoco_menagerie Franka Panda MJCF (not the mani-skill URDF the other two backends use — MJCF is what the MuJoCo ecosystem itself standardizes on), fetched once via sparse git checkout into ~/.cache/sdge/mujoco_menagerie/. Four tasks registered: pick_cube (a native MJCF box — deliberately the simplest object, to validate the render/mask/depth/pose/contact pipeline first), pick_place_can (a real YCB mesh + a full pick-and-place motion — see its own section below), dual_pick_place_kitchen (two independently-planned Pandas + real RoboTwin-OD meshes — see its own section), and tomato_bowl_handoff (two dependently-planned Pandas handing an object to each other via a bowl — see its own section for why this one is only partially working). Layered rendering reuses the exact MjvOption.geomgroup + enable_segmentation_rendering() mechanism found working in LV-Robotics-Lab/image-layered-world-model's MuJoCo GT-layering code during an earlier session (see mujoco_backend/backend.py's docstring for the real difference: visibility toggles at group granularity, not per-individual-geom). Motion planning is a self-contained damped-least-squares Jacobian IK (mujoco_backend/motion_planning.py) rather than mplib — reusing mplib here would mean pulling SAPIEN-specific machinery into the MuJoCo path for no benefit, since none of these tasks need collision-aware planning. Two real bugs were found and fixed along the way, both verified interactively (not assumed): solving IK independently for far-apart Cartesian waypoints and then linearly interpolating the joint angles between solutions let the arm swing through a completely different elbow/wrist configuration than intended (measured: ~0.67 rad of unintended elbow/wrist motion for a straight 20cm vertical lift) — fixed by re-solving IK at small warm-started Cartesian steps instead; and plan()'s IK solver was using the live physics data object as scratch and leaving the arm's qpos at wherever the last (lift) solve landed instead of the settled ready pose, silently starting every episode from the wrong configuration — fixed by restoring qpos before the first real step.

  pick_cube's grasp reliability is a known-open issue (see mujoco_backend/assets._stiffen_gripper's docstring): the stock Menagerie gripper actuator holds a grasp rock-solid while the arm is stationary but let go right as the lift motion began in about half of tested spawn positions, even with a very slow, quasi-static lift; a 20x stiffer gripper actuator gain raised that to ~75% (6/8 episodes) but didn't fully fix it. The actual root cause was found and fixed while building pick_place_can (see that task's own section) — the 100Hz default timestep was too coarse for the 20x-stiffened actuator, causing real numerical chatter (finger joint velocities oscillating ~0.05-0.5 rad/s and never decaying, confirmed via data.qvel, not assumed) rather than holding still; a 400Hz timestep eliminated it. pick_cube itself hasn't been switched over to confirm this closes its remaining ~25% gap too — likely does, not yet verified, left as a follow-up rather than assumed.

sdge/tasks/pick_place_can_mujoco.py — Franka Panda mounted directly on a kitchen counter (not floor-mounted beside it: the counter's footprint needs to span both a pick spot and a place spot ~30cm apart, and a floor-level counter box that size would interpenetrate the robot's own base geometry), a real YCB 010_potted_meat_can, full pick → carry → place → release → retreat motion via motion_planning.build_pick_place_trajectory.

• Countertop: flat white (rgba=[0.95, 0.95, 0.95, 1]) — originally a real CC0 wood photo-scan (kitchen_wood), switched to white on request; the texture fetcher was removed since nothing uses it anymore.
• Real mesh import, new for MuJoCo here: mujoco_backend/assets.load_ycb_object() reads the exact same downloaded YCB cache SAPIEN's load_ycb_object uses — the textured .obj loads directly, but YCB's collision.ply has no MuJoCo decoder (MjSpec.add_mesh only recognizes OBJ/STL/MSH; verified interactively, not assumed) — converted once via trimesh (already a transitive dependency) and cached per-id as .stl under ~/.cache/sdge/mujoco_ycb_stl/.
• Object choice wasn't the obvious round can: 005_tomato_soup_can (6.9cm diameter) leaves only ~5.5mm clearance per finger against the Panda's 8cm stroke, and no amount of tuning tried (gripper stiffness up to 60x, collision friction up to 6x, close-phase duration up to 80 steps) reliably held it through the lift — close to physically marginal for a rigid-body friction-only parallel grasp regardless of controller tuning. 010_potted_meat_can rests upright with a 10.4cm x 5.9cm footprint; gripping across the narrower side gives ~1.05cm clearance per finger and was reliable. Since the default top-down grasp orientation closes along world X but the can's narrow axis sits along world Y, the grasp uses a 90-degree-yawed orientation (GRASP_YAW_QUAT), derived by composing quaternions and confirmed by checking the fingers' actual world-frame separation vector at that orientation, not assumed from the algebra alone.
• The timestep/chatter fix (see status note above) was found here first: data.qvel on the finger joints during the close phase showed real, non-decaying oscillation at the 100Hz default; a 400Hz timestep (requested by the task itself, since it's the one that needs it — MuJoCoScene.reset() recomputes substeps from whatever timestep actually got compiled, not a cached default, so a task overriding it doesn't desync output fps) fixed it. Measured result: 8/8 tested seeds completed a real grasp+lift, 7/8 placed the can within ~2cm of the target marker.

Both MuJoCo kitchen tasks call the same mujoco_backend/assets.add_kitchen_backdrop() (added on request — the counter used to float in black space) — a tiled back wall and a window, mounted behind the counter and outside the robots' operating envelope (wall_x is set behind the counter's back edge; everything added here is contype=0/conaffinity=0, pure visual dressing, same as the place markers).

• Wall material: a real CC0 Polyhaven photo-scan, long_white_tiles (polyhaven.com/a/long_white_tiles) for the backsplash, fetched once into ~/.cache/sdge/textures/. Its texrepeat was tuned by rendering it standalone first (a naive [4, 6] repeat looked like a fine checkerboard, not tile-sized grout lines — [1.5, 2.5] reads as actual tile at this scene's scale).
• Window is built from primitives, not a texture: Polyhaven's library has no window/glass texture, and a geometric window — a sky-colored pane box plus a white cross-mullion frame built from thin boxes — reads clearly enough at this scene's scale without one.
• Camera framing had to change to actually show any of this: both tasks' cameras were originally tuned tight on the workspace (tuned before the backdrop existed), so the window — mounted well above counter height — fell outside the frame entirely. Both cameras were pulled back and re-aimed higher (verified by rendering standalone test scenes at a few candidate eye/target pairs, not guessed) so the backdrop is actually visible without losing the workspace as the main subject. Confirmed afterward that this was purely a rendering change — both tasks' grasp success and mask/segmentation output were re-verified unaffected.
• Upper cabinets (Polyhaven painted_wooden_cabinet mesh) were added, then removed on request — the loader (_ensure_cabinet_mesh_cached) and placement code existed briefly in this module but were deleted once the user asked for the plain wall/window look instead; the backdrop is deliberately simpler now than the dual-arm/handoff screenshots taken while cabinets were still present.

sdge/tasks/dual_pick_place_kitchen_mujoco.py — two Franka Pandas mounted 0.7m apart on the same kitchen counter, each independently planned (no inter-arm coordination — workspaces are kept physically separate by construction, not by collision-checking one arm's plan against the other's) to pick a real RoboTwin-OD object from its own spot and place it on its own colored marker. First task with more than one robot or more than one manipulable object in a scene, and the first to pull assets from RoboTwin-Platform/RoboTwin instead of YCB.

• Multi-arm/multi-object plumbing, new in mujoco_backend/backend.py: load_panda(..., prefix=...) now returns a PandaHandle (resolved joint/actuator/site names for that specific instance — see assets.py's module docstring) instead of relying on one hardcoded "panda_" prefix; build() returns built["arms"]: list[dict] and built["object_body_names"]: list[str] (every task, even single-arm/single-object ones, now returns these as one-element lists — pick_cube_mujoco.py/pick_place_can_mujoco.py were migrated, not left on a separate code path) instead of flat singular keys; step(action)'s action vector is the concatenation of each arm's 9-dim block, in built["arms"] order.

• RoboTwin-OD asset licensing: researched via gh api against the actual repo/dataset before using anything (not assumed) — RoboTwin's HuggingFace dataset (TianxingChen/RoboTwin2.0) blanket-tags its whole 731-object library license: mit, but with no per-object provenance breakdown; part of that library is sourced from Objaverse/PartNet-Mobility, which normally carry their own separate upstream terms RoboTwin doesn't document per-object. Flagged to the user as a real ambiguity; used here under RoboTwin's own stated MIT label, a decision the user made explicitly, not glossed over. load_robotwin_object also only reaches for rigid, non-articulated props (identified by having a visual/+collision/ GLB pair, not a mobility.urdf) as an extra hedge — that structure is specific to objects NOT sourced from PartNet-Mobility, whose own terms are the murkiest of the three source buckets. Objects fetched from one 3.7GB objects.zip (no per-object selective download exists), cached once at ~/.cache/sdge/robotwin_download/.

• RoboTwin trajectories were investigated and deliberately not used: RoboTwin's own trajectories come from its own SAPIEN seed-search-and-replay loop, tied to its own task scripts and motion planner, and its 100k+ pre-collected episodes on HuggingFace are aloha_agilex-only (no dual_franka data exists to download) — "borrowing" them would really mean re-running RoboTwin's own generation pipeline, not porting files, and wouldn't match SDGE's own architecture (plan trajectories in-engine; see the top-level "Why CPU sim, and why no replay stage" section). Only RoboTwin's object meshes are reused here; the actual grasp/place trajectories are planned by this project's own motion_planning.build_pick_place_trajectory, same as every other MuJoCo task.

• Three real bugs were found and fixed getting this to actually work, in order, each isolated before the next was found — all verified via data.xpos/data.xquat/data.qvel readback, not theorized: (1) a missing close_steps=60 override (present in pick_place_can_mujoco.py, forgotten here) left the default too short to settle before lifting; (2) RoboTwin's own collision/*.glb is not actually convex (verified: trimesh's is_convex is False, 10384 faces for 071_can) and MuJoCo's mesh-mesh contact pipeline assumes single mesh geoms are — even after taking the convex hull, several objects still failed reliably, isolated to two distinct causes (checked one variable at a time: swapping in a plain MJCF box in the exact same scene lifted cleanly, isolating the mesh itself as the problem): some "cans" actually rest lying on their side (long axis horizontal, not a standing cylinder), so pinching across the round cross-section let them roll out of the grip regardless of clearance or friction; thin objects produced near-degenerate hulls that visibly sank into the table instead of resting on it. Fixed by using a plain bounding box for collision instead of any mesh-derived shape — never degenerate, and flat-faced so it doesn't have the round-cross-section rolling failure mode either (see assets.load_robotwin_object's docstring); (3) RoboTwin mesh origins sit at the object's top, not its center (the bounding box's own center-offset came back equal to -half_extents.z) — grasp targets computed directly from the body's reported position aimed at the top edge, not the middle; small enough to not matter on the smaller object but the larger one's gripper missed it outright until plan() corrected for the offset using the same bounding-box data build() already computed. Measured result after all three fixes: 8/8 tested episodes had both arms grasp, lift, and place their object within ~1-2cm of its target marker.

  ⚠️ Known bug: --episodes N with N>1 reliably crashes on this backend on the 2nd episode — PhysX tensors plugin: Simulation view object is invalidated and cannot be used again when IsaacScene.reset() tears down and rebuilds /World for episode 2, then RuntimeError: Failed to create articulation. Not investigated further (episode 1 always succeeds; ran one collect process per episode, seed varied per invocation, as a workaround to build the demo dataset in data/demo_dataset/pick_ycb/). Likely needs the physics simulation view explicitly released/recreated alongside the USD stage teardown in reset(), not just the stage content.

sdge/tasks/tomato_bowl_handoff_mujoco.py: right arm picks up an object and places it in a bowl in the middle of the counter; left arm then picks that same object back out of the bowl and places it to the bowl's left. Unlike dual_pick_place_kitchen, the two arms' plans are genuinely dependent — the left arm needs to know where the right arm actually left the object — which is new territory this codebase hadn't needed before. Real, current status, not smoothed over:

• Object choice: no tomato mesh was found with a clean license (RoboCasa has one, CC-BY-4.0, but only as part of a multi-GB Box-hosted download requiring robosuite installed to extract — not pulled in for a placeholder); the real YCB 013_apple stands in instead. Bowl is the real YCB 024_bowl (~16cm diameter, confirmed to settle upright and stay put — heavy enough, ~1.2kg, that neither gripper nudges it).
• Right arm's leg (table -> bowl) works reliably. Left arm's leg (bowl -> table) does not — 0 of 8 tested episodes actually placed the object at its target, and 2 of 8 triggered a genuine MuJoCo numerical divergence (object position exploding to 50+ meters within a fraction of a second — not an ordinary miss). Root cause not fully solved:
  • The SAPIEN pick_apple.py task's fix for gripping a sphere (grip above center, exploiting the curvature for a narrower cross-section) turned out to be actively wrong for this backend — verified by sweeping the offset from the equator up to the SAPIEN value: contacts formed fine at every offset during the close phase, but only the equator grip (GRASP_HEIGHT_OFFSET=0.0) actually survived the lift transition. The two backends' Panda models don't share the same stroke-vs-clearance trade-off; a value tuned on one doesn't transfer.
  • A bowl is curved, not flat — a dropped object does not reliably settle at "bowl-bottom-center plus object-radius" the way every flat-table place_p in this codebase does (verified: it actually rested ~6cm higher, against the bowl's sloped wall, enough to make a blind pick at the naive predicted height miss outright). Fixed by having plan() actually simulate phase 1 with real dynamics (_step_actions, mj_step, not build_pick_place_trajectory's IK-only scratch), reading back where the object truly comes to rest, then rewinding every body phase 1 touched (_save_free_body_state/_restore_free_body_state) before phase 2 is planned — so the real episode execution, replaying the same actions deterministically from the same rewound state, reproduces what was just observed. This part works correctly (verified: phase 2's pick target lands within a few mm of the object every time).
  • What's still unsolved: even grasping at the correct, observed location, the left arm's regrasp is unreliable, and occasionally provokes the numerical divergence above — plausibly the moving arm's stiff position servo dragging through the object's geometry once its planned trajectory continues past an unplanned grip loss, but not confirmed. close_steps=150 for this leg (vs. 60 everywhere else) measurably reduced how often the divergence happened but did not fix the underlying unreliability or eliminate it entirely.
• A new safety net came out of chasing this (MuJoCoScene.step() in backend.py): every step now checks data.qpos for non-finite values or any magnitude over 50 (every real task's workspace is under ~2m) and raises if so, so a divergent episode becomes a normal collect.py failure (caught, logged to failures.json, discarded) instead of silently writing physically-nonsensical data to disk. Originally set the threshold at 1000 and verified that was too permissive — the real divergent episodes topped out around 330-350, comfortably under that but nowhere near a real reach.
• Measured on this machine (single process, no --workers): 8 episodes in 232s (6 completed without diverging, 2 caught by the new safety check and discarded, 0 actually succeeded at the task) — reported as-is, not rounded up.

sdge/backends/sapien_backend/motion_planning.py wraps mplib to plan reach/grasp trajectories against wherever an object actually ended up, instead of hand-probing fixed joint waypoints per task (pick_ycb.py's original approach). Needs mplib>=0.2.1 specifically — mani-skill's own pin (mplib==0.1.1) segfaults under this project's numpy (ABI mismatch); see that module's docstring. Along the way this surfaced two real, previously-unnoticed bugs, both fixed: load_ycb_object() silently never applied its pose argument (every object spawned exactly on top of the robot base and got launched by the resulting interpenetration), and list_ycb_ids() always raised KeyError (stale reference to a dict mani_skill rebinds rather than mutates).

Scene.camera_configs returns a list of CameraConfigs (was a single camera_config) — a task's build() returns "cameras": [...] instead of "camera": ... to render from more than one viewpoint per episode. Each camera independently sets CameraConfig.layered (default True): layered=True gets the full 4-pass treatment (composite/robot_only/object_only/background_only + masks + per-entity depth), layered=False gets only a composite video — cheaper, for a camera you just want a plain view from. Old single-camera tasks ("camera" key, no "cameras") still work unchanged — SapienScene.reset() wraps that into a 1-element list. Every per-camera output file/directory now carries a _{camera_name} suffix (video/composite_front.mp4, mask/robot_visible_front/, depth/object_front/) — this changed the output filenames for every task, including the single-camera ones (e.g. pick_ycb_planned now writes composite_front.mp4, not composite.mp4); episodes already generated before this change (data/demo_dataset/) keep their old unsuffixed filenames on disk, only newly-generated ones use the new convention. sdge/cli/view.py handles both meta.json shapes (cameras list vs. old singular camera) and takes --camera <name> to pick which one to display — it shows one camera at a time, not a multi-camera mosaic (that's real UI work, not done). pick_ycb_multicam (sdge/tasks/pick_ycb_multicam.py) is the demo task: same mplib-planned grasp as pick_ycb_planned, rendered from 3 cameras (front/side/top), all layered=True. The Isaac Lab backend only got a compat shim (camera_configs returns its existing single camera wrapped in a 1-list) — it has no camera registry, so it can't actually add more cameras yet; render()'s own comment already describes what that would take.

cuRobo was successfully built with real GPU-accelerated CUDA kernels on this machine (RTX 5070, Blackwell/sm_120) — this itself required: NVIDIA's CUDA 12.8 apt repo (Ubuntu's own nvidia-cuda-toolkit package only ships CUDA 12.0, whose nvcc doesn't recognize compute_120 and fails outright), python3.12-dev, and building with CUROBO_USE_PYBIND=1 (cuRobo's own setup.py default, 0, silently installs a pure-Python package with zero compiled kernels). None of that is the blocker — see the Isaac Lab status note above and pick_ycb_isaac_curobo.py's docstring for why it's not wired into real data collection.

Backend code contains only ABC plumbing (Scene/Backend implementation) — it doesn't know which robot, which object, or what trajectory it's running. Scene composition lives in per-backend task modules, dispatched by name through a registry:

sdge/
├── core/            # Backend/Scene ABCs + shared dataclasses (Pose6D, ContactRecord, EntityInfo, ...)
├── backends/
│   ├── sapien_backend/
│   │   ├── backend.py     # SapienScene/SapienBackend — generic plumbing only
│   │   └── assets.py      # load_panda(), load_ycb_object() — SAPIEN asset loaders
│   ├── isaac_backend/
│   │   ├── backend.py     # IsaacScene/IsaacBackend — generic plumbing only
│   │   └── assets.py      # load_panda(), load_ycb_object() — Isaac Lab asset loaders
│   │                       #   (OBJ->USD conversion + physics-baking for real YCB import)
│   └── mujoco_backend/
│       ├── backend.py     # MuJoCoScene/MuJoCoBackend — generic plumbing only, multi-arm/multi-object
│       ├── assets.py      # load_panda() [PandaHandle, multi-instance], load_ycb_object(),
│       │                   #   load_robotwin_object() — Menagerie/YCB/RoboTwin-OD fetch/cache
│       └── motion_planning.py  # self-contained Jacobian IK (no mplib) + grasp/pick-place trajectories
├── tasks/
│   ├── __init__.py            # get_task_module(name, backend) registry
│   ├── pick_ycb.py             # SAPIEN: hand-tuned baseline
│   ├── pick_ycb_planned.py     # SAPIEN: same scene, mplib-planned trajectory
│   ├── pick_cube.py            # SAPIEN: ManiSkill3 PickCube-v1-style, mplib-planned
│   ├── stack_cube.py           # SAPIEN: ManiSkill3 StackCube-v1-style, mplib-planned
│   ├── pick_ycb_random.py      # SAPIEN: random YCB object per episode, mplib-planned
│   ├── pick_ycb_multicam.py    # SAPIEN: 3-camera (front/side/top) demo, all layered=True
│   ├── pick_apple.py           # SAPIEN: real YCB apple, mplib-planned
│   ├── pick_ycb_isaac.py       # Isaac Lab: hand-tuned baseline
│   ├── pick_ycb_isaac_curobo.py  # Isaac Lab: cuRobo-planned — KNOWN BROKEN, see its docstring
│   ├── pick_cube_mujoco.py     # MuJoCo: native box geom, Jacobian-IK-planned
│   ├── pick_place_can_mujoco.py  # MuJoCo: real YCB can, kitchen counter, full pick-and-place
│   ├── dual_pick_place_kitchen_mujoco.py  # MuJoCo: 2 Pandas, real RoboTwin-OD meshes, independent plans
│   └── tomato_bowl_handoff_mujoco.py  # MuJoCo: 2 Pandas, dependent plans (bowl handoff) — partially working, see its section
├── pipeline/        # episode orchestration, multi-pass layered rendering, video encoding
├── io/              # per-episode directory writer
└── cli/             # `python -m sdge.cli.collect ...` / `python -m sdge.cli.view <episode_dir>`

Backend.create_scene(task_cfg) looks up task_cfg["task"] in the registry and calls that module's build()/plan() — adding a new task means adding a (task_name, backend) module pair, not touching backend code. A task module for one backend is not portable to the other (the scene-construction APIs share nothing — sapien.Scene vs Isaac's config-object system), only numeric task parameters (e.g. joint waypoints) are shared, by value.

Both backends' assets.py load from the same downloaded mesh cache — ~/.maniskill/data/assets/mani_skill2_ycb/models/<id>/ (fetched once via python -m mani_skill.utils.download_asset ycb from the SAPIEN .venv; see docs/robot-sim-survey.md for how that was verified). SAPIEN consumes YCB's .obj/.ply files directly via ManiSkill3's builder function. Isaac Lab has no equivalent built-in loader, so isaac_backend/assets.py converts the OBJ to USD via omni.kit.asset_converter and bakes physics schemas into the file, caching the result under ~/.cache/sdge/ycb_usd/<id>/<id>.usd — conversion only happens once per object id, ever.

Both design calls are explained in detail in the plan/commit history, summarized here:

• CPU sim, single env (not GPU-parallel): ManiSkill3's GPU-mode Actor.hide_visual() only works for actors without collision shapes — YCB objects have collision shapes, so GPU-mode hide/show silently fails for exactly the entities we need to isolate for layered rendering. CPU sim's visibility toggle has no such restriction, and we don't need thousands of parallel envs for a sequential data generator anyway.
• No two-stage sim-then-replay: multi-pass renders happen inline during the single forward simulation rollout (step physics → capture all GT at that exact state → continue), avoiding any replay-determinism drift.

Two changes make it safe/practical to generate thousands of episodes of one task in one collect invocation, not just the handful used for smoke-testing everywhere above:

• One episode's failure no longer kills the batch. Every backend's task-planning step (most visibly the MuJoCo IK solver's RuntimeError on non-convergence, but this is generic) used to propagate straight out of collect.py and abort every remaining episode. _run_episode_range() now catches per-episode exceptions, deletes that episode's (possibly partial) output directory, logs {episode, seed, error} to <out>/<task>/failures.json, and moves on. The run's exit code is still non-zero if anything failed (so it stays a visible problem, not a silently swallowed one), but the episodes that did succeed are all there. Fixing this exposed a real, independent bug while wiring it up: sdge/pipeline/episode_runner.py's run_episode() used to call scene.reset() (where that same RuntimeError originates) before its own try/finally, so a planning failure skipped scene.close() entirely and leaked that scene's MuJoCo/EGL render context — harmless for one failure, a real problem across a large batch's worth of them. Fixed by widening the try to cover reset() too.
• --workers N parallelizes across N OS processes, each loading its own Backend (and therefore, for MuJoCo, its own EGL render context — verified interactively that giving workers independent contexts via multiprocessing.get_context("spawn"), not the Linux-default fork, was necessary: forking after a GL/EGL context already exists in the parent is a known source of corrupted child contexts). --workers 1 (the default) stays exactly the original single-process, zero-subprocess-overhead path — every example in this README that doesn't pass --workers is still running the simple way. Measured on this machine: --episodes 8 --workers 4 on MuJoCo pick_cube completed in 6.5s wall-clock (28s of total CPU time across workers) vs. ~24s running the same 8 sequentially — roughly the expected ~4x from 4 workers, verified by actually timing it, not assumed.

# large batch, fault-tolerant, parallel — a handful of bad seeds won't lose the whole run
MUJOCO_GL=egl python -m sdge.cli.collect --backend mujoco --task pick_cube --episodes 2000 --workers 8 --out data/bulk_pick_cube
# if anything failed: data/bulk_pick_cube/pick_cube/failures.json has {episode, seed, error} per failure

One directory per episode, one file per modality — see docs/robot-sim-survey.md §5 / the plan for the full schema, or sdge/io/episode_writer.py for the authoritative layout. Identical across both backends.

SAPIEN backend:

python3 -m venv .venv
source .venv/bin/activate
pip install -e .
python -m mani_skill.utils.download_asset ycb -y   # shared YCB mesh cache, see above

Isaac Lab backend (separate environment — see sdge/backends/isaac_backend/backend.py's module docstring for the full install story):

python3 -m venv .venv-isaac
source .venv-isaac/bin/activate
pip install "isaacsim[all,extscache]==6.0.1.0" --extra-index-url https://pypi.nvidia.com
pip install cmake  # avoids needing sudo/apt for the Isaac Lab build step
git clone --depth 1 https://github.com/isaac-sim/IsaacLab.git /tmp/IsaacLab
cd /tmp/IsaacLab && ./isaaclab.sh -i core
pip install opencv-python-headless pandas pyarrow pyyaml tqdm  # shared sdge pipeline deps

MuJoCo backend (same .venv as SAPIEN — pip install -e . above already includes it):

# First run auto-fetches the Menagerie Panda MJCF into ~/.cache/sdge/mujoco_menagerie/
# (shallow + sparse git checkout, only franka_emika_panda/) — nothing extra to install.
MUJOCO_GL=egl python -m sdge.cli.collect --backend mujoco --task pick_cube --episodes 2 --out data/smoke_test_mujoco

Resolved incident, worth knowing about: this repo's directory used to be named with spaces ("Simulator Dataset Generation Engine"), and whatever originally created .venv/.venv-isaac truncated that path to "...Enginee" (missing the final "r") inside pyvenv.cfg/activate/every generated console-script shebang. source activate silently set PATH to a nonexistent directory, which fell through to whatever python came next on PATH (no error) — commands "in" an activated env could silently run a completely different interpreter with none of the project's dependencies installed. Fixed by renaming the directory to Simulator_Dataset_Generation_Engine (underscores, no spaces) and bulk-patching every shebang/pyvenv.cfg/activate* under both venvs' bin/ to match — verified by activating both venvs in a fresh shell and checking sys.executable/sys.prefix resolve correctly. If you ever see ModuleNotFoundError for something that's clearly installed (e.g. cv2), check sys.executable first — it's this class of bug. /tmp/IsaacLab does not survive a reboot (/tmp is cleared) — if import isaaclab fails after one, re-clone and re-run ./isaaclab.sh -i core; isaacsim itself (pip-installed, ~19GB, unaffected by any of this) does not need reinstalling.

Optional — cuRobo (only needed if working on the Isaac motion-planning dead end documented in pick_ycb_isaac_curobo.py; not required for normal data collection):

# CUDA 12.8+ toolkit from NVIDIA's own repo — Ubuntu's `nvidia-cuda-toolkit` apt package (CUDA 12.0)
# doesn't recognize Blackwell-class GPUs (sm_120) and fails with "Unsupported gpu architecture"
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb && sudo apt update
sudo apt install -y cuda-toolkit-12-8 python3.12-dev

git clone --depth 1 https://github.com/NVlabs/curobo.git /tmp/curobo
cd /tmp/curobo
export PATH="/usr/local/cuda-12.8/bin:$PATH"
export CUROBO_USE_PYBIND=1   # cuRobo's setup.py default (0) skips building any CUDA kernels at all
"<repo>/.venv-isaac/bin/python3" -m pip install -e . --no-build-isolation --no-deps

# SAPIEN — hand-tuned baseline, or any of the mplib-planned tasks:
python -m sdge.cli.collect --backend sapien --task pick_ycb --episodes 2 --out data/smoke_test
python -m sdge.cli.collect --backend sapien --task pick_ycb_planned --episodes 2 --out data/smoke_test
python -m sdge.cli.collect --backend sapien --task pick_cube --episodes 2 --out data/smoke_test
python -m sdge.cli.collect --backend sapien --task stack_cube --episodes 2 --out data/smoke_test
python -m sdge.cli.collect --backend sapien --task pick_ycb_random --episodes 2 --out data/smoke_test
python -m sdge.cli.collect --backend sapien --task pick_ycb_multicam --episodes 2 --out data/smoke_test  # 3 cameras, all layered
python -m sdge.cli.collect --backend sapien --task pick_apple --episodes 2 --out data/smoke_test

# Isaac Lab (from .venv-isaac — see the activation gotcha above) — first run converts+caches the YCB mesh, ~10-20s one-time cost
python -m sdge.cli.collect --backend isaac --task pick_ycb --episodes 2 --out data/smoke_test_isaac

# MuJoCo (same .venv as SAPIEN) — first run fetches+caches the Menagerie Panda MJCF, one-time cost
MUJOCO_GL=egl python -m sdge.cli.collect --backend mujoco --task pick_cube --episodes 2 --out data/smoke_test_mujoco
# first run also fetches+caches the kitchen wall tile texture + converts the YCB can's collision mesh
MUJOCO_GL=egl python -m sdge.cli.collect --backend mujoco --task pick_place_can --episodes 2 --out data/smoke_test_mujoco
# first run also fetches RoboTwin-OD's objects.zip (~3.7GB, one-time, ~6min at ~10MB/s)
MUJOCO_GL=egl python -m sdge.cli.collect --backend mujoco --task dual_pick_place_kitchen --episodes 2 --out data/smoke_test_mujoco
# partially working — see its README section; most episodes won't actually complete the handoff
MUJOCO_GL=egl python -m sdge.cli.collect --backend mujoco --task tomato_bowl_handoff --episodes 2 --out data/smoke_test_mujoco

# Desktop viewer — synchronized playback of one camera's 4 layered videos + 4 masks + 2 depth maps
# (backend-agnostic: reads only mp4/png/npy/json from disk, works on either backend's output
# from the SAPIEN .venv — no need to switch to .venv-isaac just to look at Isaac's episodes)
python -m sdge.cli.view data/smoke_test/pick_ycb/episode_000000
python -m sdge.cli.view data/smoke_test/pick_ycb_multicam/episode_000000 --camera side  # pick a camera; default is the first one