Kaifeng Zhang^1,2*, Shuo Sha^1,2*, Hanxiao Jiang¹, Matthew Loper², Hyunjong Song², Guangyan Cai², Zhuo Xu³, Xiaochen Hu², Changxi Zheng^1,2, Yunzhu Li^1,2

¹Columbia University, ²SceniX Inc., ³Google DeepMind, ^*Equal contribution. Work partially done while interning at SceniX Inc.

This repository contains the official implementation of the Real-to-Sim-Policy-Eval framework. The main components are:

• Construct simulation assets: visualize object Gaussians, articulate robot Gaussians, define and render different object layouts.

• Deploy trained PhysTwin to simulate deformable object Gaussians.

• Train policies using the policy_training submodule located in policy/.

• Evaluate policy in the constructed simulation environment.

• Interactive control of the Gaussian-based simulation.

• This project uses uv, a fast Python package and project manager that serves as a lightweight replacement for Conda. A detailed uv tutorial is available here. You can also install all dependencies with Conda following similar steps.

• The installation has been tested with CUDA 12.8. The CUDA 12.8 installation guide is available here. If you use a different CUDA version, make sure to install PyTorch with the matching CUDA toolkit; otherwise, the build of diff-gaussian-rasterizer-w-depth may fail.

• GLM is required to compile diff-gaussian-rasterizer-w-depth. Install it with: sudo apt-get update && sudo apt-get install -y libglm-dev

• FFmpeg is required for policy rollout video generation. FFmpeg cannot be installed directly via uv, but you can install it either system-wide or through Conda. In the latter case, activate your Conda environment before activating the uv virtual environment.

# create and activate a python venv
uv venv --python=3.11
source .venv/bin/activate

# install main dependencies
uv pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128
uv pip install -r pyproject.toml

# pull the policy submodule and install dependencies
git submodule update --init --recursive
cd policy
uv pip install -r pyproject.toml

# install Gaussian Splatting rasterizer
cd ../third-party/diff-gaussian-rasterization-w-depth
uv pip install --no-build-isolation -e .

# install a modified version of urdfpy to avoid version conflicts
cd ../urdfpy-0.0.22
uv pip install -e .

# return to the project home
cd ../..

We provide the Gaussian Splatting assets, PhysTwin checkpoints, and the policy checkpoints we used in the paper on Huggingface.

After downloading, extract the files to the following folders:

• Gaussian Splatting scans: log/gs/scans/. E.g., there should be log/gs/scans/rope/, log/gs/scans/sloth/, etc.

• PhysTwin assets: log/phystwin/<object_name>/, and rename <object_name> to one of rope, sloth, T. E.g., there should be log/phystwin/rope/data/, log/phystwin/sloth/data/, log/phystwin/T/data/, etc.

• Policy checkpoints: log/policy_checkpoints/. E.g., there should be log/policy_checkpoints/act-rope-routing/checkpoints/001000/ (lerobot-style) and log/policy_checkpoints/pi0-rope-routing/10000/ (openpi-style).

The config files of all the experiments are managed using hydra under cfg/. The subfolders of the config, and their functionalities are listed follows:

• cfg/env: camera intrinsics/extrinsics, robot model/initial poses, simulation duration, etc.
• cfg/gs: object meshes, GS assets, poses, randomizations, etc.
• cfg/physics: phystwin-related configs
• cfg/policy: paths to policy and task configs for the policy/ submodule
• cfg/renderer: additional camera configs for GS rendering

Make sure you download the GS scans, PhysTwin checkpoints, and policy checkpoints for the corresponding task. Specify them in configs, for example, if you want to run the xarm-gripper environment with the rope GS assets, and using the rope PhysTwin, with pretrained ACT policy at iteration 7000, the command is:

CUDA_VISIBLE_DEVICES=0 python experiments/eval_policy.py \
gs=rope \
env=xarm_gripper \
physics.ckpt_path=log/phystwin/rope \
physics.case_name=rope_0001 \
policy.inference_cfg_path=policy/configs/inference/insert_rope.json \
policy.checkpoint_path=log/policy_checkpoints/act-rope-routing/checkpoints/007000

Execution-ready bash files are located in experiments/scripts/eval_policy.

Additional: When multiple GPUs are available on a single node, episodes of policy evaluation can be distributed across multiple GPUs. For example, for the same evaluation as above but with 200 episodes and 8 GPUs, the parallelized launching script is:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python experiments/eval_policy_parallel.py \
gs=rope \
env=xarm_gripper \
physics.ckpt_path=log/phystwin/rope \
physics.case_name=rope_0001 \
policy.inference_cfg_path=policy/configs/inference/insert_rope.json \
policy.checkpoint_path=log/policy_checkpoints/act-rope-routing/checkpoints/007000 \
gs.use_grid_randomization=False \
policy.n_episodes=200

Execution-ready bash files are located in experiments/scripts/eval_policy_parallel.

After running the evaluation, rollouts will be saved to log/experiments/output_eval_policy/ by default. To visualize the start and end frames of each episode in a grid, use the visualization function provided by:

python experiments/utils/visualize_rollouts.py --data_dir log/experiments/output_eval_policy/<run_name>

You can calculate the success rates of each task by designing criterias based on simulation states, which are saved along with renderings in the policy rollout episodes. For the three tasks covered in our paper, the corresponding commands for calculating success rates are:

# rope routing
python experiments/utils/calculate_success_rope.py --data_dir log/experiments/output_eval_policy/<run_name>

# toy packing
python experiments/utils/calculate_success_sloth.py --data_dir log/experiments/output_eval_policy/<run_name>

# T-block pushing
python experiments/utils/calculate_success_T.py --data_dir log/experiments/output_eval_policy/<run_name>

CUDA_VISIBLE_DEVICES=0 python experiments/keyboard_teleop.py \
gs=rope \
env=xarm_gripper \
physics.ckpt_path=log/phystwin/rope \
physics.case_name=rope_0001

Execution-ready bash files are located in experiments/scripts/keyboard_teleop.

In motion replay mode, instead of running a policy, we can simply read pre-computed trajectories (which may come from a real-world rollout) and follow the trajectories in sim. To test this function, download the sample replay data from here. Then, extract the file into log/policy_rollouts/rope_act_7000/, and the replay script is

CUDA_VISIBLE_DEVICES=0 python experiments/replay.py \
gs=rope \
env=xarm_gripper \
physics.ckpt_path=log/phystwin/rope \
physics.case_name=rope_0001 \
gt_dir=log/policy_rollouts/rope_act_7000

Execution-ready bash files are located in experiments/scripts/keyboard_teleop.

For policy training, please refer to the policy submodule README here for details.

Creating an asset for this simulator includes the following steps:

We use the Scaniverse app for Gaussian Splatting reconstruction in all our experiments. It is possible to use any Gaussian Splatting training tools, as long as the checkpoints/reconstructions are saved as the PLY format.

We use the SuperSplat interactive editor for positional alignment (cropping, scaling, rotating, translating). The goal is to make Gaussians of individual objects match their corresponding mesh files. The alignment between Gaussians and mesh vertices can be checked using

python assets/scans/visualize_scan.py --gs_path path/to/ply/file

after exporting the modified Gaussians from SuperSplat as PLY format.

For positionally aligning the background scans (including the robot), we use SuperSplat to make the robot orientations and scene origins roughly align with the orientation and origins of the robot URDF file, respectively. Then, fine-grained robot point ICP alignment and robot segmentation is done by

# for xarm7 robot with gripper; if you downloaded our gs scans, 
# no need to run these as the scans have already been processed; 
# for new scenes, change the paths and robot bounding box extents 
# in construct_scene_gripper.py first
python assets/scans/construct_scene_gripper.py --construct  # run ICP, get aligned robot GS
python assets/scans/construct_scene_gripper.py --visualize  # visualize the results

# for xarm7 robot with pusher; if you downloaded our gs scans, 
# no need to run these as the scans have already been processed; 
# for new scenes, change the paths and robot bounding box extents 
# in construct_scene_pusher.py first
python assets/scans/construct_scene_pusher.py --construct  # run ICP, get aligned robot GS
python assets/scans/construct_scene_pusher.py --visualize  # visualize the results

To align the color between raw GS renderings and the desired color space (e.g., images from real-world cameras), we use a quadratic RGB color transformation. First, collect paired sim-real (source-reference) images. Sample images can be downloaded from here, and extracted into log/color_alignment. Then, the transformation can be estimated by

python assets/scans/color_alignment.py  # change paths in file for your custom data

Please refer to the original PhysTwin codebase for phystwin training. After the training, within the saved checkpoint, the required files are:

• data/<case_name>/final_data.pkl
• experiments/<case_name>/train/best_xxx.pth
• experiments_optimization/<case_name>/optimal_params.pkl

After saving the checkpoints, specify the path to the checkpoint as ckpt_path in cfg.physics.

Special case. If you wish to create phystwins of rigid objects, we provide a easy script that samples points uniformly inside mesh and connect dense high-stiffness springs to form a rigid digital twin. The following command uses T-block as example (suppose you already have the mesh model of the object):

python experiments/utils/create_rigid_phystwin.py \
--rigid_mesh_path experiments/log/gs/scans/T/T_mesh.ply \
--ckpt_path experiments/log/phystwin/T \
--case_name T_0001 \
--cfg_path experiments/cfg/physics/default.yaml

• Initial release: sim engine, Gaussian Splatting, policy training, policy evaluation, keyboard teleoperation
• More object scans
• PhysTwin training
• Gello teleoperation
• More robot embodiments
• Integration with mainstream simulation platforms

If you find this repo useful, please consider citing the paper

@article{zhang2025real,
    title={Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions},
    author={Zhang, Kaifeng and Sha, Shuo and Jiang, Hanxiao and Loper, Matthew and Song, Hyunjong and Cai, Guangyan and Xu, Zhuo and Hu, Xiaochen and Zheng, Changxi and Li, Yunzhu},
    journal={arXiv preprint arXiv:2511.04665},
    year={2025}
}