MetaFold: Language-Guided Multi-Category Garment Folding Framework via Trajectory Generation and Foundation Model
- Use Isaac Sim 4.1 as the main simulation environment.
- Update the Python interpreter of your virtual environment to use the
python.shscript provided in the Isaac Sim root directory.- Example:
/path/to/isaac-sim/python.sh
- Example:
pip install -r requirements.txtInstall the required dependencies, including but not limited to:
Ensure all packages are correctly installed within the Isaac Sim Python environment.
The model file is split into multiple parts. Merge them into a single .pth file:
cat data/model_1113_199.pth.part_* > data/model_1113_199.pthThis section describes how to run experiments, either in single-instance mode or batch mode.
Use the following script to run a single experiment:
python start_single.pyPay attention to the input parameters in start_single.py:
# input
# --------------------------------------------------------------------------------------------
descriptions = [...] # Folding instruction sentences
cloth_config_path = "./data/Cloth-Simulation/Configs/Pants.json" # Cloth configuration file
cloth_root = "./data/Cloth-Simulation/Assets/eval/" # Root directory of cloth assets
cloth_name = "5PS_2" # Name of the cloth folderExample of a cloth configuration file:
{
"type": "Pants",
"scale": [0.5, 0.5, 0.5],
"stage_num": 2, # Total number of folding stages (actions)
"frame_per_stage": [60, 60], # Frame count for each stage
"force_per_stage": [2, 2], # Force count for each stage
"step_turn": [1, 1], # Currently unused
"frames_per_step": 10, # Frames per step (for closed-loop prediction)
"threshold": 0.05 # Currently unused
}Use the following script to run a single experiment:
python start_batch.pyPay attention to the input parameters in start_batch.py:
# input
# --------------------------------------------------------------------------------------------
batch_input_path = "./data/Cloth-Simulation/Batch_Input/batch_input.json"
instructions_path = "./data/Cloth-Simulation/Batch_Input/Instructions.json" # Name of the cloth folderExample of a cloth configuration file:
[
{
"cloth_root": "./data/Cloth-Simulation/Assets/eval" # Initial entry: specifies the root directory for cloth assets
},
{
"cloth_type": "No-sleeve", # Cloth type; must match a key in Instructions.json
"cloth_name": "0DSNS_0" # Name of the cloth folder
},
{
"cloth_type": "Pants",
"cloth_name": "3PS_0"
}
]{
"No-sleeve": {
"cloth_config_path": "./data/Cloth-Simulation/Configs/No-sleeve.json", # Path to cloth configuration file
"descriptions": [ # Folding instructions
"fold the no-sleeve bottom-up"
]
},
"Long-sleeve": {
"cloth_config_path": "./data/Cloth-Simulation/Configs/Long-sleeve.json",
"descriptions": [
"Please fold Long-Sleeve top from the left sleeve",
"Please fold Long-Sleeve top from the right sleeve",
"Please fold Long-Sleeve top from the bottom-up"
]
}
}After the simulation finishes, check the output directory:
./isaac_sim/outputEach experiment will generate a corresponding folder:
.../mesh*Inside, you will find two point cloud files:
initial_pcd.txt — Point cloud of the initial state
fin_pcd.txt — Point cloud of the final state
During the simulation, you can monitor the predicted trajectories in:
./vis_pc_eval_real.png
./vis_pc2_eval_real.png
These files visualize trajectory predictions across different steps of the folding process.