You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: instruct_to_policy/README.md
+17-1Lines changed: 17 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,6 +9,22 @@ You can download the processed data for evaluation from [ADD_LINK_HERE](#).
9
9
10
10
You can also refer to the [data document](./data.md)s more details about the data generation process to process your custom data.
11
11
12
+
## Quick Example
13
+
Assume you have had the data downloaded properly and you have configured the main environment for Roboscript and second environment for grasp detection.
14
+
For example, `~/catkin_ws` with `catkin_ws` conda env and `~/anygrasp_ws` with `anygrasp` conda env.
15
+
You can run the following command to run the evaluatation of GPT4 generated code in the real time:
16
+
17
+
```bash
18
+
# Make sure the RoboScript is in catkin_ws and you have already catkin built and activated the workspace
19
+
cd /path_to_RoboScript/instruct_to_policy
20
+
# Or roscd instruct_to_policy
21
+
bash scripts/bash/run_demo.sh
22
+
```
23
+
24
+
You should see the Gazebo Simulation, the pop-up terminal for grasp detection model, and the robot will start to execute the generated code.
25
+
26
+
27
+
12
28
## Inetractive Demo with JupyterRos (Optional)
13
29
14
30
You might need some hack to install jupyter ros with ipython kernel aware of ros workspace. Please refer to [Installation](https://jupyter-ros.readthedocs.io/en/latest/user_installation.html) and [Troubleshooting](https://jupyter-ros.readthedocs.io/en/latest/user_troubleshooting.html) for more details.
@@ -62,7 +78,7 @@ With gazebo, moveit and other services ready, you can run the code evaluation wi
"query": "# objects = [table, cabinet, apple, banana, white_ceramic_plate, white_and_pink_box, brown_box, brown_medication_bottle, bleach_cleanser, red_plastic_bowl, mustard_bottle] ; # place all fruits on the table into the white plate.",
5
+
"src_fs": {
6
+
"check_object_on_receptacle": "def check_object_on_receptacle(object_name, receptacle):\n # Object above has a larger z coordinate\n return get_object_center_position(object_name)[2] > get_object_center_position(receptacle)[2]"
7
+
},
8
+
"code_str": "# Detailed planning:\n# Step 1: Identify all fruits on the table\n# Step 2: For each fruit on the table, grasp it and put it on the white plate\n\n# Identify all fruits on the table\nfruit_names = ['apple', 'banana']\nfruits_on_table = []\nfor fruit_name in fruit_names:\n if check_object_on_receptacle(object_name=fruit_name, receptacle='table'):\n fruits_on_table.append(fruit_name)\n\n# For each fruit on the table, grasp it and put it on the white plate\nfor fruit_name in fruits_on_table:\n # Grasp the fruit\n open_gripper()\n grasp_pose = parse_adaptive_shape_grasp_pose(fruit_name)\n grasp(grasp_pose)\n close_gripper()\n attach_object(fruit_name)\n\n # Move the fruit to the white plate\n place_pose = parse_place_pose(fruit_name, 'white_ceramic_plate')\n move_to_pose(place_pose)\n\n # Release the fruit\n open_gripper()\n detach_object(fruit_name)",
9
+
"gvars": [
10
+
"os",
11
+
"rospy",
12
+
"np",
13
+
"Pose",
14
+
"PoseStamped",
15
+
"Point",
16
+
"Quaternion",
17
+
"move_group",
18
+
"get_object_center_position",
19
+
"get_object_pose",
20
+
"get_3d_bbox",
21
+
"get_object_name_list",
22
+
"detect_objects",
23
+
"get_object_joint_info",
24
+
"parse_adaptive_shape_grasp_pose",
25
+
"parse_horizontal_grasp_pose",
26
+
"parse_place_pose",
27
+
"get_gripper_pose",
28
+
"open_gripper",
29
+
"close_gripper",
30
+
"attach_object",
31
+
"detach_object",
32
+
"move_to_pose",
33
+
"follow_path",
34
+
"grasp",
35
+
"gripper_group",
36
+
"objects",
37
+
"check_object_on_receptacle"
38
+
],
39
+
"lvars": []
40
+
},
41
+
{
42
+
"context": "",
43
+
"query": "# objects = [table, cabinet, apple, banana, white_ceramic_plate, white_and_pink_box, brown_box, brown_medication_bottle, bleach_cleanser, red_plastic_bowl, mustard_bottle, cabinet.drawer_0, cabinet.drawer_1, cabinet.drawer_2, cabinet.handle_0, cabinet.handle_1, cabinet.handle_2] ; # open the second top cabinet drawer and place all boxes into it .",
44
+
"src_fs": {
45
+
"move_in_direction": "def move_in_direction(direction, distance=0.25):\n # Get the current pose of the gripper\n current_pose = get_gripper_pose()\n\n # Calculate the target pose\n target_pose = Pose()\n target_pose.position.x = current_pose.position.x + direction[0] * distance\n target_pose.position.y = current_pose.position.y + direction[1] * distance\n target_pose.position.z = current_pose.position.z + direction[2] * distance\n target_pose.orientation = current_pose.orientation\n\n # Move the gripper to the target pose\n move_to_pose(target_pose)"
46
+
},
47
+
"code_str": "# Detailed planning:\n# Step 1: Grasp the handle of the second top cabinet drawer\n# Step 2: Pull the handle of the second top cabinet drawer, move in +x direction\n# Step 3: Release the handle of the second top cabinet drawer\n# Step 4: Grasp the boxes one by one\n# Step 5: Move the boxes into the drawer\n# Step 6: Release the boxes\n\n# Grasp the handle of the second top cabinet drawer\nopen_gripper()\ngrasp_pose = parse_horizontal_grasp_pose('cabinet.handle_1')\ngrasp(grasp_pose)\nclose_gripper()\nattach_object('cabinet.handle_1')\n\n# Pull the handle of the second top cabinet drawer, move in +x direction\ndirection = [1, 0, 0] # (x, y, z)\nmove_in_direction(direction, distance=0.25)\n\n# Release the handle of the second top cabinet drawer\nopen_gripper()\ndetach_object('cabinet.handle_1')\n\n# Grasp the boxes one by one\nbox_names = ['white_and_pink_box', 'brown_box']\nfor box_name in box_names:\n # Grasp the box\n grasp_pose = parse_adaptive_shape_grasp_pose(box_name)\n grasp(grasp_pose)\n close_gripper()\n attach_object(box_name)\n\n # Move the box into the drawer\n place_pose = parse_place_pose(box_name, 'cabinet.drawer_1')\n move_to_pose(place_pose)\n\n # Release the box\n open_gripper()\n detach_object(box_name)",
48
+
"gvars": [
49
+
"os",
50
+
"rospy",
51
+
"np",
52
+
"Pose",
53
+
"PoseStamped",
54
+
"Point",
55
+
"Quaternion",
56
+
"move_group",
57
+
"get_object_center_position",
58
+
"get_object_pose",
59
+
"get_3d_bbox",
60
+
"get_object_name_list",
61
+
"detect_objects",
62
+
"get_object_joint_info",
63
+
"parse_adaptive_shape_grasp_pose",
64
+
"parse_horizontal_grasp_pose",
65
+
"parse_place_pose",
66
+
"get_gripper_pose",
67
+
"open_gripper",
68
+
"close_gripper",
69
+
"attach_object",
70
+
"detach_object",
71
+
"move_to_pose",
72
+
"follow_path",
73
+
"grasp",
74
+
"gripper_group",
75
+
"objects",
76
+
"move_in_direction"
77
+
],
78
+
"lvars": []
79
+
},
80
+
{
81
+
"context": "",
82
+
"query": "# objects = [table, cabinet, apple, banana, white_ceramic_plate, white_and_pink_box, brown_box, brown_medication_bottle, bleach_cleanser, red_plastic_bowl, mustard_bottle] ; # exchange the position of the apple and the banana on the table.",
83
+
"src_fs": {},
84
+
"code_str": "# Detailed planning:\n# Step 1: Grasp the apple\n# Step 2: Move the apple to a temporary location on the table\n# Step 3: Release the apple\n# Step 4: Grasp the banana\n# Step 5: Move the banana to the original location of the apple\n# Step 6: Release the banana\n# Step 7: Grasp the apple from the temporary location\n# Step 8: Move the apple to the original location of the banana\n# Step 9: Release the apple\n\n# Grasp the apple\nopen_gripper()\ngrasp_apple_pose = parse_adaptive_shape_grasp_pose(object_name='apple')\ngrasp(grasp_apple_pose)\nclose_gripper()\nattach_object('apple')\n\n# Move the apple to a temporary location on the table\napple_temp_place_pose = parse_place_pose(object_name='apple', receptacle_name='table', position=np.array([0.0, 0.0, 0.0]))\nmove_to_pose(apple_temp_place_pose)\n\n# Release the apple\nopen_gripper()\ndetach_object('apple')\n\n# Grasp the banana\ngrasp_banana_pose = parse_adaptive_shape_grasp_pose(object_name='banana')\ngrasp(grasp_banana_pose)\nclose_gripper()\nattach_object('banana')\n\n# Move the banana to the original location of the apple\nbanana_place_pose = parse_place_pose(object_name='banana', receptacle_name='table', position=get_object_center_position('apple'))\nmove_to_pose(banana_place_pose)\n\n# Release the banana\nopen_gripper()\ndetach_object('banana')\n\n# Grasp the apple from the temporary location\ngrasp_apple_pose = parse_adaptive_shape_grasp_pose(object_name='apple')\ngrasp(grasp_apple_pose)\nclose_gripper()\nattach_object('apple')\n\n# Move the apple to the original location of the banana\napple_place_pose = parse_place_pose(object_name='apple', receptacle_name='table', position=get_object_center_position('banana'))\nmove_to_pose(apple_place_pose)\n\n# Release the apple\nopen_gripper()\ndetach_object('apple')",
"query": "# objects = [table, cabinet, apple, footed_bowl, cabinet.drawer_0, cabinet.drawer_1, cabinet.handle_0, cabinet.handle_1] ; # place the apple into the footed bowl.",
5
+
"src_fs": {},
6
+
"code_str": "# Detailed planning:\n# Step 1: Grasp the apple\n# Step 2: Move the apple to the footed bowl\n# Step 3: Release the apple\n\n# Grasp the apple\nopen_gripper()\ngrasp_pose = parse_adaptive_shape_grasp_pose('apple')\ngrasp(grasp_pose)\nclose_gripper()\nattach_object('apple')\n\n# Move the apple to the footed bowl\nplace_pose = parse_place_pose('apple', 'footed_bowl')\nmove_to_pose(place_pose)\n\n# Release the apple\nopen_gripper()\ndetach_object('apple')",
"query": "# objects = [table, cabinet, white_ceramic_plate, turquoise_salad_plate, lemon, brown_ceramic_cup] ; # place the brown cup into an empty plate, you need to find empty plate by checking if there is any object in the plate's 2D region on the table.",
5
+
"src_fs": {
6
+
"check_plate_empty": "def check_plate_empty(plate_name):\n # Use the bbox to check if there is any object on the plate\n objects = get_object_name_list()\n bbox_plate = get_3d_bbox(plate_name)\n for obj in objects:\n if obj != plate_name:\n # check if the object is on the plate\n obj_center = get_object_center_position(obj)\n if_obj_on_plate_x = bbox_plate[0] < obj_center[0] < bbox_plate[3]\n if_obj_on_plate_y = bbox_plate[1] < obj_center[1] < bbox_plate[4]\n if_obj_on_plate_z = obj_center[2] > bbox_plate[5] # object is above the plate\n if if_obj_on_plate_x and if_obj_on_plate_y and if_obj_on_plate_z:\n return False # there is an object on the plate\n return True # no object on the plate"
7
+
},
8
+
"code_str": "# Detailed planning:\n# Step 1: Check which plates are empty\n# Step 2: If there is an empty plate, grasp the brown cup\n# Step 3: Move the brown cup to the empty plate\n# Step 4: Release the brown cup\n\n# Check which plates are empty\nplate_names = ['white_ceramic_plate', 'turquoise_salad_plate']\nempty_plates = []\nfor plate_name in plate_names:\n if check_plate_empty(plate_name):\n empty_plates.append(plate_name)\n\n# If there is an empty plate, grasp the brown cup\nif len(empty_plates) > 0:\n open_gripper()\n grasp_cup_pose = parse_adaptive_shape_grasp_pose(object_name='brown_ceramic_cup')\n grasp(grasp_cup_pose)\n close_gripper()\n attach_object('brown_ceramic_cup')\n\n # Move the brown cup to the empty plate\n place_cup_pose = parse_place_pose(object_name='brown_ceramic_cup', receptacle_name=empty_plates[0])\n move_to_pose(place_cup_pose)\n\n # Release the brown cup\n open_gripper()\n detach_object('brown_ceramic_cup')\nelse:\n print(\"No empty plate found.\")",
"query": "# objects = [table, cabinet, orange, toy_bus, basket_with_fabric_liner] ; # clear the table by moving all table objects into the basket.",
5
+
"src_fs": {
6
+
"check_object_on_receptacle": "def check_object_on_receptacle(object_name, receptacle):\n # Object above has a larger z coordinate\n return get_object_center_position(object_name)[2] > get_object_center_position(receptacle)[2]"
7
+
},
8
+
"code_str": "# Detailed planning:\n# Step 1: Check which objects are on the table\n# Step 2: For each object on the table, grasp it and put it in the basket\n\n# Check which objects are on the table\nobject_names = ['orange', 'toy_bus']\nobjects_on_table = []\nfor object_name in object_names:\n if check_object_on_receptacle(object_name=object_name, receptacle='table'):\n objects_on_table.append(object_name)\n\n# For each object on the table, grasp it and put it in the basket\nfor object_name in objects_on_table:\n # Grasp the object\n open_gripper()\n grasp_pose = parse_adaptive_shape_grasp_pose(object_name)\n grasp(grasp_pose)\n close_gripper()\n attach_object(object_name)\n\n # Move the object to the basket\n place_pose = parse_place_pose(object_name, 'basket_with_fabric_liner')\n move_to_pose(place_pose)\n\n # Release the object\n open_gripper()\n detach_object(object_name)",
0 commit comments