Hi, great work! I’ve been adapting the code in example/position_retargeting to other hand datasets beyond DexYCB and noticed two small bugs:

1. Line 145, hand_robot_viewer.py: the wrist quaternion is computed in the camera frame, but should be in the world frame.
2. Line 70, seq_retargeting.py: the matrix should be @operator2mano, not @operator2mano.T.

Here’s why:

• SeqRetargeting.retarget uses last_qpos → forward_kinematics → target_points and compares them to ref_value (human keypoints in the world frame) so last_qpos must also be expressed in the world frame.
• But last_qpos is initialized in warm_start, which from han_pose in raw data (in camera frame). Despite root2wrist, the current code leaves it as identity because the FK joints are all zeros. No camera information is incorporated afterward. Consequently, even if the fingertips are aligned, the wrist quaternion remains in the camera frame, creating a gap between the human and robot wrists in the first frame (see figure 1, this is from 20201002_104620 in DexYCB, the points and lines are 21 human hand joints and the blue mesh is allegro hand.).

• Besides, @operator2mano (no transpose) is the correct transformation; setting all quaternions to zero to see whether human hands align with dexterous hand confirms this.

After these two tweaks, both fingertips and wrists align correctly.

I suppose the original code still looks reasonable on DexYCB because the camera extrinsic is close to operator2mano.T, but it is acutally buggy, e.g. fails on OakInkV2 or TACO.