Disambiguate Gripper State in Grasp-Based Tasks: Pseudo-Tactile as Feedback Enables Pure Simulation Learning

Abstract

Grasp-based manipulation tasks are fundamental to robots interacting with their environments, yet gripper state ambiguity significantly reduces the robustness of imitation learning policies for these tasks. Data-driven solutions face the challenge of high real-world data costs, while simulation data, despite its low costs, is limited by the sim-to-real gap. We identify the root cause of gripper state ambiguity as the lack of tactile feedback. To address this, we propose a novel approach employing pseudo-tactile as feedback, inspired by the idea of using a force-controlled gripper as a tactile sensor. This method enhances policy robustness without additional data collection and hardware involvement, while providing a noise-free binary gripper state observation for the policy and thus facilitating pure simulation learning to unleash the power of simulation. Experimental results across three real-world grasp-based tasks demonstrate the necessity, effectiveness, and efficiency of our approach.

Overview

The principle of tactile sensors is that they convert the deformation of sensor material into measurable changes in resistance, capacitance, or visual signals. A force-controlled gripper can act as a tactile sensor providing pseudo-tactile information. We designed a closed-loop gripper controller with pseudo-tactile information as feedback to convert the empty close state into the empty open state, disambiguating gripper state.

Once the gripper state is disambiguated, there is no need to perform gripper disturbance during data collection, and the policy can use the binary gripper state observation, enabling pure simulation training. We design a state-based expert policy for automatic data collection and apply real-to-sim techniques, randomization, and admittance control to mitigate the visual sim-to-real gap and reduce the stress caused by kinematic discrepancies.

Pure Simulation Training

Sim Demos Collection

We design state-based expert policies to collect demos in simulation automatically.

Eval in Sim

The performance of the learned policy in simulation.

Real-World Pick-and-Lift

Eval without Gripper Disturbance

Diffusion Policy with 50 Real-World Demos

RDT-1B Finetuned with 50 Real-World Demos

Diffusion Policy with Pure Sim Training

Eval with Gripper Disturbance

Diffusion Policy with 50 Real-World Demos

RDT-1B Finetuned with 50 Real-World Demos

Diffusion Policy with Pure Sim Training

Real-World Drawer-Opening

Eval without Gripper Disturbance

ACT with 60 Real-World Demos

Diffusion Policy with 60 Real-World Demos

RDT-1B Finetuned with 60 Real-World Demos

Diffusion Policy with Pure Sim Training

Eval with Gripper Disturbance

ACT with 60 Real-World Demos

Diffusion Policy with 60 Real-World Demos

RDT-1B Finetuned with 60 Real-World Demos

Diffusion Policy with Pure Sim Training

Real-World Oven-Opening

Eval without Gripper Disturbance

ACT with 54 Real-World Demos

Diffusion Policy with 54 Real-World Demos

RDT-1B Finetuned with 54 Real-World Demos

Diffusion Policy with Pure Sim Training

Eval with Gripper Disturbance

ACT with 54 Real-World Demos

Diffusion Policy with 54 Real-World Demos

RDT-1B Finetuned with 54 Real-World Demos

Diffusion Policy with Pure Sim Training

Ablation - Gripper Randomization

In ablation studies, we perform gripper randomization during sim demos collection to include cases where the gripper is closed without successfully grasping the object. We use such dataset to train ACT, DP, and RDT-1B (finetune). The results showed that while disturbance resilience was achieved, task accomplishment was severely compromised. When the robot grasps the handle, the policies often fail to perform the expected pulling motion.

Diffusion Policy (Ablation-2)

ACT (Ablation-4)

RDT-1B (Ablation-5)

BibTeX


@misc{yang2025disambiguategripperstate,
  title={Disambiguate Gripper State in Grasp-Based Tasks: Pseudo-Tactile as Feedback Enables Pure Simulation Learning}, 
  author={Yifei Yang and Lu Chen and Zherui Song and Yenan Chen and Wentao Sun and Zhongxiang Zhou and Rong Xiong and Yue Wang},
  year={2025},
  eprint={2503.23835},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2503.23835}, 
}