Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

[Project Page] [Paper]

Jianren Wang^*1, Sudeep Dasari^*1, Mohan Kumar¹, Shubham Tulsiani¹, Abhinav Gupta¹

¹Carnegie Mellon University, * indicates equal contribution

Data

Data is located here.

Usage

Below are example scripts for training and testing on franka emika.

Setup

1. Clone repo.

git clone https://github.com/AGI-Labs/manipulate-by-seeing

2. Create and activate conda environment.

conda env create -f environment.yml
conda activate seeing

3. Set env path.

export PYTHONPATH=$PYTHONPATH:path_to_proj/

Training

1. Download the data from the above google drive.

2. Put action_*.pickle under dataloader.

3. Edit yaml files

root_dir: path_to_data/
save_dir: path_to_save_the_weights/

4. To train:

   python scripts/trainer.py --experiment='knob_turning_0113' --task='knob_turning'

Testing

1. Follow this to install polymetis for robot control.

2. Edit yaml files

model_weights: path_to_pretrained_models/
gripper_weights: path_to_pretrained_gripper_predictor/

3. Edit GripperController

close configuration: [here](https://github.com/AGI-Labs/manipulate-by-seeing/blob/1d1f91d8bee98fa7fee399da3993efe0a71c7671/inference/robot_setup.py#L76)
open configuration: [here](https://github.com/AGI-Labs/manipulate-by-seeing/blob/1d1f91d8bee98fa7fee399da3993efe0a71c7671/inference/robot_setup.py#L79)

4. This command to test on Franka Emika.

python inference/run.py --task='pick_and_place'

Acknowledgement

The controller of dynamixel motors is borrowed from Vikash Kumar

If you find our paper or code useful, please cite our papers:

@article{wang2023manipulate,
  title={Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations},
  author={Wang, Jianren and Dasari, Sudeep and Srirama, Mohan Kumar and Tulsiani, Shubham and Gupta, Abhinav},
  booktitle={International Conference on Computer Vision (ICCV)},
  year={2023}
}