GitHub

This code release accompanies the following project:

Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, Thomas Funkhouser

Project Page | PDF | arXiv | Video

Abstract: For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people's preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions. This approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios.

Here is an overview of how this codebase is organized:

We recommend using Conda environments. Our setup (tested on Ubuntu 20.04.6 LTS) uses the following 3 environments:

See the respective READMEs inside the server and robot directories for detailed setup instructions.

Unless otherwise specified, the tidybot Conda env should always be used:

We provide a teleoperation interface (teleop.py) to operate the robot using primitives such as pick, place, or toss.

First, run this command to start the teleop interface on the server (workstation), where is 1, 2, or 3, depending on the robot to be controlled:

On the robot (mobile base computer), make sure that the convenience stop and mobile base driver are both running. Then, run this command to start the controller:

Once the server and robot both show that they have successfully connected to each other, use these controls to teleop the robot:

Notes:

To generate paths with an occupancy map rather than manually clicking waypoints, use the --shortest-path flag.

This will load the receptacles specified in scenarios/test.yml as obstacles and build an occupancy map to avoid running into them.

For additional debugging visualization, the --debug flag can be used.

Server:

Robot:

To operate the robot in fully autonomous mode, we use the demo interface in demo.py. By default, the demo will load the test scenario in scenarios/test.yml along with the corresponding LLM-summarized user preferences in preferences/test.yml.

To start the demo on the server, first start the object detector server with the vild Conda env:

Then, in a separate terminal, start the demo interface (with the tidybot env):

On the robot, make sure that the convenience stop and mobile base driver are both running. Then, run this command to start the controller:

These are the controls used to run the demo:

To load a different scenario (default is test), use the --scenario-name argument:

For example, to load scenario-08 and use robot #1, you can run:

For additional debugging visualization, the --debug flag can be used.

Server:

Robot:

The marker detection setup should output 2D robot pose estimates with centimeter-level accuracy. For instance, our setup can reliably pick up small Lego Duplo blocks (32 mm x 32 mm) from the floor. Inaccurate marker detection can be due to many reasons, such as inaccurate camera alignment or suboptimal camera settings (see get_video_cap in utils.py). Also note that the mobile base motors should be calibrated (.motor_cal.txt) for more accurate movement.

The 3 Kinova arms are repeatable but have slightly different zero heading positions, so they require some compensation to be consistent with each other. See the arm-dependent heading compensation in controller.py.

If multiple people have been using the server, you may run into this error:

To kill all processes using the occupied ports, you can use the clear-ports.sh script (requires sudo):

For reference, here are all of the ports used by this codebase:

The overhead cameras may occasionally output errors such as this:

Typically, these errors can be resolved by unplugging the camera and plugging it back in.

Be sure to also check the quality and length of the USB extension cable, as USB 2.0 does not support cable lengths longer than 5 meters.

If you find this work useful for your research, please consider citing: