The task was to design a fully autonomous system in which a UR5 robotic arm and a mobile robot work together to sort packages in a warehouse. The competition had two stages. We first developed everything in simulation (Gazebo), then deployed and tested it on real hardware in a warehouse at IIT Bombay through remote access.

Tech stack: ROS 2, Gazebo, MoveIt 2, RViz 2, Nav2, slam_toolbox, OpenCV, Intel RealSense, Git

How the system works

Perception: finding the packages

Developed a framework that estimates the pose of ArUco-tagged boxes in the UR5 base frame using an Intel RealSense camera.

1. Convert the camera image to grayscale.
2. Detect the ArUco markers with OpenCV and store each marker's corners and ID.
3. For every marker in the frame, compute its area and drop the ones below a threshold, since they are too far from the arm's reach.
4. Get the rotation and translation vectors of the remaining markers with respect to the camera, along with their center points.
5. Convert each pose into a TF in the base_link frame and publish it for the manipulation module.

Manipulation: picking and placing with the UR5

Programmed the UR5 pick-and-place operation, driving the gripper to the estimated box poses using MoveIt Servo.

• The pose error between the box and the gripper is fed into MoveIt 2's forward position controller service.
• Once the gripper touches the box, the gripper is commanded to attach through its control service.
• The arm then moves to a pre-defined drop position, detaches the box and drops it into a container.

Navigation: moving around the warehouse

Configured and tuned slam_toolbox mapping, AMCL localization and Nav2 navigation on a live autonomous mobile robot (AMR). Wheel odometry was fused with the IMU using an EKF.

Docking: attaching the rack

The mobile robot aligns itself underneath a rack, and an electromagnet turns on to attach it. Built an IMU and ultrasonic docking controller for this alignment.

Videos