Intelligent robotic object manipulation is a line of research which is concerned with how robots (particularly robotic arm manipulators) can manipulate their environments to accomplish a certain high-level task. For instance, a robot that can open a door, fold clothes, or retrieve an object from a shelf.

Firstly, it’s worth taking the time to reflect on the excellent human ability to manipulate environments with great success. We perform highly complex manipulation tasks with our hands and with objects around us fairly successfully. Think of preparing a meal (chopping onions, grating cheese, etc), folding clothes, loading objects into the fridge, and so on.

We will use the following illustration to represent the challenges of intelligent robotic object manipulation (the illustration looks better in light mode rather than dark mode):

A robot assisting a motor-impaired person with everyday chores, such as retrieving an orange juice bottle from the back of a cluttered fridge. Obstructions at the front of the fridge complicate the robot’s direct access to the object. Hence, the robot must plan to rearrange these obstacles, possibly by safely pushing them aside, to retrieve the desired object.

Perception and Computer Vision

At the forefront is perception. A robot requires the capability to sense the environment, build a model of the world to work with, and identify objects of interest, obstacles and their position in space. Even with substantial advancements in deep learning and computer vision, determining the position and orientation of objects in 3D space, especially in cluttered and partially obscured scenes, remains a challenge. In real-life scenarios, like our fridge example, nearby items can confuse the most advanced systems, sometimes resulting in significant errors and false positives.

Task and Motion Planning

A robot needs a capability to plan its motion. The term motion planning refers to the problem of finding a solution to drive a robot from an initial state to a goal state. In the context of a robotic arm, with a more specific term, manipulation planning, the problem is to find the sequence of controls that will move the arm from the current configuration, to achieve a certain manipulation goal (say, the robot reaching into the fridge to grasp an object). Working with a robotic arm, it usually involves a much higher degree of freedom (DOF) system, which makes it much harder and computationally expensive to plan.

Motion planning has indeed progressed, especially for collision-avoidance motion. But in certain scenarios, collision isn’t always detrimental; for instance, while a robot must avoid damaging the fridge, it may need to interact with obstacles to move them out of the way, for example by pushing them. The primary challenge is the vast dimensionality of these planning problems, demanding intelligent solutions and algorithms.

Physics-based Manipulation

While we touched on collision-free planning earlier, also known as “geometric motion planning”, there’s another dimension: physics-based planning. Here, motion planning algorithms must account for the real-world physics and dynamics of actions. Integrating these physics-aware algorithms requires physics simulation, which introduces further complexities. One major challenge is the potential discrepancy between simulated physics and real-world dynamics, posing the risk of real-world failures despite simulated successes. Physics simulation is also expensive to run¹ which poses challenges for fast and real-time motion planning.

Open world manipulation

Ideal environments, where robots operate with fixed objects, known geometries, and repetitive actions, seldom exist outside manufacturing settings. Real-world environments are dynamic, with varied objects and unpredictable human interactions. Thus, creating a system adaptive to these “open worlds” presents its own set of challenges. How does each fridge look like? How are objects expected to be placed in every fridge?

The general focus and interest of our research is in the above challenges: perception, motion planning, physics-based manipulation planning and motion control in an open world. We are interested in building algorithms that enable robots to solve real world problems to amplify what humans can do.