Seeing robots walk down the street is becoming a norm. However, the technology isn’t quite there yet to cook dinner or do chores around the house. Teaching a robot physical tasks in the real world takes way too much time and effort, mainly because of data.
“One natural idea is to use simulation as a training ground,” Russ Tedrake, a professor at MIT and principal investigator at CSAIL, explained. “While there has been significant progress over the last few years in the physics engines that power robotics simulators, one of the remaining challenges has been creating sufficiently rich and diverse simulation content to capture the complexity of the real world.”
Three AI Agents Train the Robots
Advertisement
A team from MIT CSAIL and Toyota Research Institute built a system called SceneSmith to address this. It uses three AI agents to build detailed 3D virtual rooms from simple text prompts. First, a “designer” maps out a room. Then, a “critic” checks if it makes sense. Finally, an “orchestrator” manages the whole process until it is done. They can fill a room with furniture and drop in items a robot can interact with, like cabinets it can open.
“We’ve found that the system can construct 3D scenes the way a human designer would,” Nicholas Pfaff, an MIT PhD student, said. “We made over 1,300 scenes using a leading VLM that has internet-scale priors, and it made insanely creative and diverse arrangements. I hadn’t taught the system to do that in the prompts; it just improvised.”
Virtual Testing
These virtual rooms have up to six times more items than older methods. Engineers dropped a robot program into these spaces to see what would happen. When told to move an apple to a cutting board, the robot did it perfectly. According to the researchers, it proved the virtual rooms closely match real settings.
Read Exclusive Stories
Bookmark & Save Articles For Later
