New AI co-pilot uses CAD software to generate 3D designs
MIT researchers have developed an AI model, trained on over 41,000 CAD videos, that translates 2D sketches into 3D objects by operating CAD software like a human.
MIT engineers have developed a novel AI system able to use CAD software in a human-like way, controlling the interface with clicks, drags and menu commands to build 3D models from 2D sketches.
The team created a dataset called VideoCAD, comprising more than 41,000 real CAD session videos that explicitly show how users build shapes step-by-step, including mouse movement, keyboard commands and UI interactions.
By learning from this data, the AI agent can translate high-level design intents, such as ‘draw a line’ or ‘extrude a shape’, into specific UI actions like clicking a tool, dragging over a sketch region and executing the command.
When given a 2D drawing, the AI generates a complete 3D model by replicating the sequence of UI interactions a human designer would use. The researchers tested this on a variety of objects, from simple brackets to more complex architectural shapes.
The long-term vision is to build an AI-enabled CAD co-pilot. This tool not only automates repetitive modelling tasks but also works collaboratively with human designers to suggest next steps, speed up workflows or handle tedious operations.
The researchers argue this could significantly lower the barrier to entry for CAD use, making 3D design accessible to people without years of training.
From a digital economy and innovation policy perspective, this development is significant. It demonstrates how AI-driven UI agents are evolving, not just processing text or data, but also driving complex, creative software. That raises questions around intellectual property (who owns the design if the AI builds it?), productivity (will it replace or support designers?) and education (how will CAD teaching adapt?).
Would you like to learn more about AI, tech and digital diplomacy? If so, ask our Diplo chatbot!
