InstructMesh lets novices fix flawed AI-generated 3D models before printing

Why AI-generated 3D designs often fail

MIT News describes a familiar frustration. Ask a generative AI system for a 3D-printable mug and you may get a cup that cannot hold coffee. The models grasp appearance but not function, which produces impractical designs. Fixing the errors is also hard, since such models are typically difficult to edit, especially for people new to 3D design.

InstructMesh is meant to address both issues. Users can prompt it to generate a 3D design, for instance a pair of glasses, and then highlight specific parts of the blueprint they want refined before 3D printing. The interface is designed to understand how the designs should look and which edits experts and novices want to make.

How it works and what it produced

The system pairs Microsoft's TRELLIS, which creates 3D models from text and image prompts, with the large language model GPT-4, which supports ChatGPT. Faraz Faruqi, lead author of the paper and a recent CSAIL affiliate, said: "We wanted to bring together the talents of 3D generators and the reasoning skills of LLMs in an interactive space to make objects that people actually want."

Faruqi explained that manipulation happens in the latent space of the generative model, so InstructMesh "supports natural language description of issues, and creates interpretive changes in the geometry for the user to evaluate and approve." Sliders give users extra precision for tweaks such as enlarging or extruding a part of the model.

The researchers used the tool to make a mug that appears wrapped by a dragon, with its tail as the handle, a shiny blue whistle resembling a shell, and glasses with butterfly wings above the lens. They also produced an octopus-like dispenser that pours drinks into several cups at once, and a knee brace that looks like denim to match a patient's jeans. A "bristle bot" shaped like a colorful shrimp, with a motor hidden inside, slides across surfaces when switched on.

Testing with newcomers

To see how beginners would fare, the team had TRELLIS recreate popular models from Thingiverse, a platform home to millions of 3D printable models. Nearly 80 percent of the generated models were structurally flawed in some way. CSAIL researchers then asked novices to identify and fix these issues in InstructMesh. They managed both around 90 percent of the time, as reviewed by an expert.

Participants went on to create items such as phone stands and vases and said the tool was easy to use. They also found it let them express a wide range of ideas.

What comes next

Faruqi now works at Google, where he may soon incorporate InstructMesh into an augmented reality platform. The idea is to describe what you need using the context of your surroundings and have it quickly 3D printed, such as a phone case that matches your wallet. The tool may also add physics simulations to model how a design reacts to specific uses, for example whether a bowl breaks when dropped and which materials suit it best. It might also integrate the more recent TRELLIS.2 to refine even smaller features.

Stefanie Mueller, an associate professor at MIT and a member of CSAIL, is a senior author. The work was supported in part by Google and the MIT-HPI Collaborative Research Program, and will be presented at the ACM Symposium on User Interface Software and Technology in November.

For people new to 3D modeling, the results suggest that guided, language-based editing can make it realistic to catch design flaws before printing, rather than after a failed print.