Nvidia researchers announced Magic3D on Friday, an AI model that can create 3D models from text descriptions. By entering a text prompt, such as “a koala wearing a business suit,” Magic3D generates a 3D mesh model with colored texture in about 40 minutes. These 3D models could be utilized by both game developers and CGI directors.
Nvidia presents Magic3D in its academic paper as an alternative to DreamFusion, a text-to-3D model that Google researchers announced in September. Like DreamFusion, Magic3D uses a two-stage process that starts with a text-to-image model to generate a 2D image and then optimizes it into volumetric NeRF (Neural radiance field) data. The authors of the paper claim that Magic3D can generate 3D objects two times faster than DreamFusion.
Magic3D has the ability to edit 3D meshes using prompts. By using a low-resolution 3D model and a base prompt, the model can be altered by changing the text. Magic3D’s authors also show how to maintain the same subject throughout multiple generations (coherence) and how to apply the style of a 2D image, such as a cubist painting, to a 3D model.
I am excited to see where Nvidia takes this A.I model, although image and art generators have been very popular in 2022, none of them have really ventured into the 3D design scene yet aside from a few GitHub projects within Stable Diffusion (these implement a similar technique to Google’s previous A.I 3D model generator, DreamFusion).
The ability to generate 3D models from text is a logical next step in the evolution of neural networks for content creation. The researchers behind Magic3D hope that it will enable anyone to create 3D models without specialized training. Once refined, the technology could accelerate the development of video games (and VR) and perhaps find uses in special effects for movies and TV. In their paper, the researchers write, “We hope with Magic3D, we can democratize 3D synthesis and open up everyone’s creativity in 3D content creation.”
Google has not currently released any access or code for Magic3D. We hope to see more of this A.I model in the near future and with any luck Google will release a version for public use!
