OpenAI launches Point-E, an AI that generates 3D models

The next breakthrough to conquer the AI ​​world could be 3D model generators. This week OpenAI Point-E is released as open source, a machine learning system that creates a 3D object after receiving a text message. According to an article published with the code, basePoint-E can create 3D models in 1-2 minutes on a single Nvidia V100 GPU.

Point-E does not create 3D objects in the traditional sense. Instead, it creates point clouds or separate collections of data points in space that represent a 3D shape, hence the cheeky acronym. (The “E” in Point-E is short for “efficiency” because it’s clearly faster than previous approaches to generating 3D objects.) Point clouds are computationally easier to synthesize, but they don’t capture the granularity of an object. Shape or texture: a major limitation of Point-E these days.

To get around this limitation, the Point-E team trained an additional AI system to convert Point-E point clouds into meshes. (Meshes, the collection of vertices, edges, and faces that define an object, are commonly used in 3D modeling and design.) But, as the article points out, the model sometimes ignores certain parts of objects and can result in blocked or distorted shapes .

Outside of the independent meshing model, Point-E consists of two models: a text-to-image model and an image-to-3D model. The text-to-image model, similar to generative art systems such as DALL-E 2 and Stable Diffusion, was trained on labeled images to understand associations between words and visual concepts. The image-to-3D model, on the other hand, got a series of images linked to 3D objects to learn how to effectively translate between the two.

When presented with a text message, such as “a 3D printable gear, a single gear 3 inches in diameter and 1/2 inch thick,” Point-E’s text-to-image template generates a rendered synthetic object that resembles the image which is then entered image. 3D model generating a point cloud.

After training the models with a dataset of “millions” of 3D objects and associated metadata, Point-E was able to generate clouds of colored dots that often matched text hints, the OpenAI researchers say. Not perfect: Point-E’s image-to-3D model sometimes doesn’t understand the text-to-image model’s image, resulting in a shape that doesn’t match the text message. Still, it’s much faster than the prior art, at least according to the OpenAI team.

“While our method outperforms more modern techniques, it yields samples in a fraction of the time,” they wrote in the paper. “This could make it more practical for certain applications or enable the discovery of higher quality 3D objects.”

What are the apps exactly? Well, researchers at OpenAI point out that Point-E’s point clouds can be used to create real-world objects, for example through 3D printing. With the addition of the mesh conversion model, once the system is more advanced, it could also make its way into animation and game development workflows.

OpenAI may be the last company to join the 3D object generator fray, but as mentioned above, it’s certainly not the first. Earlier this year, Google released DreamFusion, an improved version of Dream Fields, a 3D generative system the company introduced in 2021.

While all eyes are on 2D art generators these days, model synthesis AI could be the next big disruptor in the industry. 3D models are widely used in film and television, interior design, architecture, and various scientific fields. For example, architectural firms use them to demonstrate proposed buildings and landscapes, while engineers use the models to design new equipment, vehicles and structures.

However, creating 3D models often takes a while, from a few hours to a few days. AI like Point-E could change that if the issues are fixed, making OpenAI a respectable asset in the process.

The question is what kinds of intellectual property disputes may arise over time. There is a huge market for 3D models with several online marketplaces including CGStudio and CreativeMarket where artists can sell the content they create. If Point-E catches on and its models hit the market, model artists could protest, citing evidence that modern generative AI relies heavily on its training data: in Point-E’s case, existing 3D models. Like DALL-E 2, Point-E does not mention or cite any of the artists who may have influenced its generations.

But OpenAI is leaving that topic for another day. Neither the Point-E document nor the GitHub page mentions copyright.