Trumpeting A Change: Nvidia Releases Artificial Intelligence Tool To Aid Creative Innovations

Nvidia is giving flat photographs added depth — from gaming laptops to RTX data science workstations.

First, the company converted 2D images into 3D scenes, models and videos. Now, it’s turned its attention to editing. A new artificial intelligence system transforms still photos into 3D objects.

The approach, dubbed 3D MoMa, could provide game developers with an easy way to change pictures and scenes. Photogrammetry, which captures measurements from images, is commonly used for this. (Photogrammertry is the art and science of extracting 3D information from photographs.)

Through inverse rendering, 3D MoMa speeds up the process. By analyzing still photographs, this method uses AI to estimate a scene’s physical properties, such as geometry and illumination. After that, the images are reassembled in a realistic 3D format.

The models are created within an hour on a single NVIDIA Tensor Core GPU.  (A GPU is a graphics processing unit.)

“The NVIDIA 3D MoMa rendering pipeline uses the machinery of modern AI and the raw computational horsepower of NVIDIA GPUs to quickly produce 3D objects that creators can import, edit and extend without limitation in existing tools. It formulates every piece of the inverse rendering problem as a GPU-accelerated differentiable component,” said David Lubeke, Nvidia’s vice president of research.

3D MoMa creates items as triangle meshes, a format that is easy to alter using common software. On a single NVIDIA Tensor Core GPU, the models are built in under an hour.

The Computer Vision and Pattern Recognition Conference (CVPR) in New Orleans this week featured 3D MoMa. Nvidia engineers utilized the method to graphically represent jazz to honor the musical genre’s roots. Hundreds of photographs of trumpets, trombones, saxophones, drums and clarinets were originally gathered. The photos were then rebuilt into 3D representations via 3D MoMa. After that, the instruments were reworked and given new materials.

For example, a cheap plastic trumpet was converted into a magnificent gold trumpet. From the brass instruments reflecting brightly to the drum skins absorbing light, the company claims all the instruments reacted to light like they would in the real world. Finally, an animated scene was created using the 3D objects.

Although 3D MoMa is still in the works, Nvidia believes it will allow game developers and other designers to swiftly change 3D objects and then place them in any virtual setting.

Neil Trevett, vice president of developer ecosystems at Nvidia and the president of The Khronos Group, the forum’s host, says the metaverse will require a constellation of standards.

Produced in association with MetaNews.