NVIDIA Omniverse Gets a Major Upgrade With Generative AI and OpenUSD

Artificial intelligence, simulation, and robotics are continuing to rapidly advance. NVIDIA Corp. today announced a major release of its NVIDIA Omniverse software that it said offers new foundation applications and services for developers to enhance and optimize their 3D pipelines with the OpenUSD framework and generative AI.

Omniverse is a Universal Scene Description or OpenUSD-native platform for connecting, describing, and simulating across 3D tools and applications. NVIDIA said the update accelerates the creation of virtual worlds and advanced workflows for industrial digitalization.

Cesium, Convai, Move AI, SideFX Houdini, and Wonder Dynamics are now connected to Omniverse via OpenUSD, said the Santa Clara, Calif.-based company in a release.

Key highlights from the platform update include advancements to Omniverse Kit -- the engine for developing native OpenUSD applications and extensions -- as well as to the NVIDIA Omniverse Audio2Face app and spatial-computing capabilities.

“Industrial enterprises are racing to digitalize their workflows, increasing the demand for OpenUSD-enabled, connected, interoperable, 3D software ecosystems,” said Rev Lebaredian, vice president of Omniverse and simulation technology at NVIDIA. “The latest Omniverse update lets developers tap generative AI through OpenUSD to enhance their tools, and it allows enterprises to build larger, more complex world-scale simulations as digital testing grounds for their industrial applications."

"We're [also] releasing a SIM Ready specification that applies physics properties to USD assets so that AI agents such as robots and autonomous vehicles can learn about our real world from completely within simulated worlds," he added during a press briefing.

"For example, an autonomous robot tasked with sorting packages needs to be trained in simulation on 3D packages that move and react to physical contact just as they would in the real world," said NVIDIA.

Omniverse gets several upgrades

NVIDIA listed the following updates to Omniverse Kit:

• New modular app building — A new Omniverse Kit Extension Registry, a central repository for accessing, sharing, and managing Omniverse extensions, lets developers easily turn functionality on and off in their application, said NVIDIA. It said this makes it easier than ever to build custom apps from over 600 core Omniverse extensions.
• New developer templates and resources — New application and experience templates provide developers a head start with OpenUSD and Omniverse with minimal coding, according to the company.
• Boosted efficiency and user experience — New rendering optimizations take advantage of the NVIDIA Ada Lovelace architecture enhancements in NVIDIA RTX graphics processing units (GPUs) with DLSS 3 technology fully integrated into the Omniverse RTX Renderer. A new AI de-noiser enables real-time 4K path tracing of massive industrial scenes.
• Native RTX-powered spatial integration — New extended-reality (XR) developer tools let users build spatial-computing options natively into their Omniverse-based applications. This gives users the flexibility to experience their 3D projects and virtual worlds however they like, NVIDIA said.

These platform updates are showcased in Omniverse foundation applications, which are fully customizable reference applications that creators, enterprises, and developers can copy, extend, or enhance. Upgraded applications include:

• Omniverse USD Composer — Lets 3D users assemble large-scale, OpenUSD-based scenes.
• Omniverse Audio2Face — Provides access to generative AI application programming interfaces (APIs) that create realistic facial animations and gestures from only an audio file. It now includes multilingual support and a new female base model, said NVIDIA.

OpenUSD expands Omniverse ecosystem

NVIDIA also announced a broad range of frameworks, resources, and services for developers and companies to accelerate OpenUSD adoption.

In addition, the company announced new Omniverse Cloud APIs, built by NVIDIA, for developers to more seamlessly implement and deploy OpenUSD pipelines and applications. For example, ChatUSD is a large language model (LLM) copilot that can answer USD questions or generate Python-USD code scripts.

New Omniverse connections enabled by OpenUSD are now available, offering more opportunities for enterprises to break data silos in their complex production pipelines, said NVIDIA.

NVIDIA connects with more collaborators

Expanding their collaboration across Adobe Substance 3D, generative AI and OpenUSD initiatives, Adobe and NVIDIA announced plans to make Adobe Firefly — Adobe’s family of creative generative AI models — available as APIs in Omniverse. They said it will allow developers and creators to enhance their design processes.

Wonder Dynamics is connected to Omniverse with new OpenUSD export support through Wonder Studio. The AI platform automatically animates, lights, and composes computer-generated characters into live-action scenes. New OpenUSD export support will enable artists to generate and export a complete 3D scene — all from a single camera video, NVIDIA said.

Luma AI said its reality-capture models in USDZ format can be readily imported to Omniverse. Tools from avatar company Convai and character-engine company Inworld AI are also connected to Omniverse.

With AI tools like Convai, creators can add characters in their digital twin environments that can provide relevant information about the environment and objects ranging from tour guides to virtual robots. Move AI enables single-camera motion capture with the Move One app, which can generate 3D character animations that can then be exported to OpenUSD and used in Omniverse.

Omniverse users can now build content, experiences, and applications that are compatible with other OpenUSD-based spatial computing platforms such as ARKit and RealityKit. Plus, new support for the Khronos Group’s OpenXR open standard expands Omniverse use to more headsets from manufacturers such as HTC VIVE, Magic Leap, and Varjo.

Users of SideFX Houdini can also now load Houdini Digital Assets directly into the Omniverse viewport, making Houdini-based connected workflows more seamless, said NVIDIA. The Cesium for Omniverse extension enables 3D Tiles, an open standard for streaming massive geospatial datasets in virtual worlds, including those supported by OpenUSD.

NVIDIA also said that CGI.Backgrounds now has several ultra-high-definition HDRi maps available in USD Composer.

The Cadence DataCenter Design Software, now available via Omniverse, helps users see computational fluid dynamics simulations in the complete context of their digital twins. With the data center extension, users can plan, test, and validate design and operational considerations before implementing them, the company said.

In addition, NVIDIA noted that the Blackshark.AI world digital twin platform is now connected to Omniverse.

Customers use Omniverse for digitalization

Customers are using Omniverse for tasks ranging from simulating robots to training AI models and improving animation.

The Boston Dynamics AI Institute is using Omniverse to simulate robots and their interactions to enable the design of novel robotics and control systems.

Continental, a leading automotive and autonomous systems company, said it is using Omniverse to generate physically accurate synthetic data at scale to train computer-vision AI models and test systems integration for robots.

Volvo Cars has transitioned its digital twin to be OpenUSD-based, using Omniverse to create immersive visualizations to help customers make online purchasing decisions.

Marks Design, a brand design and experience agency, is using Omniverse and OpenUSD to streamline collaboration and improve its animation, visualization, and rendering workflows.

Luxury electric vehicle maker DENZA, a joint venture between Mercedes-Benz and BYD, has worked with communications firm WPP and NVIDIA Omniverse Cloud to build the next generation of car configurators. WPP can use generative AI or scans of real-world environments captured by a Boston Dynamics Spot robot to create custom environments.

New Omniverse systems and partners

NVIDIA is collaborating with global systems manufacturers to bring RTX workstations optimally configured for Omniverse to millions of designers, architects and engineers.

The new systems feature up to four NVIDIA RTX 6000 Ada Generation GPUs. They are bundled with NVIDIA Omniverse Enterprise software to accelerate OpenUSD world-building, generative AI-enhanced collaborative design, and other industrial digitalization applications.

Omniverse users can also take advantage of the new NVIDIA L40S GPU, a powerful, universal data center GPU that accelerates the most graphics-intensive workloads.

Beta of new release now available

The latest Omniverse release is now available in beta to download for free and coming soon to Omniverse Enterprise.

At SIGGRAPH 2023, NVIDIA CEO Jensen Huang provided an exclusive look at some of the company's award-winning research, OpenUSD developments, and the latest AI-powered systems for content creation.

Founded in 1993, NVIDIA said it is a pioneer in accelerated computing. The company claimed that its invention of the GPU in 1999 "sparked the growth of the PC gaming market, redefined computer graphics, ignited the era of modern AI and is fueling industrial digitalization across markets."