WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Explore the USD ecosystem

Explore the USD ecosystem

USDMaterialXvisionOS3D
Watch original video

Highlight

Apple has fully adopted USD as the core 3D content format on visionOS, introducing MaterialX shader support, Display P3 color management, and collaborating with the industry to advance the OpenUSD ecosystem—enabling developers to create spatial computing content with standard industry toolchains.

Core Content

USD: The Universal Language for 3D Content

USD (Universal Scene Description), now renamed OpenUSD, was created by Pixar and is the industry-standard format for representing 3D content, used from individual creators to large studios. Apple has supported USD on its platforms since 2017, and USD is now the core format for 3D content on visionOS.

USD’s core strength is composition. It expresses asset data through complex hierarchies, letting multiple people work in parallel on different parts and compose them into a complete scene. Reality Composer Pro leverages this, letting team members handle different aspects of a scene.

MaterialX: A Portable Shader Standard

MaterialX was created by Industrial Light & Magic for Star Wars and is now maintained by the Academy Software Foundation. It lets artists compose shader logic through node graphs without writing code. MaterialX graphs can be embedded in USD files and travel with scene data.

RealityKit on visionOS supports rendering MaterialX shaders. Reality Composer Pro’s Shader Graph is based on MaterialX nodes, with some visionOS-specific nodes added:

  • RealityKit PBR: Physically based rendering shader for realistic 3D content
  • RealityKit Unlit: Unlit shader for stylized rendering
  • Geometry Modifiers: Shader nodes that modify surface geometry deformation

Apple contributed Metal shader code generation support to the MaterialX project, released in MaterialX 1.38.7. This means developers can author MaterialX materials in third-party tools on macOS (such as Houdini and Maya) and render them in RealityKit on visionOS.

Application-Layer 3D Support

Quick Look: Quick Look on visionOS supports spatial preview of USDZ files, letting users walk around models and combine them with their surroundings.

Freeform: Apple’s whiteboard app now supports embedded USDZ content, enabling collaboration across macOS, iOS, iPadOS, and visionOS.

Safari Model Element: The new HTML <model> element (enabled in the developer menu) lets web pages embed interactive USDZ models that users can rotate within the page. Apple is working with the W3C Immersive Web Working Group to advance standardization.

Color Management: From sRGB to Display P3

Apple displays primarily use the Display P3 color space, showing approximately 25% more colors than traditional sRGB. RealityKit extends color space management to support the Display P3 wide gamut.

Textures with correctly tagged color spaces show deeper, more saturated tones. This is especially important for real-world colors such as parrot feathers and vivid clothing. Color space tagging also ensures consistent display across devices, avoiding unexpected color shifts.

Industry Collaboration in the USD Ecosystem

Hydra Storm + Metal: Apple collaborated with Pixar to add Metal support to the Hydra Storm renderer. On Apple Silicon’s unified memory architecture, a MacBook Pro can smoothly handle workstation-grade scenes (such as Animal Logic ALab) that originally required 26GB of VRAM.

Blender: Apple worked with the Blender Foundation to significantly improve USD import/export in Blender 3.5, with first-time USDZ format support. A Metal backend was also introduced—Eevee and Cycles final renders are up to twice as fast on Metal, and viewport rendering is up to 4x faster than OpenGL in some scenes.

Autodesk Maya: Apple continues contributing to the Maya USD plugin, improving geometry and material export, animation import, and more.

DaVinci Resolve: Blackmagic Design uses the Storm renderer to add fast USD viewport rendering in Fusion.

RealityKit USD Schema Extensions

RealityKit introduces new USD Schemas on visionOS to support its ECS (Entity Component System):

  • RealityKitComponent: For built-in components
  • RealityKitCustomComponent: For custom Swift components

Structs and dictionaries in Swift can be represented through equivalent RealityKit USD Schemas. This lets custom component data coexist with other USD Prims in the scene hierarchy, editable directly in any USD authoring tool.

RealityKit also extends USD’s spatialAudio with Audio File, Audio File Group, and MixGroup for more immersive audio experiences.

Detailed Content

Safari Model Element

（03:24）

Safari’s new <model> element makes embedding 3D content on web pages as simple as embedding images:

<model src="path/to/model.usdz" alt="A 3D teapot">
  <img src="fallback.jpg" alt="Teapot photo">
</model>

Key points:

  • Usage is similar to <audio>, <video>, and <img> elements
  • Provides an interactive 3D view; users can rotate to inspect objects
  • Available on all Apple platforms that support Safari
  • Must be enabled in the macOS developer menu or settings on other platforms
  • Apple is working with W3C to advance standardization

MaterialX in Reality Composer Pro

（04:51）

Reality Composer Pro’s Shader Graph embeds MaterialX nodes in USD files. The workflow:

  1. Open the Shader Graph editor in Reality Composer Pro
  2. Drag MaterialX nodes to compose shader logic
  3. Use RealityKit PBR nodes for physically based rendering
  4. Use Geometry Modifiers nodes to modify surface deformation
  5. Preview shader effects
  6. USD files automatically include the MaterialX node graph

Key points:

  • MaterialX node graphs travel with USD scenes
  • Third-party tools (Houdini, Maya’s LookdevX) also support MaterialX
  • Apple contributed a MaterialX-to-Metal code generator
  • USD on macOS will also support MaterialX in the future

Display P3 Color Space Advantages

（06:42）

Display P3 vs sRGB:

FeaturesRGBDisplay P3
Gamut rangeStandard25% wider
Red renderingMutedDeeper and more saturated
Green renderingMutedMore lush
Best forGeneral displaysApple displays

Key points:

  • Use Display P3 color profiles when creating textures
  • Ensure texture files are correctly tagged with color space
  • RealityKit supports Display P3 rendering on visionOS
  • Color space tagging ensures consistent display across devices

USD Representation of RealityKit Custom Components

（09:26）

Swift custom component:

struct EngagementComponent: Component {
    var engagementPoints: [String: SIMD3<Float>]
    var defaultEngagement: String
}

Corresponding USD representation:

#usda 1.0
def Xform "Entity"
{
    def RealityKitCustomComponent "EngagementComponent"
    {
        uniform token info:id = "MyApp.EngagementComponent"
        string defaultEngagement = "main"
        dictionary engagementPoints = {
            string main = "(0, 1, 0)"
            string secondary = "(1, 0, 0)"
        }
    }
}

Key points:

  • RealityKitCustomComponent is a USD Schema type
  • info:id identifies the corresponding Swift component type
  • Swift types such as dictionaries and arrays have corresponding USD representations
  • Custom component data coexists with geometry and other Prims in the USD hierarchy
  • Any USD authoring tool can edit this data directly

Core Takeaways

  • What to do: Add 3D product preview to e-commerce websites so users can view products in space.

  • Why it’s worth it: Safari’s <model> element makes embedding 3D content on web pages simple without WebGL development. USDZ format is consistently supported across Apple platforms.

  • How to start: Export product models as USDZ, embed with <model src="product.usdz"> on web pages, and provide a fallback image.

  • What to do: Adopt Display P3 color space in 3D content creation workflows.

  • Why it’s worth it: Apple displays support Display P3; correctly tagged textures show more vivid, realistic colors. Avoid color shifts across devices.

  • How to start: Set project color space to Display P3 in tools like Blender and Maya, preserve color profiles when exporting textures, and verify final results in Reality Composer Pro.

  • What to do: Use USD composition to organize collaboration workflows for large 3D projects.

  • Why it’s worth it: USD references and composition let multiple people work in parallel on different scene parts, with changes automatically syncing to references. Reality Composer Pro directly supports this workflow.

  • How to start: Split scenes into reusable components (characters, props, environments), save each as a separate USD file, compose via references in the main scene, and assign team members to different components.

  • What to do: Create custom components for RealityKit apps and store component data in USD.

  • Why it’s worth it: The RealityKitCustomComponent Schema lets component data be edited and transported with USD scenes, without hardcoding in code.

  • How to start: Define a Swift struct conforming to the Component protocol, store data using the RealityKitCustomComponent Schema in USD files, and load and bind to entities at app launch.

Comments

GitHub Issues · utterances