WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Explore USD tools and rendering

Explore USD tools and rendering

Spatial ComputingGraphics & GamesWWDC 2022
Watch original video

Highlight

This Session is a companion piece to “Understand USD fundamentals” (10129), focusing on the actual use of USD tools and rendering pipelines on Apple platforms. If you already understand the basic concepts of USD (Prim, Layer, Composition), this session tells you what tools to use to create, edit and render USD content.

Core Content

10129 explains what USD is - the concepts of Stage, Layer, Prim, and Composition. This 10141 will then talk about how to do it: what tools to use to convert the model into USDZ, how to verify asset compliance, and how to integrate the Hydra rendering engine into your own app.

Session is divided into two lines. The first one is the command line tool:usdzconvertResponsible for format conversion,usdzcheckResponsible for asset verification. The second is rendering: Hydra is the rendering abstraction layer of USD, and Storm is a rasterization implementation of Hydra. Apple demonstrated how to use Metal to drive Storm and render USD scenes in its own app.

Detailed Content

usdzconvert: Convert 3D model to USDZ

(03:00) Most 3D assets will not start out in USD format. Modeling tools export to OBJ, FBX, GLTF and other formats.usdzconvertThis command line tool is responsible for converting them into USDZ.

% python usdzconvert --help
usdzconvert 0.66
usage: usdzconvert inputFile [outputFile]
                   [-h] [-version] [-f file] [-v]
                   [-path path[+path2[...]]]
                   [-url url]
                   [-copyright copyright]
                   [-copytextures]
                   [-metersPerUnit value]
                   // ...
                   [-diffuseColor           r,g,b]
                   [-diffuseColor           <file> fr,fg,fb]
                   [-normal                 x,y,z]
                   [-normal                 <file> fx,fy,fz]
                   // ...

Key parameters:

  • -metersPerUnit:Set the scene scale unit. This directly corresponds to the Stage metadata mentioned in 10129. If the source model is in centimeters and the target scene is in meters, a wrong setting here will cause the model size to be completely wrong. --diffuseColor、-normal: Material properties can be overridden during conversion. For example, uniformly set basic colors for a batch of models, or specify normal maps. --copytextures: Pack texture files together into USDZ. USDZ is essentially an uncompressed zip, the textures are placed in the package, and no additional path is required when loading.

Tools are installed with the Xcode command line toolkit. Materials, textures, and animations will be preserved as much as possible during conversion, but not all features of the source format can be mapped losslessly.

usdzcheck: Verify USDZ asset compliance

Session emphasizes the verification process: every time you modify USDZ, you should run it againusdzcheck. What this tool checks includes:

  • Is the file structure correct?
  • Is the texture format supported?
  • Is the polycount within a reasonable range?
  • Whether the hierarchical structure meets Apple platform requirements

Verified USDZ files load and render correctly on all Apple platforms (iOS, macOS, watchOS). Session is recommended tousdzcheckIntegrated into the CI pipeline - automatically run verification before asset submission to prevent non-compliant assets from entering the App.

Lighting Metadata: Declare Lighting Preferences in USDA

(09:00) USD assets can declare lighting preferences in metadata. Session shows passingcustomLayerDataSpecify IBL (Image-Based Lighting) version:

// asset.usda
#usda 1.0
(
    customLayerData = {
        dictionary Apple = {
             int preferredIblVersion = 2
        }
    }
)

preferredIblVersion = 2Tells the renderer to use version 2 of the IBL scheme. This type of metadata is written at the Stage level and affects the default lighting of the entire scene. For AR scenes, correctly setting the lighting metadata can prevent the model from “floating” after being placed in the real environment - mismatch between lighting and the surrounding environment is one of the problems most likely to expose a 3D model.

Material system: UsdPreviewSurface

Apple’s USD implementation usesUsdPreviewSurfaceas standard material. This is the basic PBR material defined in the USD specification and supports the following core properties:

  • diffuseColor: Diffuse color -metallic: Metallicity -roughness: Roughness -normal:Normal map

RealityKit automatically rendersUsdPreviewSurfaceConvert to your own material system. This means asset authors don’t need to understand RealityKit’s material model – it’s written in USDUsdPreviewSurface, RealityKit will render correctly.

Build USD + Hydra

(17:50) To render a USD scene in your own app, you need to build USD and Hydra. Session gives the building steps:

// Rosetta is required on Apple Silicon Macs
% arch -x86_64 /bin/zsh

// Download the source code
% git clone https://github.com/PixarAnimationStudios/USD.git

// Build USD + Hydra (using the Xcode generator)
% python3 USD/build_scripts/build_usd.py --generator Xcode --no-python USDInstall

Key parameters:

  • --generator Xcode: Generate Xcode projects instead of Makefiles to facilitate debugging and integration in Xcode. ---no-python: Skip Python bindings and reduce build artifacts. If your app does not require Python scripting capabilities, add this parameter. -USDInstall:Specify the installation directory.

Hydra is USD’s rendering abstraction layer. It defines the interface that the renderer needs to implement, but does not bind specific rendering technologies. Storm is a rasterization implementation of Hydra, and Apple demonstrates how to drive Storm with Metal as the backend.

Load USD Stage

(18:54) Session demonstrates loading USD files through the file selector in macOS App:

// AAPLViewController.mm

- (void)viewDidAppear
{
    NSOpenPanel* panel = [NSOpenPanel openPanel];
    panel.allowedContentTypes = @[UTTypeUSD, UTTypeUSDZ];

    [panel beginWithCompletionHandler:^(NSModalResponse result) {
        if (result == NSModalResponseOK)
        {
            NSURL* url = panel.URLs[0];
            [self->_renderer setupScene:[url path]];
        }
    }];
}

// AAPLRenderer.mm

- (bool)loadStage:(NSString*)filePath
{
    _stage = UsdStage::Open([filePath UTF8String]);
    // ...
}

UsdStage::Open()Opens the USD file and returns a Stage object. This Stage is the Stage mentioned in 10129 - it contains the Composition results of all Layers. Subsequent rendering, querying, and editing are all based on this Stage.

Set scene camera

(19:30) After loading the Stage, you need to set up the observation camera:

// AAPLRenderer.mm

- (void)setupCamera
{
    _viewCamera = [[AAPLCamera alloc] initWithRenderer:self];

    [self calculateWorldCenterAndSize];

    [_viewCamera setDistance:_worldSize];
    [_viewCamera setFocus:_worldCenter];
}

calculateWorldCenterAndSizeTraverse the Stage’s bounding box and calculate the scene center and size. This information is then used to set the camera distance and focus to ensure the entire scene is in view. After this step is completed, users can automatically see the appropriate perspective when opening any USD file.

Set scene lighting

(19:54) Session demonstrates a simple lighting strategy: treat the camera position as the light source position:

// AAPLRenderer.mm

GlfSimpleLight computeCameraLight(const GfMatrix4d& cameraTransform)
{
    GlfSimpleLight light;
    light.SetPosition(GfVec4f(cameraPosition[0], cameraPosition[1], cameraPosition[2], 1));

    return light;
}

This is a simplification - the light is always coming from the direction of the viewer. Sufficient for previewing and debugging scenarios. Production environments often require more complex lighting setups, such as working with the IBL metadata mentioned earlier.

Initialize Hydra engine and render

(20:17) The core rendering process is divided into two steps: initializationUsdImagingGLEngine, and then call draw at each frame:

// AAPLRenderer.mm

- (void)initializeEngine
{
    _engine.reset(new UsdImagingGLEngine(_stage->GetPseudoRoot().GetPath(),
                                         excludedPaths,
                                         SdfPathVector(),
                                         SdfPath::AbsoluteRootPath(),
                                         driver));
}

// AAPLRenderer.mm

- (HgiTextureHandle)drawWithHydraAt:(double)timeCode
                           viewSize:(CGSize)viewSize
{
      _engine->SetCameraState(modelViewMatrix, projMatrix);
      _engine->SetLightingState(lights, _material, _sceneAmbient);

      UsdImagingGLRenderParams params;
      params.clearColor = GfVec4f(0.0f, 0.0f, 0.0f, 0.0f);
      params.frame = timeCode;

      // ...
}

Key points:

  • UsdImagingGLEngineIs a bridge between Hydra and OpenGL/Metal. It converts the USD scene data into a format that Hydra can consume, and then hands it to Storm for rendering. -SetCameraStateandSetLightingStateCamera and lights are updated every frame. This allows the user to rotate the perspective and move the light source. -UsdImagingGLRenderParamsControl rendering parameters.clearColorSet the background color,frameSet timecode (for animation).
  • Apple’s implementation,HgiTextureHandleThe returned texture can be directly passed to the Metal pipeline for subsequent synthesis—for example, superimposing the 3D rendering results onto the AR camera screen.

Quick Look and AR Quick Look

System apps such as Safari, Mail, and Messages support USDZ’s Quick Look preview. When the user clicks the USDZ link, the system pops up a 3D preview window, where the model can be rotated to view, or switched to AR mode to place the model in the real environment.

Developers can useQLPreviewControllerIntegrate the same capabilities into your own apps. If your app only needs to display the USDZ model rather than custom rendering, it is much simpler to use Quick Look directly than integrating Hydra.

Best Practices

1. Put usdzcheck into the CI pipeline

Run every time the USDZ asset is modified.usdzcheck. It can detect format non-compliance and performance issues in advance - these may not be apparent on the development machine, but can cause loading failures or frame drops on low-end devices. Integrate it into CI and automatically verify assets before submission.

2. Textures use Power-of-Two resolution

Texture sizes are set to 256, 512, 1024, 2048. RealityKit’s texture pipeline is specifically optimized for POT resolution. Non-POT textures trigger autoscaling, potentially reducing quality or increasing load times.

3. Model level matching interaction requirements

If the user needs to manipulate a certain part of the model separately (such as rotating a wheel), make sure that part is an independent Prim in USD and not merged with other geometry in the same mesh. RealityKit’s entity level directly reflects USD’s Prim level - the level design determines the interaction granularity.

4. Declare global metadata in assets

Consistent with 10129,metersPerUnit、upAxisShould be written in Stage metadata. Used when converting assetsusdzconvertCorresponding parameter settings instead of temporary corrections in the App code. Metadata is written in the asset, so all consumers can get consistent information.

Core Takeaways

1. Build USD asset CI verification pipeline

  • What: Integrate in CIusdzcheck, automatically verifying format compliance and performance metrics with every asset submission.
  • Why it’s worth doing: Session emphasizes that asset verification is an “easy to skip but serious consequence” step. Non-compliant assets work fine on the development machine, but fail to load or drop frames on low-end devices. Automated verification eliminates manual checks.
  • How ​​to start: Write a simple shell script callusdzcheck, executed in Git pre-commit hook or CI pipeline. Output inspection report, irregularities prevent merging.

2. Make a USD scene previewer

  • What to do: Use Hydra + Metal to build a lightweight USD/USDZ file preview app that supports rotating the viewing angle and switching lighting modes.
  • Why it’s worth doing: Session provides complete integration code: build USD + Hydra, load Stage, set up camera and lights, render to Metal texture. Stringing these together is a usable previewer.
  • How ​​to start: Build USD + Hydra according to the steps of Session and implement itUsdImagingGLEngineInitialize withQLPreviewControllerOr customize the Metal view to display the rendering results.

3. Make a batch format conversion tool

  • What: Based onusdzconvertEncapsulates batch conversion scripts, supports directory-level conversion from OBJ/FBX/GLTF to USDZ, and automatically setsmetersPerUnitand material parameters.
  • Why it’s worth doing: Session demonstratedusdzconvertrich parameters. In actual projects, modelers need to unify the conversion and parameter settings after exporting a large number of models, and it is too inefficient to handle them one by one manually.
  • How ​​to start: Write a Python/Shell script to traverse the directory and call each model fileusdzconvert, read unified from configuration filemetersPerUnit、diffuseColorand other parameters.

4. Make an AR scene material editor

  • What to do: Load USDZ assets in the App and edit them in real timeUsdPreviewSurfaceofdiffuseColor、metallic、roughnessParameters to preview the effect in the AR environment.
  • Why it’s worth doing: Session introducedUsdPreviewSurfaceand IBL lighting metadata. After the material parameters are adjusted, the performance in the AR environment may be completely different from that in the editor. Real-time preview can reduce the number of iterations of repeated export tests.
  • How ​​to start: UseUsdStage::Open()Load assets, read and write via USD APIUsdPreviewSurfaceThe attribute value is combined with the ARKit camera image for composite rendering.
  • Understand USD fundamentals — This is the precursor to 10141, talking about the core concepts of USD: Stage, Layer, Prim, Composition, VariantSet.
  • Bring your world into augmented reality — Demonstrates how to use Object Capture and RealityKit to bring real objects into AR scenes and directly connect with the USD asset pipeline.
  • Discover ARKit 6 — Introducing the platform capabilities of ARKit 6 to help understand the AR scenario of the final service of USD assets.
  • Create parametric 3D room scans with RoomPlan — RoomPlan generates a structured 3D representation, which has similar modeling ideas to USD’s scene hierarchy and asset organization.

Comments

GitHub Issues · utterances