Optimize app power and performance for spatial computing

Highlight

On visionOS, Apple introduced an always-rendering display pipeline where the system continuously renders every frame at 90Hz. Developers must optimize rendering, input, ARKit, audio/video, and memory usage — otherwise apps will be terminated for missing render deadlines.

Core Content

Performance Challenges of Spatial Computing

On iPhone or iPad, when an app isn’t updating, screen content stays static and the GPU can rest. On visionOS, every head turn, hand movement, or blink requires a screen update. The system always renders every frame at 90Hz, whether or not the app is updating content.

This means:

• Rendering cost extends from “dynamic content” to “static content”
• The system runs spatial algorithms handling multiple apps simultaneously
• Thermal management becomes the performance bottleneck, not battery life

(00:39)

Building a Performance Plan

Use Instruments and Xcode Gauges during development. RealityKit Trace is a new Instruments template specifically for spatial computing app performance and power analysis.

Key principles:

• Must profile on real devices — simulators cannot reproduce real workloads
• Test multiple interaction scenarios: audio/video playback, FaceTime, Personas
• Test resource contention when multiple apps run simultaneously
• Use MetricKit and Xcode Organizer to collect production data

(03:34)

Detailed Content

Rendering Pipeline Optimization

The visionOS rendering pipeline has three stages:

1. App layer: Updates content; main thread must finish within deadline
2. Render server: Continuously processes app updates, user input, and spatial data
3. Compositor: Outputs frames to the display at refresh rate (typically 90Hz)

If app rendering takes too long, the render server misses the optimal render latency deadline, causing visual updates to delay by one frame — users feel stutter. Severe rendering stalls cause app termination.

(06:28)

SwiftUI / UIKit Optimization

// Use @Observable instead of @StateObject to reduce unnecessary layout updates
import SwiftUI

@Observable
class TaskStore {
    var tasks: [Task] = []
}

struct TaskListView: View {
    let store: TaskStore
    
    var body: some View {
        List(store.tasks) { task in
            TaskRow(task: task)
        }
    }
}

Key points:

• @Observable provides finer-grained change tracking, reducing unnecessary layout updates
• Avoid semi-transparent effects on overlapping UI views — transparency causes overdraw
• Reduce shadows, blur, masks, and other visual effects that trigger offscreen rendering
• Consider reducing default window size — more pixels means more rendering work

(08:33)

RealityKit 3D Rendering Optimization

import RealityKit
import SwiftUI

struct OptimizedSceneView: View {
    var body: some View {
        RealityView { content in
            // Load complex assets asynchronously to avoid blocking the main thread
            if let entity = try? await Entity(named: "optimized_scene", in: realityKitContentBundle) {
                content.add(entity)
            }
        }
    }
}

Key points:

• Export optimized files from Reality Composer Pro for automatic texture compression
• Merge mesh parts sharing materials to reduce independent mesh count
• Use smaller triangle and vertex counts
• Use Custom material + unlit surface for large or semi-transparent content to avoid expensive lighting
• Create entities ahead of time; control visibility via isEnabled or scene hierarchy instead of frequent create/destroy

(10:58)

Metal Immersive Experience Optimization

import Metal
import CompositorServices

// Query the latest foveation map and pose prediction each frame
func renderFrame(layerRenderer: LayerRenderer) {
    guard let frame = layerRenderer.queryNextFrame() else { return }
    
    // Query input data at the last moment
    let timing = frame.predictTiming()
    let pose = frame.posePrediction(at: timing.presentationTime)
    let foveation = frame.queryFoveationMap()
    
    // Encode GPU work
    encodeRenderWork(pose: pose, foveation: foveation)
    
    // Ensure the compositor receives a new frame every frame
    frame.submit()
}

Key points:

• CompositorServices bypasses the render server, submitting render surfaces directly to the compositor
• Query new foveation map and pose prediction every frame
• Query input data at the last moment before encoding GPU work
• Avoid long frame stalls — the system will terminate the app

(16:56)

Input Performance

Users interact via eyes, hands, voice, and hardware input. Input updates are processed on the main thread and must complete within 8 milliseconds (optimal latency at 90Hz refresh rate).

Optimization tips:

• When adding physics colliders to RealityKit content, prefer static colliders
• Reduce overlapping interactive content to lower hit-testing workload

(18:57)

ARKit Optimization

import RealityKit

// Use "once" tracking mode to avoid continuous tracking overhead
let anchor = AnchorEntity(.head, trackingMode: .once)
entity.components.set(AnchorComponent(.head, trackingMode: .once))

Key points:

• Use "once" tracking mode on AnchorComponent to avoid continuous tracking cost
• Minimize total persistent and transient anchors
• Query ARKit data at the last moment before use
• Disable scene understanding mesh collision data when not needed

(20:17)

Audio/Video Optimization

Spatial audio is on by default; the system processes user position, environment, and sound source info in real time. Three ways to reduce spatial audio workload:

• Reduce simultaneous sound source count
• Reduce moving sound source count
• Shrink soundstage size

Video playback recommendations:

• Use 24 or 30fps video for best performance and power
• Reduce simultaneous video playback and rendering count
• Minimize UI or 3D content updates during video playback

(22:16)

Thermal Management and Memory Pressure

import Foundation

// Listen for thermal state changes and dynamically adjust content
NotificationCenter.default.addObserver(
    forName: ProcessInfo.thermalStateDidChangeNotification,
    object: nil,
    queue: .main
) { _ in
    let state = ProcessInfo.processInfo.thermalState
    switch state {
    case .serious, .critical:
        reduceVisualQuality()
        lowerUpdateRate()
    default:
        break
    }
}

Key points:

• Subscribe to thermalStateDidChangeNotification and dynamically adjust content based on thermal pressure
• Use Xcode thermal inducers to simulate high-heat states
• Reduce offscreen rendering, window count, and media content to lower render memory allocation
• Lower texture resolution, mesh geometry size, and particle count to reduce RealityKit memory

(25:19)

Core Takeaways

• What to do: Build a performance baseline test suite for your visionOS app

  • Why it’s worth it: Always-rendering on spatial computing means performance regressions directly affect user experience or cause app termination
  • How to start: Use RealityKit Trace in Instruments to record frame time, GPU usage, and system power in key scenarios; integrate data into CI

• What to do: Implement dynamic quality degradation

  • Why it’s worth it: Thermal pressure can happen anytime — proactive degradation beats system termination
  • How to start: Listen for thermalStateDidChangeNotification; at .serious state, reduce material complexity, particle count, and render resolution

• What to do: Optimize 3D asset loading strategy

  • Why it’s worth it: Complex assets block the main thread and increase memory pressure
  • How to start: Use Entity(named:in:) to async-load optimized files exported from Reality Composer Pro; instance entities sharing the same asset

• What to do: Implement Metal rendering pipeline for immersive experiences

  • Why it’s worth it: CompositorServices bypasses the render server for direct frame submission control
  • How to start: Use LayerRenderer to query frame timing, pose prediction, and foveation map; ensure every frame submits new content to the compositor

• What to do: Design resource usage friendly to multi-app coexistence

  • Why it’s worth it: visionOS users run multiple apps simultaneously — resource contention is normal
  • How to start: Actively run other apps during testing; monitor render latency and system power in Shared Space; ensure your app maintains 90Hz under resource constraints

Related Sessions

• Meet RealityKit Trace — New Instruments template for analyzing performance and power of spatial computing apps
• Discover Metal for immersive apps — Build immersive experiences with Metal and CompositorServices
• Explore rendering for spatial computing — Deep dive into rendering mechanisms in spatial computing
• Design for spatial input — Optimize interaction design for spatial input