WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Explore enhancements to visionOS object tracking

Explore enhancements to visionOS object tracking

visionOSObject TrackingARKitSpatial AccessoriesCreate ML
Watch original video

Highlight

Object tracking in visionOS 27 supports high-frame-rate tracking for dynamic handheld objects, metric-space coordinates, and cross-platform iOS deployment. Apple also opens a third-party spatial accessory standard based on LEDs, IMUs, and Bluetooth, so developers can build physical input devices with buttons and haptic feedback.

Core Content

From Static Props to Handheld Tools

Object Tracking in visionOS 2.0 had a serious limitation: it could only track static objects placed on a table. As soon as someone picked the object up, or moved it quickly, tracking was lost.

(02:22) visionOS 27 adds High Frame Rate Tracking. Together with Create ML’s Extended Training Mode, the system can track handheld object motion much more reliably.

The talk uses a medical probe as an example: a doctor moves a handheld probe across a spine model, and the system calculates distances between vertebrae in real time. That kind of workflow was previously impossible because the hand partially occludes the object and static tracking quickly loses the target.

Metric-Space Coordinates: From “Looks Aligned” to “Measured Accurately”

(05:04) Object tracking in visionOS 2.0 had a hidden issue: returned coordinates included display correction. Apple slightly adjusted coordinates so virtual content looked visually aligned with the physical object in the headset. That correction is fine for rendering UI, but fatal for physical measurement.

visionOS 27 adds a Coordinate Space Correction API. Developers can choose between two coordinate spaces:

  • .rendered: Includes display correction and is suitable for rendering virtual content
  • .none: Pure metric-space coordinates, suitable for physical measurement

That distinction turns object tracking from a “visual alignment tool” into a “precision measurement tool.”

Object Tracking Comes to iOS

(05:53) This year, the object tracking API officially comes to iOS. Trained .referenceobject files work across iOS and visionOS, so the same model can be reused on both platforms.

The iOS API is largely the same as visionOS: load reference objects, configure ARWorldTrackingConfiguration, and handle the lifecycle callbacks for ARObjectAnchor.

Third-Party Spatial Accessories: From Consuming to Creating

(07:25) visionOS 26 only supported officially certified spatial accessories such as Logitech Muse and PSVR2 Sense controllers. visionOS 27 opens the standard completely.

Any electronic device that meets these three conditions can become a spatial accessory:

  • LED constellation array for Vision Pro’s cameras to track
  • IMU for orientation and acceleration
  • Bluetooth chip for communicating with Vision Pro

Spatial accessories can track at the display refresh rate with very low latency, and tracking continues through brief hand occlusions. Accessories can also include physical buttons and haptic motors, making interaction feel more real.

The talk demonstrates a steering wheel accessory: place the spatial accessory inside a physical steering wheel, and the digital vehicle aligns precisely with the wheel. When users grip the wheel, they feel as if they are really sitting in the car.

Details

Enable High-Frame-Rate Tracking

(03:50) High-frame-rate tracking is configured per reference object rather than through a global switch.

// Create the reference object configuration
var configuration = ReferenceObject.Configuration()
configuration.highFrameRateTrackingEnabled = true

// Pass the configuration when loading the reference object
let refObjURL = Bundle.main.url(forResource: "flashlight", withExtension: ".referenceobject")!
let refObject = try? await ReferenceObject(from: refObjURL, configuration: configuration)

Key points:

  • ReferenceObject.Configuration() is a new ARKit API on visionOS
  • highFrameRateTrackingEnabled is configured per object, so different objects can use different tracking strategies
  • This setting is decided at runtime and does not require retraining the model

Extended Training Mode

(04:50) Create ML adds Extended Training Mode to improve tracking accuracy and robustness for handheld objects.

Command-line training:

% xrun createml objecttracker \
    --source flashlight.usdz \
    --output flashlight.referenceobject \
    --training-mode extended \
    --all-angles

Key points:

  • --training-mode extended enables extended training, which takes much longer than standard mode
  • Extended Training Mode is recommended together with high-frame-rate tracking
  • The command-line workflow supports training on a remote machine
  • --all-angles means training from every angle

Metric-Space Coordinates

(05:25) The coordinateSpace(correction:) API lets developers choose a coordinate precision strategy.

// Get display-corrected coordinates for rendering virtual content
let renderingPose = myObjectAnchor.coordinateSpace(correction: .rendered)

// Get pure metric-space coordinates for physical measurement
let metricPose = myObjectAnchor.coordinateSpace(correction: .none)

Key points:

  • .rendered returns coordinates after display correction, so virtual content visually aligns with the physical object
  • .none returns absolute physical-world coordinates, unaffected by any display correction
  • Medical probe measurement, industrial inspection, and similar workflows must use .none
  • Do not use .none coordinates directly for rendering, or visual misalignment will appear

Object Tracking on iOS

(06:22) ARKit in iOS 27 fully supports object tracking.

import ARKit
import RealityKit

class ObjectTrackingARSessionDelegate: NSObject, ARSessionDelegate {
    let arView = ARView(frame: .zero)
    var entities: [UUID: AnchorEntity] = [:]

    func start() throws {
        let stationaryObject = try ARReferenceObject(archiveURL:
            Bundle.main.url(forResource: "stationary", withExtension: "referenceobject")!)
        let movingObject = try ARReferenceObject(archiveURL:
            Bundle.main.url(forResource: "moving", withExtension: "referenceobject")!)

        let configuration = ARWorldTrackingConfiguration()
        configuration.detectionObjects = [stationaryObject]   // Lower frame rate, stationary objects
        configuration.trackingObjects = [movingObject]        // High frame rate, moving objects

        arView.session.delegate = self
        arView.session.run(configuration)
    }

    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        for case let anchor as ARObjectAnchor in anchors {
            let entity = AnchorEntity(anchor: anchor)
            entities[anchor.identifier] = entity
            arView.scene.addAnchor(entity)
        }
    }

    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        for case let anchor as ARObjectAnchor in anchors {
            entities[anchor.identifier]?.isEnabled = anchor.isTracked
        }
    }

    func session(_ session: ARSession, didRemove anchors: [ARAnchor]) {
        for case let anchor as ARObjectAnchor in anchors {
            if let entity = entities.removeValue(forKey: anchor.identifier) {
                arView.scene.removeAnchor(entity)
            }
        }
    }
}

Key points:

  • ARReferenceObject and visionOS ReferenceObject use the same .referenceobject file
  • detectionObjects is for mostly stationary objects and tracks at a lower frequency
  • trackingObjects is for moving objects and enables high-frame-rate tracking
  • The three callbacks didAdd, didUpdate, and didRemove manage the full lifecycle of anchors
  • anchor.isTracked controls whether the entity is visible, automatically hiding it when tracking is lost

Discover and Connect Spatial Accessories

(12:26) Spatial accessories are discovered and connected through the GameController framework.

import ARKit
import GameController

// Discover paired spatial accessories
if let device = GCSpatialAccessory.spatialAccessories.first {

    // ARKit automatically parses the .referenceaccessory bundle
    let accessory = try await Accessory(device: device)
    let provider = AccessoryTrackingProvider(accessories: [accessory])
    try await arkitSession.run([provider])
}

// Hot-swap at runtime; no session restart required
try await provider.updateAccessories([newAccessory])

Key points:

  • GCSpatialAccessory.spatialAccessories returns all paired spatial accessories
  • Accessory(device:) automatically parses the .referenceaccessory file in the device
  • AccessoryTrackingProvider works similarly to the object tracking provider
  • updateAccessories supports switching accessories at runtime without interrupting the tracking session
  • Accessory manufacturers need to declare .referenceaccessory as an exported UTType in Info.plist
  • Third-party developers can declare accessory files as imported UTTypes so their apps can run independently

Key Ideas

1. Handheld Medical Measurement Tool

What to build: Build an orthopedic surgery training app where doctors practice drilling positions on a bone model with a tracked handheld probe.

Why it is worth building: High-frame-rate tracking plus metric-space coordinates makes millimeter-level localization for handheld tools possible. Previously, only fixed-table surgical navigation systems could reach this level of precision.

How to start: Train the probe’s reference object with Create ML’s extended mode, enable highFrameRateTrackingEnabled, and use coordinateSpace(correction: .none) to get real physical coordinates for distance calculations.

2. Immersive Racing Simulator

What to build: Embed a spatial accessory in a real steering wheel so the digital race car and the physical wheel rotate in perfect sync.

Why it is worth building: Spatial accessories track at the display refresh rate, have very low latency, and support physical buttons and haptic feedback. That creates a much better steering wheel experience than pure visual tracking.

How to start: Build a first prototype with off-the-shelf hardware kits from DFRobot or MIKROE, discover and connect it through GCSpatialAccessory, and track its pose with AccessoryTrackingProvider.

3. Industrial Assembly Guidance System

What to build: On a factory line, use object tracking to identify parts and overlay assembly step guidance.

Why it is worth building: Now that object tracking has come to iOS, the same reference object can be reused between iPad for on-site guidance and Vision Pro for immersive training.

How to start: Train a part’s reference object from a USDZ model. On iOS, configure detectionObjects for static recognition; on visionOS, enable high-frame-rate tracking for dynamic guidance.

4. Track Any Object with a 3D-Printed Marker

What to build: Attach a 3D-printed tracking marker to everyday objects without existing 3D models, such as surgical instruments or tools, to enable low-cost tracking.

Why it is worth building: You do not need a photorealistic USDZ model. A printed marker attached to the object is enough to track it, dramatically lowering the barrier to object tracking.

How to start: Design a 3D-printed marker with a distinctive texture, train the marker’s reference object with Create ML, and mount it on the object you want to track.

5. Mixed Reality Game with Multiple Cooperative Accessories

What to build: Build an MR game that tracks multiple spatial accessories at once, such as a gun in one hand and a shield in the other.

Why it is worth building: updateAccessories supports adding and removing accessories dynamically at runtime, so players can switch hands or add new devices without restarting the game.

How to start: Connect the primary accessory during initialization and start the session. When the player picks up a second accessory, call updateAccessories to append it to the tracking list, then trigger game actions from accessory button events.

Comments

GitHub Issues · utterances