WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
What’s new in RealityKit

What’s new in RealityKit

Graphics & GamesSpatial Computing
Watch original video

Highlight

RealityKit 2025 exposes ARKit data straight to apps through AnchorStateEvents, and a single line of ManipulationComponent.configureEntity() covers grabbing, rotating, and two-hand handoff of 3D objects.

Core content

To build a small game that sits on a desk, lets you reach out and grab things, and gets correctly occluded by real objects, a developer used to juggle the ARKit session, the RealityKit scene, custom gesture recognition, and physics state changes all at once. Glue code easily outweighed business code. This year RealityKit folds these steps into the framework itself.

Laurence ties all the new features together with a spatial puzzle game: a treasure chest is anchored to a desk, the player grabs nearby props to find a key, and once found the chest opens with a small burst of fireworks. The capabilities used in the game match the RealityKit 2025 changelog one for one — SpatialTrackingSession produces AnchorStateEvents directly, ManipulationComponent takes over grab interactions, SceneUnderstandingFlags brings the real room mesh into physics, EnvironmentBlendingComponent lets virtual objects be occluded by real static surroundings, MeshInstancesComponent uses GPU instancing to render decorations efficiently, and ImagePresentationComponent and VideoPlayerComponent handle spatial photos and immersive video. RealityKit also adds tvOS support this year, covering every generation of Apple TV 4K.

Details

ARKit data flows straight into RealityKit. Before, you had to run an ARKit session and a RealityKit scene side by side. Now you start tracking with SpatialTrackingSession, describe what kind of anchor to look for with AnchorEntity, and once it hits, read the raw ARKit data through AnchorStateEvents.DidAnchor (04:33).

// Set up SpatialTrackingSession
@State var spatialTrackingSession = SpatialTrackingSession()

RealityView { content in
    let configuration = SpatialTrackingSession.Configuration(
        tracking: [.plane]
    )
    if let unavailableCapabilities = await spatialTrackingSession.run(configuration) {
        // Handle errors
    }
}

Key points:

  • SpatialTrackingSession.Configuration declares plane-only tracking with tracking: [.plane]. Turning capabilities on and off as needed cuts power use.
  • run(configuration) is async. The return value unavailableCapabilities lists what the current device cannot do, and the developer should degrade gracefully there.
  • The session must start inside the RealityView closure so it shares a lifetime with the current immersive scene.

Once an anchor event arrives, you can cast straight to ARKit’s PlaneAnchor and use the native transform (05:48):

didAnchor = content.subscribe(to: AnchorStateEvents.DidAnchor.self) { event in
    guard let anchorComponent =
        event.entity.components[ARKitAnchorComponent.self] else { return }
    guard let planeAnchor = anchorComponent.anchor as? PlaneAnchor else { return }

    let worldSpaceFromExtent =
        planeAnchor.originFromAnchorTransform *
        planeAnchor.geometry.extent.anchorFromExtentTransform

    gameRoot.transform = Transform(matrix: worldSpaceFromExtent)
    // Add game objects to gameRoot
}

Key points:

  • event.entity.components[ARKitAnchorComponent.self] is a new bridging component in RealityKit that carries the raw ARKit anchor.
  • anchorComponent.anchor has type ARAnchor. Cast it to PlaneAnchor, ImageAnchor, or another concrete type as needed.
  • originFromAnchorTransform * anchorFromExtentTransform aligns the game root with the center of the plane extent, so models do not end up half floating in the air.

ManipulationComponent takes over grabbing. A single ManipulationComponent.configureEntity() call lets any entity support grab, rotate, and even two-hand handoff (07:38):

extension Entity {
    static func loadModelAndSetUp(modelName: String,
                                  in bundle: Bundle) async throws -> Entity {
        let entity = // Load model and assign PhysicsBodyComponent
        let shapes = // Generate convex shape that fits the entity model

        ManipulationComponent.configureEntity(entity, collisionShapes: [shapes])
        var manipulationComponent = ManipulationComponent()
        manipulationComponent.releaseBehavior = .stay
        entity.components.set(manipulationComponent)
        return entity
    }
}

Key points:

  • configureEntity attaches InputTargetComponent, CollisionComponent, HoverEffectComponent, and ManipulationComponent in one shot, so you skip the manual setup.
  • releaseBehavior = .stay keeps the object where it was released. The default flies it back to the start, which does not suit a puzzle game.
  • The grab moment should switch physics modes through ManipulationEvents (09:28): set kinematic on grab, set dynamic on release so gravity takes over.

Scene Understanding joins the physics simulation. Add sceneUnderstanding: [.collision, .physics] to SpatialTrackingSession.Configuration (10:52) and the real room mesh enters the physics world. An object that drops to the floor lands on the actual floor.

EnvironmentBlendingComponent handles static occlusion. preferredBlendingMode: .occluded(by: .surroundings) (11:56) lets real static objects occlude virtual entities precisely, while dynamic things like people and pets do not. An entity with this component is treated as a background layer and is always drawn behind other virtual objects.

MeshInstancesComponent for bulk decorations. One entity plus a set of transform matrices renders hundreds of copies of the same model (14:20). Compared with cloning entities, it saves a lot on memory use and draw calls. LowLevelInstanceData(instanceCount:) hands back a writable transform buffer.

ImagePresentationComponent handles three image kinds. A plain 2D image, a spatial photo, and a spatial scene generated from a 2D image. Before setting desiredViewingMode, check with availableViewingModes.contains(.spatialStereo) (17:57), or it falls back to 2D. A spatial scene is generated by calling generate() on ImagePresentationComponent.Spatial3DImage, with the same result as the system Photos app.

VideoPlayerComponent extensions. Full spatial style for spatial video, APMP 180/360/Wide-FOV, Apple Immersive Video, and automatic comfort tuning.

Load Entity straight from Data. Entity(from: data) lets a model be built from a network stream (23:35), so you no longer have to write to disk first and load from there.

Takeaways

  1. Make ARKit-into-RealityKit the default opening move. When you start a new spatial project, use SpatialTrackingSession + AnchorStateEvents instead of a hand-managed ARKit session. Why bother: one less parallel state to track, and anchor lifetimes line up naturally with entity lifetimes. How to start: replace each ARSession.run with SpatialTrackingSession.run and subscribe to DidAnchor, WillUnanchor, and DidFailToAnchor.

  2. Replace hand-rolled drag gestures with ManipulationComponent. Why bother: two-hand handoff, inertia, and the hover visual come free; rolling them yourself is hundreds of lines of gesture code. How to start: find every DragGesture and SpatialTapGesture that handles 3D objects and swap them for ManipulationComponent.configureEntity(entity, collisionShapes:). Then switch PhysicsBodyComponent.mode with ManipulationEvents.WillBegin and WillEnd.

  3. Turn on the Scene Understanding collision and physics flags. Why bother: a virtual object can really sit on a real desk, and that is more reliable than detecting planes by hand and then doing collision. How to start: add sceneUnderstanding: [.collision, .physics] to SpatialTrackingSession.Configuration, and give anything that should fall a PhysicsBodyComponent(mode: .dynamic).

  4. Use MeshInstancesComponent for scene decoration. Why bother: cloning entities makes draw calls climb fast on visionOS; instancing keeps the frame rate noticeably steadier at dozens or hundreds of instances. How to start: turn repeated Entity.clone(recursive:) into MeshInstancesComponent and write transform matrices in bulk with LowLevelInstanceData.withMutableTransforms.

  5. Check availableViewingModes before setting desiredViewingMode. Why bother: a spatial photo does not always support every viewing mode, and a blind set falls back silently. How to start: add a availableViewingModes.contains(...) guard wherever ImagePresentationComponent is initialized.

Comments

GitHub Issues · utterances