WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Better together: SwiftUI and RealityKit

Better together: SwiftUI and RealityKit

DesignSpatial ComputingSwiftUI
Watch original video

Highlight

visionOS 26 finally fuses SwiftUI and RealityKit: Model3D plays animations directly, RealityKit Entity hosts SwiftUI views and gestures, and coordinate systems flow both ways.

Core Content

To show a 3D robot on visionOS, developers used to pick between Model3D and RealityView. Model3D drops into a SwiftUI layout in one line, but it cannot play animations or emit particles. To swap outfits, add collisions, or fire particles, you switched to RealityView and immediately hit a new problem: RealityView grabs every available pixel by default, so the NameSign next to it gets shoved off to the side.

visionOS 26 clears this path. Apple gave Model3D a new Model3DAsset that reads the animation tracks baked into the model, and a new ConfigurationCatalog that swaps appearances. When you graduate to RealityView, the new .realityViewLayoutBehavior(.fixedSize) makes it hug the model’s bounds the way Model3D did. RealityKit also gained three SwiftUI-flavored components — ViewAttachmentComponent, GestureComponent, PresentationComponent — plus a new ManipulationComponent that lets users grab virtual objects with one or two hands. Entity is now Observable, and so is AnimationPlaybackController, so a SwiftUI view can @Bindable these RealityKit objects and let state flow freely between the two sides.

Detailed Content

Loading animations and a controller into Model3D (03:34). Model3DAsset loads the model file asynchronously and hands it to Model3D for display. The selected animation surfaces as animationPlaybackController, which pairs with a SwiftUI Slider for time scrubbing.

struct RobotView: View {
  @State private var asset: Model3DAsset?
  var body: some View {
    if asset == nil {
      ProgressView().task { asset = try? await Model3DAsset(named: "sparky") }
    } else if let asset {
      VStack {
        Model3D(asset: asset)
        AnimationPicker(asset: asset)
        if let animationController = asset.animationPlaybackController {
          RobotAnimationControls(playbackController: animationController)
        }
      }
    }
  }
}

Key points:

  • Model3DAsset(named:) is an async initializer; await it inside .task.
  • Show a ProgressView while asset == nil, then swap in the real Model3D once loading finishes.
  • asset.animationPlaybackController is created by Model3DAsset after you pick an animation. Do not build one yourself.
  • AnimationPlaybackController is Observable in visionOS 26, so it works as a view model directly.

Wiring SwiftUI controls to a RealityKit controller with @Bindable (04:32).

struct RobotAnimationControls: View {
  @Bindable var controller: AnimationPlaybackController

  var body: some View {
    HStack {
      Button(controller.isPlaying ? "Pause" : "Play") {
        if controller.isPlaying { controller.pause() }
        else { controller.resume() }
      }

      Slider(
        value: $controller.time,
        in: 0...controller.duration
      ).id(controller)
    }
  }
}

Key points:

  • @Bindable var controller plugs a RealityKit class straight into SwiftUI as an observable source.
  • A change to controller.isPlaying invalidates the view, so the button label syncs on its own.
  • $controller.time writes Slider drags back to the controller, giving you scrub control.
  • .id(controller) rebuilds the Slider when the animation switches, so the previous animation’s range does not linger.

Migrating smoothly from Model3D to RealityView (07:25). RealityView takes all available space by default. Add .fixedSize and it sizes itself from the Entity’s visual bounds at first layout.

struct RobotView: View {
  let url: URL = Bundle.main.url(forResource: "sparky", withExtension: "reality")!

  var body: some View {
    HStack {
      NameSign()
      RealityView { content in
        if let sparky = try? await Entity(contentsOf: url) {
          content.add(sparky)
        }
      }
      .realityViewLayoutBehavior(.fixedSize)
    }
  }
}

Key points:

  • Pick one of .fixedSize, .flexible, or .centered: hug the bounds, fill the space, or center inside it.
  • The size is measured once after the make closure returns. Later size changes on the Entity do not stretch the RealityView.
  • These three modifiers only move RealityView’s origin. They never rewrite the Entity’s own position or scale.

ManipulationComponent lets users grab the Entity with two hands (13:18). One call to configureEntity adds ManipulationComponent, CollisionComponent, InputTargetComponent, and HoverEffectComponent at once.

RealityView { content in
  let sparky = await loadSparky()
  content.add(sparky)
  ManipulationComponent.configureEntity(
    sparky,
    hoverEffect: .spotlight(.init(color: .purple)),
    allowedInputTypes: .all,
    collisionShapes: myCollisionShapes()
  )
}

Key points:

  • configureEntity is a static convenience that adds every component interaction needs.
  • hoverEffect controls the visual cue on gaze or hover; here it is a purple spotlight.
  • allowedInputTypes: .all accepts both direct touch and indirect gaze + pinch.
  • collisionShapes defines the grabbable region. Pass nil to auto-generate one.
  • To swap the pickup and drop sounds, set audioConfiguration to .none and subscribe to ManipulationEvents.DidHandOff to play your own.

ViewAttachmentComponent + GestureComponent grow SwiftUI on an Entity (17:10).

struct AttachmentComponentAttachments: View {
  var body: some View {
    RealityView { content in
      let bolts = await loadAndSetupBolts()
      let attachment = ViewAttachmentComponent(
          rootView: NameSign("Bolts"))
      let nameSign = Entity(components: attachment)
      place(nameSign, above: bolts)
      bolts.components.set(GestureComponent(
        TapGesture().onEnded {
          nameSign.isEnabled.toggle()
        }
      ))
      content.add(bolts)
      content.add(nameSign)
    }
    .realityViewLayoutBehavior(.centered)
  }
}

Key points:

  • ViewAttachmentComponent takes any SwiftUI view as its rootView. You no longer have to declare attachments when you build the RealityView.
  • Entity(components: attachment) injects components through the initializer and skips an extra set call.
  • GestureComponent accepts standard SwiftUI gestures, and the coordinates in the callback are in the Entity’s local space by default.
  • When you use GestureComponent, the target Entity still needs InputTargetComponent and CollisionComponent, or events never arrive.

Core Takeaways

  • What to do: Upgrade an existing Model3D into a Model3DAsset-driven player.

    • Why it pays off: A static 3D asset can reuse the animations the artists already exported, and users can swap outfits or scrub the timeline.
    • Where to start: Export the model as a reality file with animations, load it with Model3DAsset(named:), and pair it with a Picker plus a Slider for a minimal playable demo.

  • What to do: Replace any “fill the room” RealityView in your visionOS app with .realityViewLayoutBehavior(.fixedSize).

    • Why it pays off: Most layout glitches come from RealityView grabbing space by default. One modifier lets it sit beside the surrounding SwiftUI controls.
    • Where to start: Search the project for every RealityView. Add .fixedSize to the display-only cases and leave .flexible on the interactive ones.

  • What to do: Add a single ManipulationComponent.configureEntity line to any 3D object you want users to pick up.

    • Why it pays off: Four common components land at once, the system supplies grab and drop sounds, and the cost is almost zero to make a virtual object respond to two-handed input.
    • Where to start: Turn it on for the hero objects in a prototype, then subscribe to ManipulationEvents.DidUpdateTransform to write the pose back into your business state.

  • What to do: Use ViewAttachmentComponent to pin info cards, buttons, and captions directly to an Entity.

    • Why it pays off: Once a SwiftUI view rides along as a component, it follows the Entity through scene management, gesture handling, and Observable state.
    • Where to start: Convert the nameSign and tooltip you wrote with RealityViewAttachments into ViewAttachmentComponent, then add GestureComponent to make them tappable.

Comments

GitHub Issues · utterances