WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Explore video experiences for visionOS

Explore video experiences for visionOS

Audio & VideoDesignSpatial Computing
Watch original video

Highlight

visionOS 26 uses one QuickTime container, the Apple Projected Media Profile, to fold 180°, 360°, and Wide FOV non-rectilinear video into a single playback pipeline across AVKit, RealityKit, QuickLook, and Safari. It also opens up the Apple Immersive Video authoring chain to developers for the first time.

Core content

To play a fisheye clip from a GoPro on Vision Pro, a developer used to do the work by hand: detect the projection type, write a mesh, correct lens distortion, and decide between window or immersive presentation. Canon EOS VR shoots stereo 180°, Insta360 shoots 360° spherical, action cameras shoot Wide FOV, and each comes with its own metadata convention. The result was that every video app rebuilt the same format-detection and projection pipeline. When a user dropped a file in from the photo library, they often had no idea whether it would play, or how.

visionOS 26 puts this under one QuickTime movie profile: the Apple Projected Media Profile (APMP). It extends the existing spatial video container with fields that describe three non-rectilinear projections — 180° hemispherical, 360° full sphere, and a parametric Wide FOV surface (16:11). The system converts source files from popular cameras — EOS VR, GoPro MAX, Insta360 X5, HERO13, Ace Pro 2 — to APMP on import (16:56), and reads the equirectangular tags from Google Spherical Video v1/v2. AVKit, RealityKit, QuickLook, Safari, and WebKit all support the APMP extensions and immersive playback, with HLS delivery (18:10). The developer’s job shrinks from “write your own projection pipeline” to “call a standard player”.

The other thread is that the Apple Immersive Video authoring chain opens up for the first time (19:25). Each lens on the Blackmagic URSA Cine Immersive camera is calibrated to a parametric projection at the factory; per eye, that gives 8160×7200 at 90fps with a 210° horizontal FOV (20:22). Footage goes into DaVinci Resolve Studio for editing, gets verified in the Apple Immersive Video Utility on macOS or visionOS, and is sliced for HLS by Compressor (21:19). Pro tool developers can also read and write Apple Immersive content programmatically with the new ImmersiveMediaSupport framework on macOS and visionOS 26 (21:51).

Detailed content

The session sorts video into five tiers, each tied to a different screen geometry. Picking the right format starts with understanding the geometry.

2D and 3D video: rectilinear projection on a flat surface. 2D can be embedded inline anywhere in the UI, sitting alongside layout content (01:55). A 3D video embedded inline gracefully degrades to 2D playback (02:09). Stereo viewing requires expanded mode. The Destination Video sample shows fading out of the expanded view and fading back in to docked playback inside a custom Reality Composer Pro environment, with dynamic light spill produced automatically (02:42). visionOS 26 adds a per-frame dynamic mask for 2D and 3D video, so the aspect ratio can change frame by frame without black bars (03:35). The spec lives in the “Rectangular Mask Payload Metadata” document on developer.apple.com.

Spatial video: stereo plus metadata, with both window and immersive modes. The default is window playback, with a glow around the edge; switching to immersive removes the window edge and scales the content to real-world size (04:57). Window mode deliberately insets the image behind the window and blurs the edges — this softens the discomfort caused by one eye getting more cropping than the other (05:38). On the capture side: the Camera app and AVCaptureDevice API on iPhone 15 Pro / 16 / 16 Pro, Vision Pro itself, and the Canon R7 / R50 with the DUAL lens (05:59). visionOS 26 carries the spatial style into QLPreviewController, AVKit, RealityKit, Safari, and WebKit, with HLS support (06:31). On the editing side, Compressor, DaVinci Resolve Studio, and Final Cut Pro all support it.

180° / 360° / Wide FOV: the three APMP projections. A 180° video projects onto a hemisphere in front of the viewer and fills the forward field of view; it is typically stereo (09:37). A 360° video uses an equirectangular projection onto a full sphere with a 2:1 aspect ratio (11:03). 180° uses half-equirectangular, mapping a square frame onto the hemisphere (11:34). A stereo 180° video traditionally packs both eyes side by side into a double-wide frame, but the two eyes look almost the same, which wastes bandwidth.

MV-HEVC removes the redundancy. Apple platforms already have hardware HEVC support, and stereo video uses MV-HEVC: one pixel buffer per eye, both written into the same video track, with the second eye encoded as a delta (12:50). This matters most for streaming.

Wide FOV uses parametric immersive projection. Action cameras like the GoPro HERO13 and Insta360 Ace Pro 2 capture a 120°–180° horizontal field of view, often with a fisheye lens, so straight lines bend (13:51). visionOS 26 takes the focal length, skew, and distortion parameters of each camera lens and builds a curved mesh; the video maps onto that mesh, and the curve cancels out the fisheye so straight lines in the real world stay straight in the view (14:44). This way of describing a lens projection by parameters is called parametric immersive projection (16:11).

APMP auto-conversion: stereo 180° from Canon EOS VR Utility, 360° from GoPro MAX and Insta360 X5, the equirectangular tags from Google Spherical v1/v2, the direct output of the HERO13 and Ace Pro 2, and per-lens slices from 360° cameras — all are recognized as APMP automatically when opened on Vision Pro (16:56). On macOS, the avconvert command line tool now handles offline 180° and 360° to APMP conversion as well (18:00).

APMP playback constraints: APMP supports expanded and immersive modes, but not inline embedded playback (18:21). Immersive playback is sensitive to camera shake (think of a camera strapped to a surfboard), so QuickLook, AVKit, and RealityKit added high-motion detection: when they detect heavy motion, they automatically reduce immersion, and the user can adjust their own sensitivity in Settings (18:46).

Opening up Apple Immersive Video: URSA Cine Immersive shoots 59 MP per eye at 90fps, more than 10 billion pixels per second (20:30). The full pipeline: shoot → edit in DaVinci Resolve Studio → verify in Apple Immersive Video Utility (on both macOS and visionOS) → segment with Compressor → deliver over HLS (21:19). The ImmersiveMediaSupport framework is the programmable read/write entry point for Apple Immersive (21:56). Format-only features include per-shot edge alpha feathering curves (stored as dynamic curves to blend each shot’s edges into a custom backdrop environment), custom backdrop environments, the new Apple Spatial Audio Format, and live preview on Apple Vision Pro (22:13).

Key takeaways

1. Treat APMP as the default import format for video apps

Why: you no longer need to write a metadata parser per camera brand. The system identifies the file on open and converts it to APMP. One player covers all three projections — 180°, 360°, and Wide FOV.

How to start: in your import flow, use the immersive playback APIs of AVKit or RealityKit directly and let the system pick the projection. Fall back to a 2D representation only when you need an inline thumbnail, since APMP does not support inline playback.

2. Add a high-motion settings entry to apps that handle non-rectilinear video

Why: action camera footage is shaky by nature, and immersive playback amplifies the discomfort. The system already auto-degrades through high-motion detection, but some users still want manual control.

How to start: in your playback UI, point the user toward the high-motion sensitivity slider in Settings, and offer a “reduce immersion” toggle so motion-sensitive users do not have to leave the app.

3. Use the ImmersiveMediaSupport framework to wire Apple Immersive into your Pro tools

Why: Apple Immersive is open to developers for the first time. If you build a non-linear editor, a transcoder, or an asset manager, whoever ships read/write support first will catch the demand wave once URSA Cine Immersive lands in studios.

How to start: begin with the ImmersiveMediaSupport framework on macOS and visionOS 26, and pair it with the Authoring Apple Immersive Video and Converting projected video to Apple Projected Media Profile sample projects to build a minimal read/write demo. Start with per-shot edge blend curves, then add the Spatial Audio Format.

4. Add QLPreviewController / AVKit / RealityKit / WebKit entry points to existing spatial video apps

Why: visionOS 26 extends the spatial style across every Apple media framework, which means the same asset can run in a web embed, an AVPlayer control, and a RealityKit entity.

How to start: list every place in the app that plays video, swap your single player for the official component for each scenario, and run the HLS stream through one delivery channel.

Comments

GitHub Issues · utterances