WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Build live production tools for Apple Immersive Video

Build live production tools for Apple Immersive Video

Apple Immersive VideoSMPTE 2110ProResSpatial Audio
Watch original video

Highlight

Apple Immersive Video live production uses the SMPTE 2110 IP transport standard for ProRes-encoded video, ASAF spatial audio, and JSON metadata, enabling lossless recording and playback workflows through AVFoundation, VideoToolbox, and the Immersive Media Support framework.

Core Content

The difference between traditional 2D live production and immersive live production is essentially the gap between “watching a game at home” and “being at the venue.”

Earlier this year, Apple worked with Spectrum SportsNet and the NBA app to deliver the first real-time immersive broadcast of an LA Lakers game on Apple Vision Pro. Viewers could sit in otherwise unreachable courtside seats, watch the real game, and feel the roar of the crowd.

That experience depends on a completely new production pipeline.

Traditional 2D broadcast video is 4K. Immersive video needs 8K stereoscopic resolution, roughly 32 times the size of a traditional signal. The frame rate also needs to double. On the audio side, traditional stereo or 5.1 surround is not enough. Apple Spatial Audio Format (ASAF) can support 64 or more channels, surrounding the viewer with spatial audio.

Those huge data volumes make traditional broadcast tools and transport methods insufficient. Apple redesigned the entire workflow:

Production domain: Cameras, microphones, graphics generators, and other devices capture content, then a video switcher handles live production.

Transport domain: Encoded content is transported to viewers over IP networks in real time.

Traditional broadcasting often uses uncompressed video, but that is not realistic for immersive video because the data volume is too large. Apple chose ProRes as the encoding format. ProRes balances image quality and bandwidth, and Apple Silicon has hardware acceleration optimized for ProRes processing.

The transport layer uses the SMPTE 2110 standard. It is the industry standard for professional media transport over IP, using RTP to carry media across the network. The 2110-22 standard transports compressed video, 2110-30 transports audio, and 2110-41 transports user-defined metadata such as lens calibration data, creative events, and motion data.

The biggest advantage of this standard is a lossless workflow. In traditional 2D production, video encoding, decoding, and re-encoding introduces generational quality loss. With immersive video, the entire pipeline uses ProRes. When recording to a file, the system does not need another encode step; it copies the ProRes frames directly into a MOV container, then reads them back for transport during playback. Original quality is preserved end to end.

Details

Media Format Components

Apple Immersive Live Video (02:53)

Apple Immersive Live Video is composed entirely of streamed ProRes frames. Unlike traditional broadcast cameras that use uncompressed video frames, ProRes compresses the signal to a practical size while preserving high fidelity.

The video resolution is 8K stereoscopic, 32 times traditional 2D broadcast, and the frame rate is twice that of traditional production. These numbers serve one goal: matching human visual acuity so viewers feel as if they are really there.

ASAF Spatial Audio (02:45)

An Apple Spatial Audio Format (ASAF) mix is made up of standard uncompressed PCM audio tracks carrying higher-order ambisonic beds and spatial audio objects. An ASAF mix can contain 64 or more channels, immersing the viewer in a rich spatial audio experience.

Metadata Format (02:41)

Metadata is delivered as a per-frame JSON object containing properties that describe the associated video and audio feeds: lens calibration, creative events, spatial audio behavior, and more.

SMPTE 2110 Transport Standard

2110-22: Compressed Video Transport (03:05)

Immersive ProRes video is transported across the network as a 2110-22 stream, the standard for compressed media over IP. This 2110-22 stream contains both the left and right eyes of the immersive content as two independent essences, but within a single stream.

That means you do not need to pack the frame for each eye into a single image raster, nor do you need a separate IP stream for each eye. This removes the complex management of independent left-eye and right-eye video feeds from the production architecture.

2110-30: Audio Transport (03:46)

ASAF audio is transported across the network as standard 2110-30 streams. These streams contain the higher-order ambisonic and audio object channels that make up the ASAF spatial audio mix.

2110-41: Metadata Transport (04:11)

JSON objects are transported per frame through 2110-41, the standard for user-defined metadata over IP. It carries important metadata, such as lens calibration, creative events, and motion data, in real time alongside the -22 video and -30 audio feeds.

Lossless Recording and Playback

The Advantage of ProRes (04:25)

In a traditional 2D workflow, recording live video to a file involves encoding, decoding, and re-encoding, which causes visual quality loss. In Apple Immersive Video, even a small quality drop can significantly affect the customer experience, especially as generational loss compounds over repeated compression and decompression cycles.

The immersive format solves this problem: everything is ProRes.

Recording to a File (04:47)

Because the real-time media is natively generated as file-friendly ProRes payloads, recording to disk requires no additional encoding or decoding step. The same ProRes frames are copied directly into a MOV file, then read back during playback into a live 2110 stream without being processed.

In practice, this means real-time content can be generated by cameras, transported between devices, recorded to disk, edited, and played out repeatedly with no quality impact throughout the process.

Writing the MOV Video Track (06:47)

Use AVFoundation’s AVAssetWriter to save the video feed into a QuickTime MOV video track:

import VideoToolbox

let compressionProperties: [String: Any] = [
    // ...
    kVTCompressionPropertyKey_ProjectionKind as String: kVTProjectionKind_AppleImmersiveVideo
    // ...
]

Key points:

  • kVTProjectionKind_AppleImmersiveVideo is a new VideoToolbox property
  • It adds the correct VEXU (Video Extended Usage) static metadata to the file
  • This metadata identifies the file as immersive content to other apps
  • The resulting MOV file contains the same unmodified resolution, frame rate, and stereoscopic image data as the real-time stream

Writing the Audio Track (06:30)

The audio feed is saved in the usual way: the uncompressed PCM from the 2110 stream is written directly into the MOV audio track with AVAssetWriter.

Writing the Metadata Track (06:41)

Streamed JSON data is written with AVAssetWriter into a Metadata Box Exchange format (MEBX) track inside the MOV container. Before storage, the streamed JSON data must be deserialized and parsed, then the Immersive Media Support (IMS) framework is used to create lens calibration objects, camera IDs, and other metadata objects. Those objects are written into the MOV in sync with the video and audio.

Immersive Media Support Framework

Introduction to IMS (07:09)

IMS is introduced for the first time in visionOS 26. It supports reading and writing the essential metadata for Apple Immersive Video, and it provides features for previewing content in creative workflows.

Although immersive video and audio are already powered by well-known technologies such as AVFoundation, VideoToolbox, and Core Audio, IMS is a powerful framework built specifically for Apple Immersive Video.

File Playback Workflow (07:49)

In a file playback scenario, every step is reversed. Video, audio, and metadata media types are read directly from tracks in the MOV, then all the same frameworks and libraries retransmit them back into 2110 output streams for broader production use.

Core Ideas

1. Real-Time Immersive Stream Analyzer

What to build: Use AVFoundation to parse ProRes frames from a 2110-22 video stream and monitor quality metrics such as bitrate, frame rate, and stereoscopic disparity in real time, so potential production issues can be found early.

Why it matters: 8K stereoscopic video has 32 times the data volume of traditional 2D. Any quality drop can significantly affect the viewer experience. Real-time monitoring catches issues when they happen, instead of waiting for audience complaints.

How to start: Use AVAssetReader to read the 2110-22 stream, pair it with a VideoToolbox decoder to parse ProRes frames, and extract metadata such as frame rate and bitrate for real-time monitoring. Entry point: AVAssetReader plus a VideoToolbox decoder

2. Metadata Validator

What to build: Use the IMS framework to parse JSON metadata from a 2110-41 stream, validate lens calibration data, check motion data consistency, and make sure the transmitted metadata meets production standards.

Why it matters: Metadata is transported in real time alongside video and audio. If lens calibration data is wrong, the immersive experience breaks immediately, for example through image distortion or inaccurate spatial positioning. Automated validation can block these errors before broadcast.

How to start: Use the metadata parsing APIs in the ImmersiveMediaSupport framework to read the JSON from a 2110-41 stream, then validate key fields such as lens parameters and camera IDs for completeness and numeric range. Entry point: metadata parsing APIs in the ImmersiveMediaSupport framework

3. Instant Replay System

What to build: Build a low-latency instant replay tool that writes a live 2110 stream directly into a MOV file, then immediately reads it back for playback while preserving lossless ProRes quality throughout the process.

Why it matters: Traditional recording requires encoding and re-encoding, losing quality on every loop. Apple Immersive Video uses ProRes end to end, copying ProRes frames directly into a MOV container during recording and reading them back for transport during playback, with zero quality loss.

How to start: Use AVAssetWriter to write ProRes frames from the 2110 stream directly into a MOV video track, write PCM audio into an audio track, and parse JSON metadata through the IMS framework before writing it into a MEBX track. During playback, use AVAssetReader to read each track and reassemble a 2110 stream. Entry point: AVAssetWriter for writing plus AVAssetReader for reading

4. Spatial Audio Monitoring Tool

What to build: Parse ASAF audio in a 2110-30 stream and visualize the spatial layout of 64 or more channels, helping audio engineers adjust mix balance in real time.

Why it matters: An ASAF mix contains 64 or more channels, and traditional stereo monitoring tools cannot show its spatial distribution. A visualization tool lets audio engineers directly see each channel’s spatial position and level, making mix issues easier to find.

How to start: Use AVAudioEngine to receive PCM data from the 2110-30 stream, use Core Audio’s ambisonic decoding capabilities to parse the ASAF format, and render each channel’s spatial position into a visual interface. Entry point: AVAudioEngine plus Core Audio ambisonic decoding

5. Network Latency Monitor

What to build: Monitor transport timestamps in SMPTE 2110 RTP streams, calculate end-to-end latency, and help optimize network configuration and device synchronization.

Why it matters: Immersive live production has strict synchronization requirements. Video, audio, and metadata streams must remain precisely aligned. RTP timestamp drift can cause audio-video desynchronization and directly harm the viewer experience.

How to start: Parse the timestamp fields in RTP packet headers, compare sender and receiver timestamps, and calculate end-to-end latency. Set threshold alerts and notify the production team to adjust network configuration when latency exceeds the limit. Entry point: RTP packet parsing plus timestamp comparison analysis

Comments

GitHub Issues · utterances