What's new in AVFoundation

Highlight

AVFoundation introduces an asynchronous attribute loading API based on Swift async/await in iOS 15 / macOS 12, replacing the past three-step callback process with one line of code; it also supports injecting timed metadata into custom video composition, and adds the ability to create .itt and .scc subtitle files.

Core Content

You have a video file and want to know its duration and track information. In the old version of AVFoundation, this required a tedious three-step process: calling the asynchronous loading method, checking the status in the callback, and then reading the synchronized properties. The code is verbose and error-prone - many people skip the loading step and read properties directly, causing the main thread to block.

iOS 15 and macOS 12 completely change this experience.

Detailed Content

Asynchronous attribute loading: from three steps to one line

（02:16）

The old API needs to be written like this:

// Old approach: three-step flow
asset.loadValuesAsynchronously(forKeys: ["duration", "tracks"]) {
    var error: NSError?
    guard asset.statusOfValue(forKey: "duration", error: &error) == .loaded else { return }
    guard asset.statusOfValue(forKey: "tracks", error: &error) == .loaded else { return }
    let duration = asset.duration
    let audioTracks = asset.tracks(withMediaType: .audio)
}

（07:16）

The new API is reduced to one line with Swift async/await:

func inspectAsset() async throws {
    let asset = AVAsset(url: movieURL)
    let duration = try await asset.load(.duration)
    myFunction(thatUses: duration)
}

（02:16）

Key points:

• asset.load(.duration)It is asynchronous and does not block the main thread.
• .durationis a compile-time type-safe attribute identifier, and the return type isCMTime
• try awaitUnified handling of both success and failure situations
• If loading fails, an error will be thrown

You can also load multiple properties at the same time:

func inspectAsset() async throws {
    let asset = AVAsset(url: movieURL)
    let (duration, tracks) = try await asset.load(.duration, .tracks)
    myFunction(thatUses: duration, and: tracks)
}

（04:02）

Key points:

• Loading multiple properties at the same time is more efficient, AVAsset can batch handle I/O
• The returned result is a tuple, in the same order as the passed attribute identifiers.
• duration is CMTime, and tracks is [AVAssetTrack]

Check attribute loading status

（04:52）

The new API providesstatus(of:)Method to check property status without waiting:

switch asset.status(of: .duration) {
case .notYetLoaded:
    // Initial state, loading has not been requested yet
    break
case .loading:
    // Loading
    break
case .loaded(let duration):
    // Loaded successfully, use the value directly
    print(duration)
case .failed(let error):
    // Loading failed
    print(error)
}

（04:52）

Key points:

• .notYetLoadedIs the initial state, AVAsset is loaded on demand
• .loadingIndicates that it is being processed asynchronously
• .loadedCarrying associated values, can be used directly
• .failedcarry error message

Asynchronous filtering methods

（06:32）

In addition to loading properties, AVAsset and AVAssetTrack also add new asynchronous filtering methods:

let asset: AVAsset
let track = try await asset.loadTrack(withTrackID: 1)
let audioTracks = try await asset.loadTracks(withMediaType: .audio)
let legibleTracks = try await asset.loadTracks(withMediaCharacteristic: .legible)
let metadata = try await asset.loadMetadata(for: .quickTimeMetadata)
let chapterGroups = try await asset.loadChapterMetadataGroups(bestMatchingPreferredLanguages: ["en-US"])
let audibleGroup = try await asset.loadMediaSelectionGroup(for: .audible)

let track: AVAssetTrack
let segment = try await track.loadSegment(forTrackTime: .zero)
let sampleTime = try await track.loadSamplePresentationTime(forTrackTime: .zero)
let associatedTracks = try await track.loadAssociatedTracks(ofType: .audioFallback)

（06:32）

Key points:

• loadTrack(withTrackID:)Get a specific track by ID
• loadTracks(withMediaType:)Filter by media type (audio/video/subtitles)
• loadTracks(withMediaCharacteristic:)Filter by characteristics (audible/visual/readable)
• loadMetadata(for:)Load metadata in a specific format
• loadMediaSelectionGroup(for:)Load media selection group (multi-language audio tracks, etc.)

Legacy API Migration Guide

（08:09）

Apple plans to deprecate the old sync API in a future release. The migration path is clear:

Old API	New API
loadValuesAsynchronously + statusOfValue+ Sync Properties	try await asset.load(.property)
loadValuesAsynchronously + statusOfValue+ Synchronized filtering methods	try await asset.loadTracks(...)
statusOfValue(forKey:)+ Sync Properties	switch asset.status(of: .property)

If you need to get the loaded value in the synchronization context, you can assert after checking the status:

guard case .loaded(let tracks) = asset.status(of: .tracks) else { return }
// tracks has loaded and can be used synchronously

（09:49）

Key points:

• call againload(.tracks)No duplication of work, cached values ​​will be returned
• butloadCan only be called in an async context
• Used in sync contextstatus(of:)Check and unpack
• After the attribute is loaded, it may also become.failedstate

Timed Metadata in custom video composition

（11:49）

New API allows passing timed metadata in custom video compositions. For example, you have GPS data and want to adjust the picture based on the location information when compositing each frame.

Setup steps:

// Tell the video composition which metadata tracks are relevant
videoComposition.sourceSampleDataTrackIDs = [4, 5]

// Each instruction specifies which tracks are needed
instruction1.requiredSourceSampleDataTrackIDs = [4]
instruction2.requiredSourceSampleDataTrackIDs = [4, 5]

（11:49）

Read metadata in the synthetic callback:

func startRequest(_ request: AVAsynchronousVideoCompositionRequest) {
    for trackID in request.sourceSampleDataTrackIDs {
        let metadata: AVTimedMetadataGroup? = request.sourceTimedMetadata(byTrackID: trackID)
        // Or use sourceSampleBuffer(byTrackID:) to get the original CMSampleBuffer
    }
    // Compose the video frame using metadata
    request.finish(withComposedVideoFrame: composedFrame)
}

（12:44）

Key points:

• sourceSampleDataTrackIDsMust be set at both composition level and instruction level
• sourceTimedMetadata(byTrackID:)Return the parsedAVTimedMetadataGroup
• sourceSampleBuffer(byTrackID:)return originalCMSampleBuffer
• Both setup steps are indispensable, otherwise metadata will not be received

Subtitle file creation

（13:57）

macOS 12 adds new authoring support for two subtitle file formats:

• iTunes Timed Text (.itt): subtitle file
• Scenarist Closed Captions (.scc): closed subtitle file

Core API:

// AVCaption represents a single caption
let caption = AVCaption(text: "Hello", timeRange: timeRange)

// Use AVCaptionConversionValidator to validate compatibility
let validator = AVCaptionConversionValidator()

// Use AVCaptionRenderer to preview caption rendering
let renderer = AVCaptionRenderer()
renderer.render(caption, to: cgContext)

（14:30）

Key points:

• AVCaptionContains attributes such as text, position, style, etc.
• AVAssetWriterInputCaptionAdaptorWrite subtitles to file
• AVAssetReaderOutputCaptionAdaptorRead subtitles from file
• AVCaptionConversionValidatorCheck subtitle compatibility with target format
• The .scc format has strict character budget restrictions, and the validator will recommend adjustments.

Core Takeaways

1. Migrate existing AVAsset property reading code to async/await. The old three-step callback code was verbose and error-prone, the new API does it in one line. Entrance API:try await asset.load(.duration)。

2. Incorporate GPS-driven dynamic compositing into video editing applications. Write GPS data as timed metadata track, read it in a custom compositor and adjust the screen. Entrance API:videoComposition.sourceSampleDataTrackIDs + request.sourceTimedMetadata(byTrackID:)。

3. Add subtitle file generation capability to the video platform. Use AVCaption and AVAssetWriterInputCaptionAdaptor to batch generate .itt or .scc files. Entrance API:AVCaption + AVCaptionConversionValidator。

4. Safely get loaded properties in synchronous code. usestatus(of:)Unwrap after checking status to avoid calling async methods in sync context. Entrance API:asset.status(of: .property)。

5. Load multiple attributes in batches to improve performance. one callload(.duration, .tracks, .metadata)More efficient than three separate calls. Entrance API:try await asset.load(.duration, .tracks)。

Related Sessions

• Evaluate videos with the Advanced Video Quality Tool — AVQT video quality evaluation tool, supports SDR/HDR
• Explore low-latency video encoding with VideoToolbox — VideoToolbox low-latency H.264 hardware encoding
• Edit and play back HDR video with AVFoundation — AVFoundation HDR video editing and playback
• Explore Core Image kernel improvements — Core Image Metal kernel new way of writing