WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Immerse your app in Spatial Audio

Immerse your app in Spatial Audio

Audio & VideoSpatial AudioAVFoundationWebKit
Watch original video

Highlight

Apple has added spatial audio support to the AVFoundation and WebKit playback paths. Developers can use multi-channel HLS, spatial format control and audio routing detection to provide media apps with an immersive playback experience with head tracking and bandwidth adaptation.

Core Content

(01:32) In the past, video sound on mobile devices mostly stayed in stereo. The left and right channels in the headphones will move with the head, and the sound will feel like it is stuck in the head. When the user turns his head while watching a movie, the sound field also rotates, and the fixed sound field experience in the theater disappears.

(02:14) Spatial Audio (spatial audio) solves this problem. It uses psychoacoustic technology to generate a virtual sound field. When using headphones that support spatial audio, the system will compare the inertial measurement data in the playback device and headphones to obtain the user’s head posture, and then dynamically adjust the rendering to keep the sound field around the screen.

(02:49) For media services, the most direct entry is multi-channel content. Apple makes it clear that if your HLS (HTTP Live Streaming, HTTP live streaming) variant already references multi-channel audio alternate, after publishing these contents, the App can enable spatial audio by default, and in many cases there is no need to change the code.

(05:39) Multi-channel audio will occupy a higher bit rate. When the network is tight, the system will downgrade the audio to stereo upmix processing, while still retaining a spatial experience. After bandwidth is restored, full multi-channel spatial audio is returned. Developers need to ensure that the volume of stereo and multi-channel renditions is consistent, and provide DRC (Dynamic Range Control, dynamic range control) and dialnorm metadata.

(06:35) If the App needs to control the spatialization strategy, AVFoundation providesAVAudioSpatializationFormats. It can allow mono and stereo spatialization, allow multichannel spatialization, allow both, or can be set to 0 to turn off spatialization.

Detailed Content

Select an audio format that allows spatialization

(06:55) Courtesy of AppleAVAudioSpatializationFormats(Spatialized Audio Format) to describe which source audio can be spatialized. This type isOptionSet, suitable for expressing multiple format combinations.

import AVFoundation

let formats: AVAudioSpatializationFormats = .monoStereoAndMultichannel

print(formats.contains(.monoAndStereo))
print(formats.contains(.multichannel))

Key points:

  • import AVFoundationIntroducing the playback API discussed in this session. -AVAudioSpatializationFormatsis a collection of spatial formats. -.monoStereoAndMultichannelAllows mono, stereo and multi-channel source audio to participate in spatialization. -contains(.monoAndStereo)Checks whether the collection contains mono and stereo spatialization. -contains(.multichannel)Checks whether the collection contains multichannel spatialization.

The corresponding API shape is as follows.

public struct AVAudioSpatializationFormats : OptionSet {

    public init(rawValue: UInt)

    public static var monoAndStereo: AVAudioSpatializationFormats { get }

    public static var multichannel: AVAudioSpatializationFormats { get }

    public static var monoStereoAndMultichannel: AVAudioSpatializationFormats { get }
}

Key points:

  • OptionSetIndicates that it can combine multiple options. -monoAndStereoAllows spatialization of audio from mono and stereo sources. -multichannelAllows spatialization of audio from multi-channel sources. -monoStereoAndMultichannelIt is a combination of the first two types of abilities.

Set the spatialization strategy on the player

(07:21) The spatial format must be set to the playback object. Session named two paths:AVPlayerItemandAVSampleBufferAudioRenderer. The latter also supports this control point starting with the fall 2021 system.

import AVFoundation

let url = URL(string: "https://example.com/movie.m3u8")!
let playerItem = AVPlayerItem(url: url)

playerItem.allowedAudioSpatializationFormats = .multichannel

let player = AVPlayer(playerItem: playerItem)
player.play()

Key points:

  • URLPoint to HLS media resource, example usage.m3u8。
  • AVPlayerItemHosts the media to be played. -allowedAudioSpatializationFormatsSpecifies the source audio type that allows spatialization. -.multichannelIndicates that only multi-channel source audio is allowed to enter spatial processing. -AVPlayeruse thisplayerItemStart playing.

The attribute entries displayed in Session are as follows.

@available(macOS 11.0, *)
var allowedAudioSpatializationFormats: Int32

Key points:

  • allowedAudioSpatializationFormatsis an attribute that controls the spatialization strategy.
  • This attribute is used forAVPlayerItem, the Fall 2021 system is also available forAVSampleBufferAudioRenderer.
  • Passing in 0 can suppress audio spatialization, which comes from the supplementary explanation of the four values ​​​​of session.

Check whether spatial audio is enabled on the current audio route

(08:21) App cannot just look at whether the content has multiple channels. The user’s current audio routing must also support spatial audio, and the user must allow it to be enabled. Starting from iOS 15,AVAudioSessionPortDescriptionsupplyisSpatialAudioEnabled。

import AVFoundation

let session = AVAudioSession.sharedInstance()
let outputs = session.currentRoute.outputs

for output in outputs {
    if #available(iOS 15.0, *) {
        print(output.portName, output.isSpatialAudioEnabled)
    }
}

Key points:

  • AVAudioSession.sharedInstance()Get the audio session of the current app. -currentRoute.outputsReturns the current output route, such as headphones or built-in speakers. -#available(iOS 15.0, *)AlignmentisSpatialAudioEnabledsystem version requirements. -isSpatialAudioEnabledSays two things at the same time: the port has the ability to render spatial audio, and the user is allowed to use it. -portNameIt is convenient to see the current output device during debugging.

The corresponding API shape is as follows.

@available(iOS 6.0, *)
class AVAudioSessionPortDescription : NSObject {

  @available(iOS 15.0, *)
  var isSpatialAudioEnabled: Bool { get }

}

Key points:

  • AVAudioSessionPortDescriptionDescribes the port in the audio session. -isSpatialAudioEnabledAvailable starting in iOS 15.
  • This attribute is read-only and is used by the App to determine the current routing status.

Changes in monitoring space playback capabilities

(08:35) Users can change spatial audio preferences in Control Center or Bluetooth settings. App needs to monitorspatialPlaybackCapabilitiesChangedNotification, and read from the notificationAVAudioSessionSpatialAudioEnabledKey。

import AVFoundation

let center = NotificationCenter.default

if #available(iOS 15.0, *) {
    center.addObserver(
        forName: AVAudioSession.spatialPlaybackCapabilitiesChangedNotification,
        object: AVAudioSession.sharedInstance(),
        queue: .main
    ) { notification in
        let enabled = notification.userInfo?[AVAudioSessionSpatialAudioEnabledKey] as? Bool
        print(enabled as Any)
    }
}

Key points:

  • NotificationCenter.defaultUsed to register system notification observers. -spatialPlaybackCapabilitiesChangedNotificationEmitted when the space playback ability changes. -objectPoint to currentAVAudioSession。
  • queue: .mainLet the callback be executed in the main queue, suitable for updating the UI. -AVAudioSessionSpatialAudioEnabledKeyused fromuserInfoRemove spatial audio enabled status.

The API displayed in Session is as follows.

extension AVAudioSession {
  @available(iOS 15.0, *)
  class let spatialPlaybackCapabilitiesChangedNotification: NSNotification.Name
}

@available(iOS 15.0, *)
let AVAudioSessionSpatialAudioEnabledKey: String

Key points:

  • Notification is defined inAVAudioSessionOn extension.
  • key is a string constant used to read the status carried by the notification.
  • This notification covers scenarios where users modify spatial audio preferences in Control Center and Bluetooth settings.

Declare to the system that the app supports multi-channel content

(09:01) If the App usesAVSampleBufferAudioRendererOr to customize the playback path, you need to tell the system that it can provide multi-channel content. The entrance issetSupportsMultichannelContent(_:)。

import AVFoundation

let audioSession = AVAudioSession.sharedInstance()

if #available(iOS 15.0, *) {
    do {
        try audioSession.setSupportsMultichannelContent(true)
        print(audioSession.supportsMultichannelContent)
    } catch {
        print(error)
    }
}

Key points:

  • AVAudioSession.sharedInstance()Represents the current app’s audio session. -setSupportsMultichannelContent(true)Statement that the app or service can provide multi-channel content.
  • This statement will be used for system prompts to let users know that the space experience can be obtained when network conditions permit and the processing method is enabled. -supportsMultichannelContentRead the current claim status. -do/catchHandle errors that may be thrown by this throwing method.

The API displayed in Session is as follows.

extension AVAudioSession {
  @available(iOS 15.0, *)
  func setSupportsMultichannelContent(_ inValue: Bool) throws
  @available(iOS 15.0, *)
  var supportsMultichannelContent: Bool { get }
}

Key points:

  • Both methods and properties are available starting in iOS 15. -setSupportsMultichannelContent(_:)It is the declaration entry. -supportsMultichannelContentIt is the reading entrance.
  • If the app usesAVPlayer, session indicates that these indications will be managed by the system.

WebKit and system version support range

(09:20) Spatial audio support covers multiple system versions. macOS Catalina, iOS 13, iPadOS 13 already supported through built-in speakersAVPlayerItemand WebKit video tag. macOS Big Sur, iOS 14, and iPadOS 14 add support for head-tracking headphones for AirPods Pro and AirPods Max.

import AVFoundation

let item = AVPlayerItem(url: URL(string: "https://example.com/movie.m3u8")!)
item.allowedAudioSpatializationFormats = .monoStereoAndMultichannel

Key points:

  • AVPlayerItemIt is an important playback portal that supports spatial audio across systems. -.monoStereoAndMultichannelAlign default orientations in iOS 15, iPadOS 15, macOS Monterey, and tvOS 15: Mono, stereo, and multichannel sources can all be spatialized when conditions permit.
  • WebKit’s MSE (Media Source Extensions) path has limited support on Fall 2021 systems.
  • MSE has no interface for customizing the spatialized experience.
  • WebKit is available through the Media Capabilities APIAudioConfigurationDictionary detection spatial audio support.

Core Takeaways

  1. What to do: Add multi-channel HLS playback to the film and television app. Why it’s worth doing: session description, after the HLS variant references multi-channel audio alternate, AVFoundation can enable spatial audio by default. How ​​to start: First organize the multi-channel audio resources in the media library, output the variant playlist according to the HLS Authoring Specification, and then useAVPlayerItemPlay and set as desiredallowedAudioSpatializationFormats。

  2. What to do: Display “Is spatial audio currently available” on the playback page? Why it’s worth doing: The user’s headphone, speaker and control center preferences will affect the actual experience, and the App needs to feedback status to the user. How ​​to start: ReadAVAudioSession.sharedInstance().currentRoute.outputs, check eachAVAudioSessionPortDescriptionofisSpatialAudioEnabled。

  3. What: Update the playback UI when the user switches headphones or modifies spatial audio preferences. Why it’s worth doing: It is recommended that the session observe route changes and listen for notifications of space playback capability changes. How ​​to get started: RegisterAVAudioSession.spatialPlaybackCapabilitiesChangedNotification,useAVAudioSessionSpatialAudioEnabledKeyRead new state and recheck when routing changesisSpatialAudioEnabled。

  4. What to do: Declare multi-channel capabilities for the custom player. Why it’s worth doing: UseAVSampleBufferAudioRendererApps need to actively tell the system that they can provide multi-channel content. How ​​to start: Called before playingtry AVAudioSession.sharedInstance().setSupportsMultichannelContent(true), and passsupportsMultichannelContentVerification status.

  5. What to do: Produce sound field narrative versions for concert, sports and feature videos. Why it’s worth doing: Session’s demo demonstrates the narrative value of spatial audio for wind sounds, ambient sounds, distant calls, flights, and movie scenes. How ​​to start: First produce multi-channel audio for key clips, ensure that the stereo and multi-channel rendition volumes are consistent, supplement DRC and dialnorm metadata, and then publish it using HLS.

Comments

GitHub Issues · utterances