WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Discover generated subtitles and subtitle styles

Discover generated subtitles and subtitle styles

AVFoundationAccessibilityAI SubtitlesAVPlayerLayerMediaAccessibility
Watch original video

Highlight

Apple introduces on-device AI-generated subtitles on iOS 27, macOS 27, tvOS 27, and visionOS 27. During video playback, the system can automatically transcribe speech into subtitles or translate existing subtitles into other languages without any developer-written AI code. A new subtitle style preview API also lets users switch and preview subtitle styles directly inside the playback interface.

Core Ideas

The subtitle problem: content exists, but the language does not match

You have built a video app with content for a global audience. But there is a practical problem: a video may only have subtitles in one language, or no subtitles at all. If users cannot understand the audio, they may simply leave.

In the past, solving this meant either adding multilingual subtitles during production, which is expensive and slow, or integrating third-party speech recognition and translation services into the app, which requires servers, raises privacy concerns, and leaves subtitle rendering for you to handle.

This time, Apple puts on-device AI models directly into AVFoundation’s playback pipeline.

Two modes for AI-generated subtitles

(02:14) The system offers two ways to generate subtitles:

Speech transcription: Source audio is processed by an on-device speech-to-text model to generate subtitles in real time. This is suited to videos that do not have subtitles.

Language translation: Source subtitles, such as English subtitles, are processed by an on-device translation model to generate subtitles in another language, such as Italian. This is suited to videos that have subtitles but lack the user’s target language.

Both modes run locally on device. They do not require network requests, and data does not leave the device.

Zero-code integration, with authored subtitles always preferred

(03:03) The best news is that you do not need to do anything. If the user chooses an AI-generated subtitle option during playback, the system handles it automatically.

The system prioritizes authored subtitles supplied by the content creator. It falls back to AI-generated subtitles only when authored subtitles do not include the language the user needs. The content and timing of authored subtitles are not modified.

Supported devices and scenarios

(04:03) Starting with iOS 27 and macOS 27, English audio can generate English subtitles. tvOS 27 and visionOS 27 support the same capability. Multilingual subtitle translation, from English subtitles into other languages, is available on iOS and macOS.

Supported playback scenarios include:

  • HLS live streams, such as TV channels
  • Video on demand, including movies, series, travel videos, and sports events
  • Local files, such as embedded app videos and downloaded media

There is no restriction on content type. Professionally produced content and user-generated content, such as iPhone videos and social media videos, are both supported.

Subtitle style preview moves from Settings into playback

(05:45) Previously, users who wanted to change subtitle style had to leave the app, go to Settings > Accessibility, adjust the style, and return to see the result. The flow was fragmented and unpleasant.

Now users can switch styles directly in the subtitle menu inside the playback interface and see the result in real time. The system includes several preset styles, and users can create custom styles, such as a bold yellow style with an outline.

AI-generated subtitles are marked with a Sparkle icon in the menu, and translated generated subtitles also carry a “Translated” label, making them easy to distinguish.

Details

Three ways to implement subtitle selection UI

(04:37) Apple provides three ways to integrate subtitle selection, ordered from lowest to highest integration complexity:

AVPlayerViewController on iOS / AVPlayerView on macOS: These built-in players provide complete subtitle selection and player controls. AI-generated subtitles and style preview work with zero code.

AVLegibleMediaOptionsMenuController: This provides only subtitle selection controls, without player controls. It fits apps that already have a custom player UI.

Custom controls: You implement the media selection UI completely yourself and handle all logic manually.

API implementation for subtitle style preview

(07:25) For developers using a custom player, AVPlayerLayer provides an API for directly displaying style previews.

Each subtitle style has a profile ID in the system. The implementation has four steps:

import AVFoundation
import MediaAccessibility

// 1. Get all subtitle style profile IDs in the system
func updateProfileList() {
    subtitleStyleProfileIDs = MACaptionAppearanceCopyProfileIDs() as? [String] ?? []
}

// 2. Show a preview on the player when the user selects a style
func showPreviewStyle(subtitleStyleProfileID: String) {
    // Passing nil for text shows the system's default localized preview text
    // position is an offset from the default position, used to avoid UI controls
    playerLayer.setCaptionPreviewProfileID(subtitleStyleProfileID, position: .zero, text: nil)
}

// 3. Stop the preview after the user confirms the selection
func stopPreviewStyle() {
    // Remove the preview text and restore the originally active subtitles
    playerLayer.stopShowingCaptionPreview()
}

// 4. Set the selected style as the system active style
func setSubtitleStyle(subtitleStyleProfileID: CFString) {
    // This affects every app on the device that supports this feature
    MACaptionAppearanceSetActiveProfileID(subtitleStyleProfileID)
}

Key points:

  • MACaptionAppearanceCopyProfileIDs() returns all system preset and user-defined style IDs as a CFArray, which needs to be bridged to Swift’s [String].
  • After calling setCaptionPreviewProfileID(_:position:text:), the system automatically hides the current subtitles in the video and shows the preview text.
  • position is a CGPoint offset. If the player has controls at the bottom, use a negative Y value to avoid covering the preview text.
  • When text is nil, the system shows localized default preview text, such as “Subtitle Preview.”
  • stopShowingCaptionPreview() removes the preview text and restores the original subtitle display.
  • MACaptionAppearanceSetActiveProfileID() is a global setting that affects the system-wide subtitle style, so call it only after the user explicitly confirms.

A complete user scenario

(08:55) The speaker demonstrates a real scenario: watching an English camping video and wanting Italian subtitles for language practice.

The user opens the subtitle menu, chooses Language, sees multiple subtitle options including authored and AI-generated subtitles, and selects the Italian generated subtitles marked as Translated. The subtitles immediately appear in Italian.

Then the user opens the Style menu, chooses Large Text, sees a larger font in the preview, selects a custom style, and confirms. The subtitles then appear in the new style.

The whole flow stays inside the playback interface and requires no AI-related code from the developer.

Practical Ideas

1. Add multilingual subtitle support to an education app

  • What to build: Let online course videos automatically support subtitles in the student’s native language.
  • Why it matters: AI-translated subtitles help course content reach non-English speakers, and on-device processing means there is no extra server cost.
  • How to start: Make sure your video player uses AVPlayerViewController or integrates AVLegibleMediaOptionsMenuController. The subtitle menu will automatically show AI-generated options.

2. Integrate style preview into a custom player

  • What to build: Add a “Subtitle Style” option to the player’s settings panel so users can adjust styles while watching.
  • Why it matters: Previously users had to leave the app and change system settings. Now they can complete the flow directly in playback.
  • How to start: Use MACaptionAppearanceCopyProfileIDs() to get the style list, and use AVPlayerLayer.setCaptionPreviewProfileID() for real-time preview.

3. Complete accessibility support for a live-streaming app

  • What to build: Support AI-generated subtitles for real-time content such as sports events and news streams.
  • Why it matters: Live content often does not have pre-authored subtitles. AI speech transcription can generate subtitles in real time and help deaf or hard-of-hearing users understand the content. HLS live streams are supported directly.
  • How to start: Check whether your live player is based on AVPlayer. If it is, generated subtitles can take effect automatically.

4. Label AI-generated content in the subtitle menu

  • What to build: Use the system-provided Sparkle icon and “Translated” label so users clearly understand which subtitles are AI-generated.
  • Why it matters: Transparency builds trust. Users know AI subtitles may be less precise than authored subtitles and can set expectations accordingly.
  • How to start: Use the system-provided AVLegibleMediaOptionsMenuController; these labels appear automatically without extra code.

Comments

GitHub Issues · utterances