What's new in camera capture

Highlight

iOS 15 brings five upgrades to camera capture: minimum focus distance reporting, 10-bit HDR video, Control Center video effects (Center Stage / Portrait / Mic Modes), performance optimization best practices, and IOSurface compression.

Core Content

To scan a 20mm QR code, the user brought the phone very close and the code became blurry.This is because the minimum focusing distance of the mobile phone lens is limited. If you get too close, you will not be able to focus.Before iOS 15, developers didn’t know what this distance was and could only guess.NowAVCaptureDeviceMade publicminimumFocusDistanceAttribute, the App can automatically calculate the required zoom factor based on this, and guide the user to retreat to the appropriate distance.

(02:17) Another big feature is 10-bit HDR video.Supported by iPhone 12, it uses the x420 pixel format (10-bit YUV), matches the HLG curve and BT.2020 color gamut, and has built-in EDR highlight recovery.What’s more, Apple automatically inserts Dolby Vision metadata into every frame, and the recorded video is directly compatible with Dolby Vision display devices.

(12:14) The protagonist of this scene is the Control Center video effect.Center Stage, Portrait mode and Mic mode are system-level functions. After the user opens them, all video calling apps will automatically take effect without developers writing any code.But Apple also provides APIs that allow apps to detect state, customize the UI, and even collaborate with other features within the app.

Center Stage uses the 12-megapixel ultra-wide-angle front camera of the M1 iPad Pro to automatically track the people in the frame and always maintain the composition.Portrait mode uses Neural Engine to generate a monocular depth map to simulate the shallow depth of field effect of a large aperture lens.There are three Mic modes: Standard, Voice Isolation (noise reduction), and Wide Spectrum (environmental sound preservation).

(30:43) Finally, iOS 15 introduces IOSurface compression, a lossless in-memory video compression format.Supports iPhone 12 series, Fall 2020 iPad Air, and M1 iPad Pro.If the app uses hardware acceleration pipelines such as Metal and Core Image, turning on compression can significantly reduce memory bandwidth usage.

Detailed Content

Automatically calculate QR code scanning distance

let deviceFieldOfView = self.videoDeviceInput.device.activeFormat.videoFieldOfView

let minSubjectDistance = minSubjectDistanceForCode(
    fieldOfView: deviceFieldOfView,
    minimumCodeSize: 20,
    previewFillPercentage: Float(rectOfInterestWidth)
)

private func minSubjectDistance(
    fieldOfView: Float,
    minimumCodeSize: Float,
    previewFillPercentage: Float
) -> Float {
    let radians = degreesToRadians(fieldOfView / 2)
    let filledCodeSize = minimumCodeSize / previewFillPercentage
    return filledCodeSize / tan(radians)
}

Key points:

• videoFieldOfViewis the horizontal field of view of the camera’s current format
• minimumCodeSizeIs the minimum code size to be scanned, in millimeters
• previewFillPercentageIt is the proportion of the width that the code should occupy in the preview screen.
• calculatedminSubjectDistanceis the minimum distance at which the code can be clearly focused

Automatically zoom when the distance is too close

let deviceMinimumFocusDistance = Float(self.videoDeviceInput.device.minimumFocusDistance)

if minimumSubjectDistanceForCode < deviceMinimumFocusDistance {
    let zoomFactor = deviceMinimumFocusDistance / minimumSubjectDistanceForCode
    do {
        try videoDeviceInput.device.lockForConfiguration()
        videoDeviceInput.device.videoZoomFactor = CGFloat(zoomFactor)
        videoDeviceInput.device.unlockForConfiguration()
    } catch {
        print("Could not lock for configuration: \(error)")
    }
}

Key points:

• minimumFocusDistanceIs a new attribute in iOS 15, unit is meters
• If the target distance is smaller than the minimum focus distance, calculate the zoom factor to allow the user to move back
• must firstlockForConfiguration()to modifyvideoZoomFactor
• The minimum focusing distance of the iPhone 12 Pro Max wide-angle lens is 15cm, and the iPhone 12 Pro is 12cm

Enable 10-bit HDR video format

func firstTenBitFormatOfDevice(device: AVCaptureDevice) -> AVCaptureDevice.Format? {
    for format in device.formats {
        let pixelFormat = CMFormatDescriptionGetMediaSubType(format.formatDescription)

        if pixelFormat == kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange /* 'x420' */ {
            return format
        }
    }
    return nil
}

// Set as the active format
if let tenBitFormat = firstTenBitFormatOfDevice(device: videoDevice) {
    do {
        try videoDevice.lockForConfiguration()
        videoDevice.activeFormat = tenBitFormat
        videoDevice.unlockForConfiguration()
    } catch {
        print("Could not set format: \(error)")
    }
}

Key points:

• x420 format is 10-bit biplanar YUV, video range (16-235)
• In the format list of iPhone 12, each resolution and frame rate corresponds to three formats: 420v, 420f, x420
• 10-bit HDR video supports 720p, 1080p, 4K and 1920x1440 (4:3)
• Apple automatically inserts Dolby Vision metadata for compatibility with Dolby Vision display devices

Detect Control Center video effect status

// Center Stage
let isCenterStageEnabled = AVCaptureDevice.isCenterStageEnabled
let isCenterStageActive = videoDevice.isCenterStageActive

// Portrait
let isPortraitActive = videoDevice.isPortraitEffectActive

// Mic Mode
let preferredMicMode = AVCaptureDevice.preferredMicrophoneMode
let activeMicMode = AVCaptureDevice.activeMicrophoneMode

Key points:

• isCenterStageEnabledIs a class attribute that reflects the switch status of the App in Control Center
• isCenterStageActiveIs an instance attribute, indicating whether the current camera is applying Center Stage
• Center Stage limitations: up to 30fps, maximum output resolution 1920x1440, zoom locked at 1x
• Portrait mode limitations: up to 30fps, maximum resolution 1920x1440, front camera only

Control Center Stage in Cooperative mode

// Set to cooperative mode so both the app and Control Center can control it
do {
    try videoDevice.lockForConfiguration()
    videoDevice.centerStageControlMode = .cooperative
    videoDevice.isCenterStageEnabled = true
    videoDevice.unlockForConfiguration()
} catch {
    print("Could not configure Center Stage: \(error)")
}

// Listen for state changes to keep the UI in sync
videoDevice.addObserver(self, forKeyPath: "isCenterStageEnabled", options: .new, context: nil)

Key points:

• .userMode (default): Only users can switch in Control Center, and the App will throw an exception if it tries to modify it.
• .appMode: Only App can control, Control Center turns gray, not recommended
• .cooperativeMode: Both App and Control Center can control, and the monitoring status needs to be kept synchronized
• FaceTime is a typical use case of cooperative mode

Handle dropped frames

func captureOutput(
    _ output: AVCaptureOutput,
    didDrop sampleBuffer: CMSampleBuffer,
    from connection: AVCaptureConnection
) {
    guard let attachment = sampleBuffer.attachments[.droppedFrameReason],
          let reason = attachment.value as? String else { return }

    switch reason as CFString {
    case kCMSampleBufferDroppedFrameReason_FrameWasLate:
        // Processing is too slow; consider reducing the frame rate or workload
        reduceFrameRate()
    case kCMSampleBufferDroppedFrameReason_OutOfBuffer:
        // Insufficient buffer space; check whether too many buffers are being retained
        releaseHeldBuffers()
    case kCMSampleBufferDroppedFrameReason_Discontinuity:
        // System-level issue, not caused by the app
        break
    default:
        fatalError("A frame dropped for an undefined reason.")
    }
}

Key points:

• alwaysDiscardsLateVideoFramesThe default is true, keep the latest frame and discard the old frame
• If you need to record all frames (such as AVAssetWriter), you should set it to false and ensure real-time performance yourself
• FrameWasLateis the most common reason, dynamic reductionactiveMinVideoFrameDurationcan alleviate
• OutOfBufferIt means that the App holds too many buffers and should be released in time.

Enable IOSurface compression

let videoDataOutput = AVCaptureVideoDataOutput()

// Check whether compressed BGRA is supported
let compressedFormat = kCVPixelFormatType_Lossless_420YpCbCr10BiPlanarVideoRange

if videoDataOutput.availableVideoPixelFormatTypes.contains(compressedFormat) {
    videoDataOutput.videoSettings = [
        kCVPixelBufferPixelFormatTypeKey as String: compressedFormat
    ]
} else {
    // Fall back to an uncompressed format
    videoDataOutput.videoSettings = [
        kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA
    ]
}

Key points:

• IOSurface compression is lossless and supports compression variants of 420v, 420f, x420 and BGRA
• Only valid if the data is completely processed in the hardware pipeline (Metal, Core Image, AVAssetWriter)
• Do not use the CPU to read and write compressed surfaces, the physical memory layout is opaque and may change
• iPhone 12 series, 2020 iPad Air, M1 iPad Pro support this feature

Core Takeaways

1. Intelligent QR code scanning assistant

• What to do: The QR code scanning app automatically detects the size of the QR code, automatically zooms in and prompts the user to go back when the distance is too close.
• Why it’s worth doing:minimumFocusDistanceLet the QR code scanning experience change from “get close and try” to “automatic adaptation”
• How ​​to start: Calculate the minimum distance based on the code size and field of view angle, andminimumFocusDistancecompare, autosetvideoZoomFactor

2. Professional video recording tool

• What: The Camera App supports 10-bit HDR video recording and direct output to Dolby Vision compatible files
• Why it’s worth it: Apple handles Dolby Vision metadata automatically, developers only need to select the x420 format
• How ​​to start: Traversaldevice.formatsFind the x420 format and set it to activeFormat, useAVCaptureMovieFileOutputorAVAssetWriterRecord

3. Video Conference Effect Control Panel

• What to do: Provide switches for Center Stage and Portrait modes in the App, synchronized with Control Center status
• Why it’s worth doing: cooperative mode allows users to control both at the system level and quickly switch within the app
• How ​​to get started: SettingscenterStageControlMode = .cooperative,monitorisCenterStageEnabledchanges, reflecting the current state in the UI

4. Multi-camera live streaming

• What to do: Use the front and rear cameras simultaneously for picture-in-picture live broadcast, and use IOSurface compression to reduce memory usage
• Why is it worth doing: The memory bandwidth of multi-camera sessions is under great pressure, and the compressed surface is processed with zero copy in the Metal pipeline.
• How ​​to start: ConfigurationAVCaptureMultiCamSession, VideoDataOutput uses compressed pixel format, and Metal shader directly reads the compressed surface for synthesis.

5. Adaptive frame rate camera

• What: The camera app dynamically adjusts frame rate based on system stress status to avoid overheating and dropped frames
• Why it’s worth doing:systemPressureStateProvides nominal/fair/serious/critical/shutdown five-level status, so you can take measures in advance
• How ​​to start: MonitoringsystemPressureStateChange, serious level decreasesactiveMinVideoFrameDuration, turn off non-essential functions when critical

Related Sessions

• Capture high-quality photos using video formats — Improve the quality of still photos without modifying the code
• Edit and play back HDR video with AVFoundation — Editing, playback and tone mapping of HDR video
• Improve access to Photos in your app — PHPicker improvements and PhotoKit cross-device synchronization