WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Capture and process ProRAW images

Capture and process ProRAW images

Graphics & GamesPhotos & Camera
Watch original video

Highlight

Apple has integrated ProRAW’s acquisition, library storage, RAW resource reading and Core Image editing into AVFoundation, PhotoKit and Core Image. Developers can process DNG RAW files with computational photography results in their own cameras and photo editing apps.

Core Content

The two common paths for camera apps are both trouble-free. HEIC and JPEG are suitable for immediate display, the file size is small, and the system has done Smart HDR, Deep Fusion or Night Mode fusion. When photography users make adjustments later, the margin is limited. Bayer RAW retains more sensor data and has more room for post-processing, but developers have to deal with demosaicing, noise and rendering speed issues.

ProRAW solves this production chain. It is still placed in a standard Adobe DNG file, and the system frameworks ImageIO and Core Image already have basic support. Its pixels come from linearized, scene-referenced data, which can be generated by fusion of multi-frame exposures. It uses 12-bit RGB lossless compression by default, and the file size is usually between 10MB and 40MB.

For developers, the changes happen in three places. In the shooting stage, AVFoundation (audio and video capture framework) is used to request the ProRAW pixel format; in the storage stage, PhotoKit (photo library framework) is used to save DNG as the main photo resource; in the editing stage, Core Image (image processing framework) is used to loadCIRAWFilter, adjust exposure, white balance, sharpening, local tone mapping, and extract linear scene data for analysis if necessary.

Start with shooting

In the past when making RAW cameras, developers usually had to choose between Bayer RAW and processed JPEG. ProRAW lets apps get a DNG file while retaining the default look and feel of Apple’s computational photography. It can also bring full-size JPEG preview in the file, reducing the cost of library management caused by RAW + JPEG dual resources.

The user may already have RAW files in their photo library. New in iOS 15.smartAlbumRAW, developers can directly access the RAW smart album in the system Photos, and then usePHAssetResourceManagerRead real RAW data.

From editor to implementation

Core Image adds features more suitable for Swift in iOS 15 and macOS 12CIRAWFilterAPI. Instead of just adjusting a RAW filter with a string key, developers can set properties directly. The default look and feel of ProRAW comes from the DNG tag; by turning off these default inputs, you can also get linear scene reference outputs for histograms, analysis, or custom rendering.

Detailed Content

1. Configure the acquisition pipeline that supports ProRAW

(07:52) ProRAW only supports the highest quality photo formats. The simplest way is toAVCaptureSessionUse on.photopreset. Apple recommends enabling it before the session startsphotoOutput.isAppleProRAWEnabled, otherwise it will trigger time-consuming pipeline reconfiguration.

import AVFoundation

private let session = AVCaptureSession()
private let photoOutput = AVCapturePhotoOutput()

private func configureSessionAndPhotoOutput() {
    session.beginConfiguration()
    session.sessionPreset = .photo

    photoOutput.isHighResolutionCaptureEnabled = true
    photoOutput.isAppleProRAWEnabled = photoOutput.isAppleProRAWSupported
    photoOutput.maxPhotoQualityPrioritization = .quality

    session.commitConfiguration()
}

Key points:

  • AVCaptureSession()Create a camera acquisition session.
  • AVCapturePhotoOutput()Responsible for still photo output.
  • session.beginConfiguration()Start batch modification of session configurations.
  • session.sessionPreset = .photoChoose the preset that supports the highest quality photos.
  • photoOutput.isHighResolutionCaptureEnabled = trueEnable high-resolution photo collection.
  • photoOutput.isAppleProRAWEnabled = photoOutput.isAppleProRAWSupportedOnly enable ProRAW if the device supports it.
  • photoOutput.maxPhotoQualityPrioritization = .qualitySet the highest photo quality priority to Quality First.
  • session.commitConfiguration()Submit configuration.

(09:26) After enabling ProRAW,availableRawPhotoPixelFormatTypesProRAW pixel format will be included. useAVCapturePhotoOutput.isAppleProRAWPixelFormat(_:)Differentiate between ProRAW and Bayer RAW.

import AVFoundation

func firstProRAWPixelFormat(from photoOutput: AVCapturePhotoOutput) -> OSType? {
    return photoOutput.availableRawPhotoPixelFormatTypes.first { pixelFormat in
        AVCapturePhotoOutput.isAppleProRAWPixelFormat(pixelFormat)
    }
}

Key points:

  • availableRawPhotoPixelFormatTypesReturns the RAW pixel formats supported by the current output.
  • first { pixelFormat in ... }Find the first ProRAW format from the list.
  • isAppleProRAWPixelFormat(pixelFormat)Determine if the format is Apple ProRAW.
  • function returnsOSType?, returns when not foundnil, the caller should treat it as if the device or format is not supported.

2. Create ProRAW shooting settings

(10:09) When shooting only ProRAW,AVCapturePhotoSettings(rawPixelFormatType:)That’s enough. If you also want to get the processed photos at the same time, you can provideprocessedFormat. session mentioned that this would make the app handle multiple resources and be harder to track in PhotoKit.

import AVFoundation

func makeProRAWSettings(
    photoOutput: AVCapturePhotoOutput,
    proRawPixelFormat: OSType
) -> AVCapturePhotoSettings? {
    guard let processedPhotoCodecType = photoOutput.availablePhotoCodecTypes.first else {
        return nil
    }

    let photoSettings = AVCapturePhotoSettings(
        rawPixelFormatType: proRawPixelFormat,
        processedFormat: [AVVideoCodecKey: processedPhotoCodecType]
    )

    return photoSettings
}

Key points:

  • availablePhotoCodecTypes.firstGet the processed photo encoding format supported by the current output.
  • guardReturn early when no encoding format is available.
  • AVCapturePhotoSettings(rawPixelFormatType:processedFormat:)Create a shot request that contains both ProRAW and processed photos.
  • AVVideoCodecKeySpecifies the type of encoding used for processed photos.
  • returnAVCapturePhotoSettings?, the caller will continue to add thumbnails, previews and quality priorities after getting it.

(10:53) ProRAW supports embedding up to full-size JPEG thumbnails in DNG. Apple suggested using this model in the session to replace the RAW + JPEG dual resources. The gallery will take up less space and the user experience will be clearer.

import AVFoundation

func addEmbeddedThumbnail(
    to photoSettings: AVCapturePhotoSettings,
    device: AVCaptureDevice
) {
    guard let thumbnailPhotoCodecType = photoSettings.availableRawEmbeddedThumbnailPhotoCodecTypes.first else {
        return
    }

    let dimensions = device.activeFormat.highResolutionStillImageDimensions

    photoSettings.rawEmbeddedThumbnailPhotoFormat = [
        AVVideoCodecKey: thumbnailPhotoCodecType,
        AVVideoWidthKey: dimensions.width,
        AVVideoHeightKey: dimensions.height
    ]
}

Key points:

  • availableRawEmbeddedThumbnailPhotoCodecTypes.firstGet the available encodings for DNG embedded thumbnails.
  • device.activeFormat.highResolutionStillImageDimensionsReads the dimensions of a high-resolution photo in the current format.
  • rawEmbeddedThumbnailPhotoFormatSet the format of inline thumbnails.
  • AVVideoWidthKeyandAVVideoHeightKeyMake thumbnails use the specified size.

(11:08) You can also set quality priority and quick preview for this request before shooting. Here’sphotoQualityPrioritizationCan’t go beyond the frontphotoOutput.maxPhotoQualityPrioritization。

import AVFoundation

func finishProRAWSettings(
    _ photoSettings: AVCapturePhotoSettings,
    photoOutput: AVCapturePhotoOutput,
    delegate: AVCapturePhotoCaptureDelegate
) {
    photoSettings.photoQualityPrioritization = .quality

    if let previewPixelFormat = photoSettings.availablePreviewPhotoPixelFormatTypes.first {
        photoSettings.previewPhotoFormat = [
            kCVPixelBufferPixelFormatTypeKey as String: previewPixelFormat
        ]
    }

    photoOutput.capturePhoto(with: photoSettings, delegate: delegate)
}

Key points:

  • photoQualityPrioritization = .qualityLet the quality of this shoot take precedence.
  • availablePreviewPhotoPixelFormatTypes.firstGets one of the available preview pixel formats.
  • previewPhotoFormatRequest a quick preview.
  • capturePhoto(with:delegate:)ProRAW shooting is initiated using the same still photo API.

3. Receive DNG files and customize compression

（11:44）didFinishProcessingPhotowill receiveAVCapturePhoto. If you requested a preview, you can do so frompreviewPixelBufferGet the display image. useisRawPhotoDistinguish between ProRAW and processed photos before usingfileDataRepresentation()Get DNG data.

import AVFoundation

final class ProRAWCaptureDelegate: NSObject, AVCapturePhotoCaptureDelegate {
    var proRAWFileDataRepresentation: Data?
    var proRAWPixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?

    func photoOutput(
        _ output: AVCapturePhotoOutput,
        didFinishProcessingPhoto photo: AVCapturePhoto,
        error: Error?
    ) {
        guard error == nil else {
            return
        }

        if let preview = photo.previewPixelBuffer {
            previewPixelBuffer = preview
        }

        if photo.isRawPhoto {
            guard let fileData = photo.fileDataRepresentation() else {
                return
            }
            guard let pixelBuffer = photo.pixelBuffer else {
                return
            }

            proRAWFileDataRepresentation = fileData
            proRAWPixelBuffer = pixelBuffer
        }
    }
}

Key points:

  • AVCapturePhotoCaptureDelegateReceive photo processing completion callback.
  • guard error == nilFilter collection failed.
  • photo.previewPixelBufferAn uncompressed preview is provided, which can be used to quickly update the interface.
  • photo.isRawPhotoDetermine whether the current callback is a RAW photo.
  • photo.fileDataRepresentation()Returns savable ProRAW DNG file data.
  • photo.pixelBufferReturn RAW pixel data, suitable for continuing the custom processing chain.

(12:52) ProRAW uses 12-bit companding curve lossless encoding by default. passAVCapturePhotoFileDataRepresentationCustomizer, developers can reduce the lossless compression bit depth, or set the quality value below 1 to use 8-bit lossy compression.

import AVFoundation

final class AppleProRAWCustomizer: NSObject, AVCapturePhotoFileDataRepresentationCustomizer {
    func replacementAppleProRAWCompressionSettings(
        for photo: AVCapturePhoto,
        defaultSettings: [String: Any],
        maximumBitDepth: Int
    ) -> [String: Any] {
        return [
            AVVideoAppleProRAWBitDepthKey: min(10, maximumBitDepth),
            AVVideoQualityKey: 1.00
        ]
    }
}

Key points:

  • AVCapturePhotoFileDataRepresentationCustomizerCompression settings used to override ProRAW file data representation.
  • replacementAppleProRAWCompressionSettingsReturns new compression parameters.
  • maximumBitDepthIs the maximum bit depth allowed for the current photo.
  • min(10, maximumBitDepth)Limit lossless compression bit depth to 10-bit or lower available values.
  • AVVideoQualityKey: 1.00Maintain lossless compression.

(13:51) When generating file data, pass in the customizerfileDataRepresentation(with:), you can output DNG according to the new compression settings.

import AVFoundation

func customizedProRAWData(from photo: AVCapturePhoto) -> Data? {
    guard photo.isRawPhoto else {
        return nil
    }

    let customizer = AppleProRAWCustomizer()
    return photo.fileDataRepresentation(with: customizer)
}

Key points:

  • guard photo.isRawPhotoMake sure to only process RAW photos.
  • AppleProRAWCustomizer()Create a compression settings customizer.
  • fileDataRepresentation(with:)Generate ProRAW DNG data by customizer.
  • returnData?, the caller needs to handle when the generation failsnil。

4. Save and read RAW resources with PhotoKit

(15:19) When saving ProRAW to the gallery, use the DNG file as.photoMain resource addedPHAssetCreationRequest. session explicitly recommends using inline preview for ProRAW and deprecating the old RAW + JPEG dual resource model.

import PhotoKit

func saveProRAWToPhotoLibrary(proRawFileURL: URL) {
    PHPhotoLibrary.shared().performChanges {
        let creationRequest = PHAssetCreationRequest.forAsset()
        creationRequest.addResource(
            with: .photo,
            fileURL: proRawFileURL,
            options: nil
        )
    } completionHandler: { success, error in
        if success {
            return
        }

        _ = error
    }
}

Key points:

  • PHPhotoLibrary.shared().performChangesModify the library in the Photo Library transaction.
  • PHAssetCreationRequest.forAsset()Create a new photo resource request.
  • addResource(with:.photo,fileURL:options:)Save ProRAW DNG as the main photo asset.
  • completionHandlerReceive save results and errors.

(15:45) iOS 15 New.smartAlbumRAW. It corresponds to the RAW album in Photos App and contains assets with RAW resources.

import PhotoKit

func fetchRAWSmartAlbum() -> PHFetchResult<PHAssetCollection> {
    return PHAssetCollection.fetchAssetCollections(
        with: .smartAlbum,
        subtype: .smartAlbumRAW,
        options: nil
    )
}

Key points:

  • PHAssetCollection.fetchAssetCollectionsQuery the asset collection.
  • .smartAlbumSpecify the system smart album type.
  • .smartAlbumRAWSpecify RAW smart album.
  • returnPHFetchResult<PHAssetCollection>, the caller can continue to retrieve the assets.

(17:16) Read aPHAssetof RAW data, first enumeratePHAssetResource, just watch.photoand.alternatePhoto, then useUTType.rawImageDetermine the resource type.

import PhotoKit
import UniformTypeIdentifiers

func requestRAWData(for asset: PHAsset, dataHandler: @escaping (Data) -> Void) {
    let resources = PHAssetResource.assetResources(for: asset)

    for resource in resources {
        if resource.type == .photo || resource.type == .alternatePhoto {
            if let resourceUTType = UTType(resource.uniformTypeIdentifier) {
                if resourceUTType.conforms(to: UTType.rawImage) {
                    let resourceManager = PHAssetResourceManager.default()
                    resourceManager.requestData(for: resource, options: nil) { data in
                        dataHandler(data)
                    } completionHandler: { error in
                        _ = error
                    }
                }
            }
        }
    }
}

Key points:

  • PHAssetResource.assetResources(for:)Remove all resources under the asset.
  • .photois the main photo resource,.alternatePhotoIs the RAW resource location in the old RAW + JPEG model.
  • UTType(resource.uniformTypeIdentifier)Convert resource UTI to type object.
  • conforms(to: UTType.rawImage)Determine whether the resource is a RAW image.
  • PHAssetResourceManager.default()Responsible for reading resource data.
  • requestData(for:options:dataReceivedHandler:completionHandler:)Data is returned in chunks and the completion callback is called at the end.

5. Load, edit and export ProRAW with Core Image

(18:36) iOS 15 and macOS 12CIRAWFilterAdded a convenient API for reading preview images. it’s better than going directlyCGImageSourceCreateThumbnailAtIndexCloser to the RAW editing process.

import CoreImage

func previewImage(from url: URL) -> CIImage? {
    let rawFilter = CIRAWFilter(imageURL: url)
    return rawFilter.previewImage
}

Key points:

  • CIRAWFilter(imageURL:)Create RAW filters from ProRAW files.
  • previewImageReturns the preview image within the DNG. -The return value isCIImage?, may be empty if no preview is available for the file.

(19:54) If the App wants to provide editing capabilities, it should createCIRAWFilterand modify the properties. Common adjustments displayed in sessions include exposure, color temperature, tint, sharpening, and local tone mapping.

import CoreImage

func adjustedRawImage(url: URL) -> CIImage? {
    let rawFilter = CIRAWFilter(imageURL: url)

    rawFilter.exposure = -1.0
    rawFilter.neutralTemperature = 6500
    rawFilter.neutralTint = 0.0
    rawFilter.sharpnessAmount = 0.5
    rawFilter.localToneMapAmount = 0.5

    return rawFilter.outputImage
}

Key points:

  • CIRAWFilter(imageURL:)Create modifiable RAW filters.
  • exposure = -1.0Adjust the image brightness to half its original value.
  • neutralTemperature = 6500Sets the scene neutral point temperature in Kelvin.
  • neutralTint = 0.0Set hue offset.
  • sharpnessAmount = 0.5Set the default sharpening strength to half.
  • localToneMapAmount = 0.5Set the local tone map intensity to half.
  • outputImageGenerates an image with these RAW parameters applied.

(21:40) The default look and feel of ProRAW is driven by the DNG tag. To obtain the linear scene reference data, you need to turn off the default look and feel related input and then read it.outputImage。

import CoreImage

func linearSceneReferredImage(url: URL) -> CIImage? {
    let rawFilter = CIRAWFilter(imageURL: url)

    rawFilter.baselineExposure = 0.0
    rawFilter.shadowBias = 0.0
    rawFilter.boostAmount = 0.0
    rawFilter.localToneMapAmount = 0.0
    rawFilter.isGamutMappingEnabled = false

    return rawFilter.outputImage
}

Key points:

  • baselineExposure = 0.0Turn off default baseline exposure.
  • shadowBias = 0.0Turns off the default shadow bias.
  • boostAmount = 0.0Turn off default boost.
  • localToneMapAmount = 0.0Turn off local tone mapping.
  • isGamutMappingEnabled = falseTurn off gamut mapping.
  • outputImageReturns a linear scene reference image that can be used for analysis or continued input into other Core Image filters.

(23:54) After editing is completed, the App canrawFilter.outputImageWritten as 8-bit HEIC. session recommends using Display P3 for output color space.

import CoreImage

func writeHEIC(
    rawFilter: CIRAWFilter,
    ciContext: CIContext,
    to theURL: URL
) throws {
    guard let outputImage = rawFilter.outputImage else {
        return
    }

    try ciContext.writeHEIFRepresentation(
        of: outputImage,
        to: theURL,
        format: .RGBA8,
        colorSpace: CGColorSpace(name: CGColorSpace.displayP3)!,
        options: [:]
    )
}

Key points:

  • rawFilter.outputImageTake the edited RAW output.
  • guardReturn directly when there is no output image.
  • writeHEIFRepresentationWrite out the HEIC file.
  • format: .RGBA8Use 8-bit RGBA output format.
  • CGColorSpace.displayP3Use Display P3 color space.
  • options: [:]Keep the default write-out option.

(25:13) On Mac displays that support Extended Dynamic Range (EDR), Apple recommends usingMTKViewSubclass shows ProRAW. View usagergba16Float，CAMetalLayerOpen the EDR content, and thenrawFilter.extendedDynamicRangeAmountSet to 1.

import CoreImage
import MetalKit

class MyView: MTKView {
    var context: CIContext
    var commandQueue: MTLCommandQueue

    init(frame frameRect: CGRect, device: MTLDevice, context: CIContext, commandQueue: MTLCommandQueue) {
        self.context = context
        self.commandQueue = commandQueue
        super.init(frame: frameRect, device: device)

        colorPixelFormat = MTLPixelFormat.rgba16Float

        if let caml = layer as? CAMetalLayer {
            caml.wantsExtendedDynamicRangeContent = true
        }
    }

    required init(coder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

    func prepareForEDRDisplay(rawFilter: CIRAWFilter) {
        rawFilter.extendedDynamicRangeAmount = 1.0
    }
}

Key points:

  • MTKViewSuitable for passing Core Image output to Metal rendering.
  • CIContextResponsible for Core Image rendering.
  • MTLCommandQueueResponsible for submitting Metal commands.
  • colorPixelFormat = .rgba16FloatMakes the view use half-floating point pixel format.
  • wantsExtendedDynamicRangeContent = trueAllows the Metal layer to display EDR content.
  • extendedDynamicRangeAmount = 1.0Request RAW filter output EDR version.

Core Takeaways

  • What to do: Make a ProRAW camera for photographers. Why it’s worth it: AVFoundation can request ProRAW DNG directly and embed a full-size JPEG preview. How ​​to start: Use.photopreset configurationAVCaptureSession, enablephotoOutput.isAppleProRAWEnabled,chooseisAppleProRAWPixelFormatReturn the pixel format of true before callingcapturePhoto(with:delegate:)。

  • What to do: Make a RAW library organizer tool to automatically find RAW files in the user’s photo library. Why it’s worth doing: iOS 15.smartAlbumRAWAbility to directly locate assets with RAW assets. How ​​to start: UsePHAssetCollection.fetchAssetCollections(with:subtype:options:)Get RAW smart photo albums and use them againPHAssetResource.assetResources(for:)andUTType.rawImageFind DNG data.

  • What to do: Make a lightweight ProRAW color palette. Why it’s worth doing:CIRAWFilterCommonly used adjustment properties such as exposure, color temperature, tint, sharpening, and local tone mapping are exposed. How ​​to get started: Create from file URLCIRAWFilter(imageURL:), mapping the slider value toexposure、neutralTemperature、neutralTint、sharpnessAmount、localToneMapAmount, real-time displayoutputImage。

  • What to do: Make a high dynamic range analysis tool. Why it’s worth doing: After turning off the default look and feel input, you can get a linear scene reference image for histogram or brightness statistics. How ​​to start: PutbaselineExposure、shadowBias、boostAmount、localToneMapAmountSet to 0 to turn offisGamutMappingEnabled, againoutputImagepass toCIFilter.areaHistogram()。

  • What to do: Make an EDR ProRAW viewer for Mac. Why it’s worth it: ProRAW’s dynamic range can be rendered on EDR-capable Mac displays with MetalKit View. How ​​to get started: CreateMTKViewsubclass, settingscolorPixelFormat = .rgba16Float,rightCAMetalLayerOpenwantsExtendedDynamicRangeContent, set before renderingrawFilter.extendedDynamicRangeAmount = 1.0。

Comments

GitHub Issues · utterances