Enhance RAW image processing with Core Image

Highlight

In iOS, iPadOS, macOS, and visionOS 27, Apple introduces the RAW 9 decoding pipeline. It uses a Core ML model on the Apple Neural Engine to handle demosaicing and denoising together. Third-party apps only need to set CIRAWFilter’s decoderVersion to version9 to get a major image-quality upgrade, and new CIImageProcessor APIs provide precise control over tile size and temporary buffer lifetime.

Core Content

Old photos can become sharper

The appeal of RAW is that a photo you capture today can be reprocessed five years later with a better algorithm. Apple started with calibration for 21 cameras in 2006 and has expanded to 784 camera models today. Every pipeline upgrade improves demosaicing, noise reduction, and color. (02:48)

But there was a long-standing pain point: in high-ISO scenes, RAW 8 was still not good enough at noise reduction and color recovery. In ISO 51200 photos, crayon colors could smear together, and small text could become nearly unreadable. Nontraditional sensor layouts such as Fujifilm X-T5 were especially prone to color artifacts. (04:27)

RAW 9 solves this by combining demosaicing and denoising into one tile-based Core ML model that runs on the Apple Neural Engine. The result: sharper text in low-noise photos, more accurate color in high-noise photos, and even restored highlight reflections on crayons. (03:33)

Interactive editing and batch export need different strategies

RAW 9 improves quality, but it costs more performance. When the same photo is rendered repeatedly, Core Image caches intermediate results, so user feedback while dragging an exposure slider remains fast. Export is different: it performs one full-resolution output, where caching only wastes memory. (08:40)

Apple gives two explicit best-practice configurations:

• Interactive editing: Use scaleFactor for downsampling, one CIContext per view, enable cacheIntermediates, and render directly into a Metal-backed view. Add the Extended Virtual Addressing entitlement so the system can allocate more cache memory. (09:39)
• Batch export: Set cacheIntermediates to false, raise memoryLimit from the 256 MB default to 512 MB or 1024 MB, and use CIContext’s heifRepresentation or jpegRepresentation methods for direct output. (10:56)

Mix these configurations up and your app will either stutter or crash.

Details

Enable RAW 9

RAW 9 is not enabled automatically. You need to check and set it manually. (05:50)

let rawFilter = CIRAWFilter(imageURL: rawURL)!

// Check whether the current device supports RAW 9
if rawFilter.supportedDecoderVersions.contains(.version9) {
    rawFilter.decoderVersion = .version9
}

// Query the list of camera models supported by a decoder version
let models = CIRAWFilter.supportedCameraModels(for: .version9)

Key points:

• supportedDecoderVersions returns the decoder versions available for the current RAW file.
• decoderVersion does not default to version9. You must set it explicitly.
• supportedCameraModels(for:) is a class method that returns an array of all camera models supported by the specified version.
• The support list continues to expand through system OTA updates.
• Devices that natively capture DNG, including iPhone, automatically work with RAW 9.

Adjustable parameters in CIRAWFilter

RAW 9 keeps 20 calibrated parameters for user adjustment and deprecates three old ones. (07:31)

Core parameters that remain:

• exposure - Controls overall brightness.
• luminanceNoiseReductionAmount - Controls luminance noise.
• sharpnessAmount - Controls edge sharpening.
• contrastAmount - Controls local contrast near edges.

Parameters that no longer take effect in RAW 9:

• colorNoiseReductionAmount - The Core ML model automatically handles color noise reduction.
• detailAmount - No longer needed.
• moireReductionAmount - No longer needed.

Call isSupported to check whether a property is valid for the current filter instance.

CIContext configuration for batch export

(11:08)

let exportCtx = CIContext(options: [
    .cacheIntermediates: false,
    .memoryLimit: 512
])

Key points:

• cacheIntermediates: false - Export is a one-time operation, so caching is not useful.
• memoryLimit is measured in MB. iOS defaults to 256 MB; increasing it to 512 or 1024 can significantly speed up export.
• Using exportCtx.heifRepresentation(of:) or jpegRepresentation(of:) uses less memory than calling Image IO directly.

Explicit output tile size in CIImageProcessor

When memory is tight, Core Image automatically splits images into smaller pieces. RAW 9 adds the ability for developers to control tile size themselves. (12:23)

import CoreImage

class MyProcessor: CIImageProcessorKernel {
    override class func roi(forInput input: Int32,
                            arguments: [String : Any]?,
                            outputRect: CGRect) -> CGRect { return outputRect }

    override class func process(with inputs: [CIImageProcessorInput]?,
                                arguments: [String : Any]?,
                                output: CIImageProcessorOutput) throws {
        guard let input = inputs?.first,
              let iBuffer = input.pixelBuffer,
              let oBuffer = output.pixelBuffer else { return }

        let iRegion = input.region
        let oRegion = output.region

        // Process pixel data here
    }
}

let extent = inImg.extent
let tileSize = 512.0
var tiles: [CIVector] = []
for y in stride(from: extent.minY, to: extent.maxY, by: tileSize) {
    for x in stride(from: extent.minX, to: extent.maxX, by: tileSize) {
        let tile = CGRect(x: x, y: y,
                          width: min(tileSize, extent.maxX - x),
                          height: min(tileSize, extent.maxY - y))
        tiles.append(CIVector(cgRect: tile))
    }
}

let result = try MyProcessor.apply(withTiledExtent: tiles, inputs: [inImg], arguments: [:])

Key points:

• roi(forInput:arguments:outputRect:) defines the mapping between input and output regions.
• input.region and output.region in the process callback are the actual processing regions allocated by Core Image.
• After manually generating the tile array, pass it through apply(withTiledExtent:inputs:arguments:).
• Tile size can be adjusted based on GPU memory and algorithm requirements; it is not limited to 512x512.

Temporary buffers in CIImageProcessor

When processing many tiles, repeatedly creating and destroying temporary buffers hurts performance. RAW 9 adds temporary PixelBuffers managed by Core Image. (14:24)

import CoreImage

class MyProcessor: CIImageProcessorKernel {
    override class func process(with inputs: [CIImageProcessorInput]?,
                                arguments: [String: Any]?,
                                output: CIImageProcessorOutput) throws {
        guard let input = inputs?.first,
              let srcPixelBuffer = input.pixelBuffer,
              let dstPixelBuffer = output.pixelBuffer else { return }

        // Request a temporary buffer from the Core Image cache pool
        guard let scratch = output.temporaryPixelBuffer(
            identifier: "myScratch",
            format: kCVPixelFormatType_64RGBAHalf,
            width: Int(output.region.width),
            height: Int(output.region.height),
            pixelBufferAttributes: nil
        ) else { return }

        // Step 1: Copy input data to scratch
        // Step 2: Process pixels in scratch
        // Step 3: Copy the result to the output buffer
    }
}

Key points:

• temporaryPixelBuffer(identifier:format:width:height:pixelBufferAttributes:) obtains a buffer from Core Image’s internal cache pool.
• identifier distinguishes multiple temporary buffers and must be unique within the same processor.
• Core Image automatically manages the lifecycle: it reclaims the buffer after the current tile is processed and reuses it for the next tile.
• A typical use case is converting Core Image’s interleaved image data into planar data required by Core ML.

Key Ideas

• Build a RAW browser and lightweight editing app: Use RAW 9’s image-quality improvements and the 20 adjustable CIRAWFilter parameters to build an app focused on fast preview and parameter tuning. The entry point is CIRAWFilter(imageURL:), and the core interaction is real-time preview while dragging exposure and sharpness sliders. (07:15)

• Build a batch RAW transcoding tool: For photographer workflows, create a background tool that exports RAW files to HEIF or JPEG in batches. The key configuration is cacheIntermediates: false + memoryLimit: 1024, combined with direct output through heifRepresentation(of:colorSpace:). (10:56)

• Create custom RAW post-processing filters: Use CIImageProcessorKernel to plug in your own Metal or Core ML algorithm after RAW 9 decoding. Control memory usage for large images with explicit tile sizes, and avoid repeated allocation with temporary buffers. (12:17)

• Add old-photo reprocessing: Scan old RAW files in the user’s photo library, re-decode them with RAW 9, and show quality comparisons. Use supportedCameraModels(for:) up front to decide which photos can benefit. (06:27)

Related Sessions

• Capture photos and video with the Camera app - New Camera app capture capabilities that form a complete workflow with RAW processing.
• Take your photo capture to the next level - Lower-level photo capture APIs for apps that need to control both capture and post-processing.
• Display EDR content with Core Image, Metal, and SwiftUI - RAW 9 recommends rendering directly into a Metal-backed view; this session covers the related foundations.
• SwiftUI graphics - Integrate Core Image rendering results in SwiftUI.
• Metal performance - A deeper look at Metal rendering performance, complementing RAW 9 CIContext tuning.