WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Discover Swift enhancements in the Vision framework

Discover Swift enhancements in the Vision framework

WWDC 2024VisionSwiftComputer Vision
Watch original video

Highlight

Vision is Apple’s computer vision framework, offering 31 image analysis requests including face detection, text recognition, body pose estimation, and barcode scanning. This year Apple redesigned the framework with Swift-native APIs supporting async/await, throws, and Swift 6 strict concurrency checking (01:09).

Core Content

The Vision framework has long been the go-to tool for computer vision on Apple platforms. It detects faces and facial landmarks (eyes, nose, mouth), supports text recognition in 18 languages (including Korean, Swedish, and Chinese), and tracks body pose and hand trajectories (00:24).

But the old API had a clear problem: Objective-C-based completion handler design clashed with Swift’s modern async patterns. Code was full of nested callbacks and NSError handling, and didn’t support Swift 6 concurrency safety checks (11:13).

This year Apple completely rewrote Vision’s Swift API. Three core rules: remove the VN prefix, remove completion handlers, replace callbacks with async/await. Result: code drops from 10 lines to 6, with type safety and concurrency safety (12:11).

Beyond API modernization, Apple solved two long-standing pain points: normalized coordinate conversion (new toImageCoordinates() API), and streaming result handling when executing multiple requests simultaneously (performAll returns AsyncStream) (05:37, 08:22).

Two new features round it out: image aesthetics scoring (CalculateImageAestheticsScoresRequest, evaluating photo quality and memorability), and holistic body pose (set detectsHands = true on DetectHumanBodyPoseRequest to detect both body and hand pose simultaneously) (14:51, 15:21).

Detailed Content

Vision’s core concept is simple: ask an image a question (request), get an answer (observation). The framework provides 31 request types covering image classification, text recognition, barcode detection, body pose estimation, and more (02:59).

Basic usage: scanning barcodes

The new API’s minimal style in three lines:

// （[04:06](https://developer.apple.com/videos/play/wwdc2024/10163/?time=246)）
let request = DetectBarcodesRequest()
let barcodeObservations = try await request.perform(on: image)
// barcodeObservations contains each detected barcode
  • Line 1: Create request—no more VN prefix
  • Line 2: Execute with try await; results return directly, no completion handler
  • Line 3: Process observations; each observation represents one detected barcode

To specify barcode types for better performance (such as EAN-13 only):

// （[06:38](https://developer.apple.com/videos/play/wwdc2024/10163/?time=398)）
request.symbologies = [.ean13]

Coordinate conversion

Vision’s coordinate system is normalized (0 to 1) with origin at bottom-left. SwiftUI’s origin is top-left. The new API provides toImageCoordinates():

// （[05:42](https://developer.apple.com/videos/play/wwdc2024/10163/?time=342)）
let boundingBox = observation.boundingBox
let convertedBox = boundingBox.toImageCoordinates(
    imageSize: image.size,
    origin: .upperLeft
)
  • boundingBox: Normalized coordinate box from Vision
  • imageSize: Original image dimensions
  • origin: Output coordinate origin; upperLeft adapts to SwiftUI and similar frameworks

Batch requests: execute together

For multiple simultaneous requests (such as barcodes and text), use ImageRequestHandler:

// （[07:36](https://developer.apple.com/videos/play/wwdc2024/10163/?time=456)）
let handler = ImageRequestHandler(image: image)
let (barcodeResults, textResults) = try await handler.perform(
    DetectBarcodesRequest(),
    RecognizeTextRequest()
)
  • ImageRequestHandler: Image container for batch request execution
  • perform: Parameter pack syntax; pass any number of requests
  • Return value: One result per request; unpack with tuple

Batch requests: streaming

The above method waits for all requests. For immediate processing as each completes, use performAll:

// （[08:44](https://developer.apple.com/videos/play/wwdc2024/10163/?time=524)）
let handler = ImageRequestHandler(image: image)
for await result in try await handler.performAll(
    DetectBarcodesRequest(),
    RecognizeTextRequest()
) {
    switch result {
    case .barcode(let observations):
        // Barcode detection finished; handle it immediately
        handleBarcodes(observations)
    case .text(let observations):
        // Text recognition finished; handle it immediately
        handleText(observations)
    }
}
  • performAll: Returns AsyncStream; pushes result as each request completes
  • for await: Standard Swift Concurrency stream processing syntax
  • switch: Dispatch by result type

Concurrency optimization: processing multiple images

For large image batches, use TaskGroup for parallel execution but limit concurrency to control memory:

// （[10:18](https://developer.apple.com/videos/play/wwdc2024/10163/?time=618)）
await withTaskGroup(of: Void.self) { group in
    var imageIterator = images.makeIterator()
    var activeTasks = 0
    let maxConcurrency = 5

    while activeTasks < maxConcurrency, let nextImage = imageIterator.next() {
        group.addTask {
            await generateThumbnail(from: nextImage)
            activeTasks -= 1
        }
        activeTasks += 1
    }
}
  • withTaskGroup: Swift Concurrency task group
  • maxConcurrency: Limit simultaneous tasks; Vision recommends around 5 to reduce memory usage
  • addTask: Add concurrent task; each processes one image

Migrating old code

Three-step migration:

// Old API
// （[12:32](https://developer.apple.com/videos/play/wwdc2024/10163/?time=752)）
let handler = VNImageRequestHandler(image: image)
let request = VNDetectBarcodesRequest()
request completionHandler = { request, error in
    guard let observations = request.results else { return }
    // Process the result
}
try handler.perform([request])

// New API
let request = DetectBarcodesRequest()
let observations = try await request.perform(on: image)
// Process the result
  • Step 1: Remove VN prefix (VNDetectBarcodesRequest → DetectBarcodesRequest)
  • Step 2: Remove completion handler; use async/await
  • Step 3: Remove VNImageRequestHandler; use request.perform(on:) directly

Image aesthetics scoring

New API evaluates image quality:

// （[14:54](https://developer.apple.com/videos/play/wwdc2024/10163/?time=894)）
let request = CalculateImageAestheticsScoresRequest()
let observation = try await request.perform(on: image)
print(observation.overallScore) // -1 to 1; higher is better
print(observation.isUtility) // true means a utility image, such as a screenshot or receipt
  • overallScore: Overall quality score, range -1 to 1
  • isUtility: Whether it’s a utility image (technically fine but not worth sharing)

Holistic Body Pose

Detect both body and hand pose simultaneously:

// （[15:38](https://developer.apple.com/videos/play/wwdc2024/10163/?time=938)）
let request = DetectHumanBodyPoseRequest()
request.detectsHands = true
let observation = try await request.perform(on: image)
print(observation.leftHandObservation)  // Left hand pose
print(observation.rightHandObservation) // Right hand pose
  • detectsHands: When true, returned observation includes hand data
  • leftHandObservation / rightHandObservation: New properties for left and right hand pose

Core Takeaways

  • 1. Migrate to new API: Use new API for new projects; migrate old projects gradually. Start with most common requests (DetectBarcodesRequest, RecognizeTextRequest); three steps—remove prefix, remove handler, add async/await. Benefits: less code, type safety, Swift 6 strict concurrency checking.

  • 2. Replace manual calculation with toImageCoordinates(): Vision’s normalized coordinates (bottom-left origin) often conflict with SwiftUI (top-left origin) and other frameworks. Use the new coordinate conversion API to eliminate manual calculation errors. Specify origin: .upperLeft for most UI frameworks.

  • 3. Use performAll for multi-request scenarios: When executing multiple Vision requests simultaneously (such as barcodes and text), use performAll for AsyncStream. Process each result immediately without waiting for all. Better UX—show barcode results as soon as detected, don’t wait for text recognition.

  • 4. Limit concurrency to control memory: For large image batches, use TaskGroup with limited concurrency (around 5 recommended). Vision requests can use significant memory; limiting concurrency prevents memory spikes causing system pressure.

  • 5. Use image aesthetics scoring to optimize photo library: Use CalculateImageAestheticsScoresRequest to evaluate user photo libraries—auto-filter low-quality photos (blur, exposure issues) or identify utility images (screenshots, receipts). Build “highlights” features showing only memorable photos.

Comments

GitHub Issues · utterances