WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Integrate on-device AI models into your app using Core AI

Integrate on-device AI models into your app using Core AI

Core AIFoundation ModelsOn-device AISAM3Qwen
Watch original video

Highlight

Core AI lets developers load and run third-party open-source models such as Qwen and SAM3 through the same API style as Foundation Models. Data stays on device, no server is required, and Ahead-of-Time compilation can remove first-load stalls.

Core Content

From hand-authored cards to AI-generated cards

Imagine you are building a language-learning app that helps students memorize words. The traditional approach is to prebuild vocabulary cards with words, translations, and example sentences, then bundle them into the app. The problem is that a student walking down the street may see a hummingbird and want to learn that word immediately. It is not in your prebuilt card deck. Student curiosity and the real world are unlimited; static cards can never keep up.

(00:17) Core AI solves exactly this problem. The student opens the camera, points it at the hummingbird, and says “I want to learn this.” The app segments the image, identifies the object, and generates a vocabulary card locally, without network access. There are no token costs, no cloud latency, and the student’s photo data never leaves the device.

Two models working together

(02:56) This scenario needs two capabilities: extracting the target object from the image, and generating multilingual vocabulary information from the object name. Apple’s solution is to split the problem into two specialized models, each kept under 1 billion parameters so total resource use stays manageable.

The first model is SAM 3 (Segment Anything Model 3), an image segmentation model based on Vision Transformer. The student provides the text prompt “flower”, and SAM 3 precisely extracts the flower from the photo while also producing an English label.

The second model is a multilingual large language model. The English label “Hummingbird” goes in, and the model outputs the Chinese vocabulary word, translation, and example sentence. The speaker chose Qwen, which supports 119 languages; the 0.6B-parameter version is small enough to fit on iPhone.

(04:57) Why use two models instead of one? Task-specific models deliver better quality, each model is smaller, and they can be upgraded independently. SAM 3 focuses on segmentation, Qwen focuses on language, and each does its own job.

Where the models come from

(07:02) You can get the original PyTorch models from Hugging Face or GitHub, then convert them with the Core AI PyTorch extension package. The simpler path is to use the Core AI Models repository directly. It already provides conversion scripts and precompiled .aimodel files for popular models such as SAM 3 and Qwen.

The repository also includes a Swift Package that wraps preprocessing and postprocessing logic such as text encoding and mask extraction. You do not need to manage tensor shapes yourself; you call the Swift API.

Solving first-load stalls

(13:08) When a model loads for the first time, Core AI performs Specialization: it compiles the generic model into a format tailored to the current device’s Neural Engine. This can be slow. If the user takes a photo and then stares at a spinner, the experience is poor.

Apple’s solution has two parts. First, do not put the models in the app bundle. SAM 3 plus Qwen is over 1 GB, and every updating user would have to download it whether or not they use the feature. Use Background Assets instead, and download models on demand only when the user actively enables the feature.

Second, use coreai-build on the development machine for Ahead-of-Time (AOT) compilation. The compiled model still needs final device-specific specialization, but the amount of work drops sharply. User wait time falls from tens of seconds to a few seconds.

The same code runs on macOS

(19:57) The iOS code can be reused directly on macOS. Macs have more memory and compute, so Qwen 0.6B can be replaced with Qwen3 8B for stronger reasoning and longer context.

The speaker added batch processing to the Mac app: import an entire trip’s photo folder, let SAM 3 segment all images in parallel to find objects, then use the 8B model to generate every vocabulary card at once. It can also sort cards by difficulty and build a lesson plan. Work that used to take an afternoon of manual entry now takes one prompt.

Details

Load and run SAM3 image segmentation

(11:01) The CoreAIImageSegmenter module in the Core AI Models repository wraps all tensor preprocessing and postprocessing for SAM 3.

import CoreAIImageSegmenter

// Load the SAM3 model from disk
let segmenter = try await ImageSegmenter(resourcesAt: sam3ModelURL)

// Pass an image and text prompt to get segmentation results directly
let response = try await segmenter.segment(image: inputImage, prompt: "flower")

// Extract the best segmentation mask
let mask = response.segments.first?.mask

Key points:

  • ImageSegmenter is the high-level API provided by the coreai-models Swift Package. It automatically handles image encoding, feature extraction, and mask decoding.
  • segment(image:prompt:) accepts a UIImage or CGImage plus a text prompt, and returns segmentation results sorted by confidence.
  • response.segments.first?.mask gives the mask that best matches the prompt and can be used directly for image composition or cropping.

Load the language model and create a session

(11:28) Loading and calling a third-party language model reuses the same API style as the Foundation Models framework.

import FoundationModels
import CoreAILanguageModels

// Create the model instance; one line loads resources, creates the engine, and sets up the tokenizer
let model = try await CoreAILanguageModel(resourcesAt: qwen3ModelURL)

// Use the familiar LanguageModelSession API
let session = LanguageModelSession(model: model)

// Generate a response
let response = try await session.respond(to: "...")

Key points:

  • CoreAILanguageModel is Core AI’s wrapper around third-party models, and its initializer takes the path to an .aimodel file.
  • LanguageModelSession is identical to the API used with Apple’s built-in large model; both respond(to:) and streaming output are supported.
  • FoundationModels is imported because LanguageModelSession is defined there. Third-party models and AFM share the same session abstraction.

Generate structured output with @Generable

(12:29) Vocabulary cards require a strict field structure, so parsing free text is not enough. The @Generable macro lets the model produce structured data that conforms to a Swift type.

import FoundationModels
import CoreAILanguageModels

@Generable
struct VocabCard {
    let chineseWord: String
    let englishMeaning: String
    let exampleSentence: String
}

let model = try await CoreAILanguageModel(resourcesAt: modelURL)
let session = LanguageModelSession(model: model)

let response = try await session.respond(
    to: "Create a vocab card for flower",
    generating: VocabCard.self
)

let card: VocabCard = response.content

Key points:

  • The @Generable macro generates JSON Schema constraints at compile time, forcing model output to decode into VocabCard.
  • The second parameter of session.respond(to:generating:) passes the target type; the framework handles prompt wrapping, JSON generation, and type conversion.
  • If the model output does not match the schema, the call throws a decoding error and must be handled with try.

AOT-compile the model

(17:22) The coreai-build command-line tool performs the most expensive compilation steps ahead of time on the development machine.

$ xcrun coreai-build compile MyModel.aimodel --platform iOS

Key points:

  • coreai-build is part of the Core AI toolchain and is installed with Xcode 27.
  • --platform iOS specifies the target platform and can generate compiled artifacts for a specific chip architecture.
  • The compiled model still needs final specialization on device, but the time drops from tens of seconds to a few seconds.
  • In real deployments, distribute the appropriate compiled version for each device architecture, such as A17, M2, or M3.

Core Takeaways

1. Learn by taking a photo: real-time vocabulary cards for any object

What to build: a language-learning app where users photograph any object and the app automatically segments it and generates vocabulary cards in the target language.

Why it is worth doing: Core AI reduces SAM 3 and Qwen calls to a few lines of code. Features that previously required a cloud vision API plus a translation API can now run fully locally, with no network dependency and no token cost.

How to start: import CoreAIImageSegmenter and CoreAILanguageModels from the Core AI Models repository, use ImageSegmenter.segment(image:prompt:) for object extraction, then use LanguageModelSession.respond(to:generating:) to generate structured card data.

2. Batch-generate study materials from a travel album

What to build: a Mac tool that imports a whole trip’s photo folder, automatically identifies objects, generates vocabulary cards, and groups them by topic.

Why it is worth doing: On Mac, you can use a larger Qwen3 8B model and reuse the same Swift code directly. Batch parallel processing plus long-context lesson planning turns hours of manual work into minutes.

How to start: build on the iOS code, add NSFileCoordinator to read photos in batches, use TaskGroup to call SAM 3 segmentation in parallel, then aggregate results into one large prompt for the 8B model to generate the lesson structure.

3. Offline, privacy-first AI feature modules

What to build: add an optional AI module to an existing app where all inference is local and sensitive data such as user photos or documents is never uploaded.

Why it is worth doing: Core AI’s on-demand download and AOT compilation mechanisms keep large-model features from bloating the app bundle or slowing launch. Background Assets downloads models only when the user actively enables the feature, so regular users are unaffected.

How to start: design a first-run onboarding page. When the user taps “Enable AI features”, trigger BGDownloadingRequest. After the download completes, load and specialize the model in the background so the next use hits the cache directly.

4. A unified cross-platform AI architecture

What to build: one Swift codebase for both iOS and macOS, using a small model on iOS for responsiveness and automatically switching to a larger model on Mac for quality.

Why it is worth doing: The APIs for CoreAILanguageModel and LanguageModelSession are identical across platforms. At runtime, choose different model sizes based on ProcessInfo. You do not need two codebases, and the Mac can use more memory to run stronger models.

How to start: at app launch, detect device model and memory size. Load Qwen 0.6B on iPhone and Qwen3 8B on Mac while leaving business logic unchanged. Entry point: ProcessInfo.processInfo.operatingSystemVersion plus conditional model loading.

5. Use AOT compilation to remove first-use stalls

What to build: precompile models during development with coreai-build to reduce user-side specialization wait time.

Why it is worth doing: First-load Specialization can take tens of seconds, which is a terrible experience after the user takes a photo. AOT compilation moves the most expensive work into development, leaving only final device specialization and reducing the wait to a few seconds.

How to start: integrate xcrun coreai-build compile MyModel.aimodel --platform iOS into CI and distribute the compiled artifact with the app. Entry point: xcrun coreai-build compile

Comments

GitHub Issues · utterances