WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Platforms State of the Union

Platforms State of the Union

Apple IntelligenceFoundation ModelsCore AIApp IntentsSwiftUIXcode
Watch original video

Highlight

Apple expanded the Foundation Models framework into a unified multi-model API, added the Core AI framework and enhanced App Intents, and paired them with agentic coding in Xcode 27 so developers can build AI features with no upfront API cost and gain a system-level discovery path.

Core Content

The biggest barrier to bringing large models into apps is cost

Developers who want AI in their apps face a familiar dilemma: connecting to OpenAI costs money, self-hosting requires operations work, and on-device models are often too limited. Small and midsize teams frequently get stuck at step one: they cannot make the cost model work, so the project is shelved.

This year, Apple essentially paid that bill for developers. The Foundation Models framework now supports server models, including third-party cloud models such as Claude and Gemini. More importantly, for developers whose apps have fewer than two million first-time downloads, calls to Apple Foundation Model running on Private Cloud Compute have no API fee.

That means an independent developer can start building an AI feature that requires complex reasoning today without worrying about an enormous bill at the end of the month.

If an app cannot be discovered, even great features go unused

You may spend three months building an AI feature, but users still may not know it exists. Siri cannot understand what your app can do, Spotlight cannot find your content, and the system AI cannot see you.

This year, App Intents became the “router” for system-level AI. Through entity schemas and intent schemas, your app content is compiled into the Spotlight semantic index, and Siri can directly understand and invoke your features. With the new View Annotations API, users can even point to something on screen and say “send this to Kevin,” and the system can accurately locate and execute the action.

Your app is no longer an island; it becomes a node in the system intelligence network.

From writing code to running tests, AI agents handle the whole flow

Writing code is only one part of development. Design, testing, localization, and bug fixing consume a huge amount of developer time.

Xcode 27’s agentic coding brings AI agents into the entire workflow. You describe a requirement, and the agent plans, writes code, runs previews, tests features, and fixes crashes. Device Hub unifies simulators and physical devices. Previews can show variants for any property. Xcode Cloud builds are twice as fast and no longer require App Store Connect configuration.

The developer’s role shifts from “handwrite every line of code” to “describe the problem, review the plan, and verify the result.”

Details

Foundation Models framework: one API for any model

(02:56) This year, the Foundation Models framework expands from a single on-device model into a unified multi-model API. There are three core changes.

First, multimodal input. You can now attach images directly in a prompt, and the model understands text and image content together.

import FoundationModels

let session = LanguageModelSession()

func brainstormProject(image: Image, theme: String) async throws -> String {
    let prompt = Prompt(
        "Give me three origami project ideas based on this image and theme.",
        images: [image]
    )
    let response = try await session.generateText(for: prompt)
    return response.text
}

Key points:

  • LanguageModelSession() creates a session for interacting with the model
  • Prompt accepts both text and images, enabling multimodal understanding
  • generateText(for:) sends the request and retrieves the generated result

The Vision framework is also integrated, providing tools such as precise OCR text extraction and barcode scanning, all completed on device.

Second, server models. Complex tasks can switch seamlessly to cloud models such as Claude and Gemini. Any model provider can implement the Language Model protocol through a Swift package.

import FoundationModels
import ClaudeProvider // Third-party Swift package

let profile = DynamicProfile {
    Profile("Creative generation") {
        LanguageModel.claude
        Temperature(0.8)
    }
    Profile("Deep reasoning") {
        LanguageModel.appleFoundationModel
        ReasoningLevel(.deep)
    }
}

Key points:

  • Third-party model providers are brought in through Swift packages
  • DynamicProfile allows dynamic model switching within the same session
  • Temperature(0.8) controls the creativity of the generated result

Third, Dynamic Profiles. This new declarative API lets you switch models, tools, and instructions within the same conversation based on the scenario.

import FoundationModels

let session = LanguageModelSession(profile: origamiProfile)

var origamiProfile: DynamicProfile {
    DynamicProfile {
        Profile("Creative generation") {
            LanguageModel.appleFoundationModel
            Temperature(0.8)
        }
        Profile("Tutorial generation") {
            LanguageModel.appleFoundationModel
            ReasoningLevel(.deep)
        }
        Profile("Term explanation") {
            LanguageModel.systemLanguageModel // On-device model
        }
    }
}

Key points:

  • Three profiles share one continuous transcript, so context is not lost
  • Simple tasks use the on-device systemLanguageModel; complex tasks use the cloud
  • Profiles are recalculated on every model turn, keeping them automatically up to date

(12:17) The supporting tools have also been upgraded across the board: the Evaluations framework tests prompt reliability, the FM instrument visualizes model behavior for debugging, and the fm CLI lets you call models directly from the terminal. There is also a private RAG tool based on Core Spotlight, plus an upcoming open-source Foundation Models Swift package, so the same API can run both in apps and on servers.

Core AI: bring any model to the device

(13:15) If you have your own model to deploy on device, Core AI is the best choice. It provides memory-safe Swift APIs, supports fine-grained quantization management and model specialization, and lets you write custom GPU kernels.

The companion Python tools can convert and optimize PyTorch models. The compiler supports ahead-of-time compilation, and Instruments plus the visual debugger can trace tensor values directly back to the original Python source.

From real-time vision models on iPhone to multi-billion-parameter LLM agents on Mac, Core AI can handle it. There is no server dependency and no token cost.

App Intents: help system AI discover and use your app

(14:45) The core of App Intents is schemas: system-defined structured templates that Siri deeply understands thanks to years of language model training.

import AppIntents

@EntitySchema
struct MessageEntity: IndexedEntity {
    let id: UUID
    let content: String
    let sender: ContactEntity
}

@IntentSchema
struct SendMessageIntent: AppIntent {
    static var title: LocalizedStringResource = "Send Message"

    @Parameter(title: "Recipient")
    var recipient: ContactEntity

    @Parameter(title: "Content")
    var message: String

    func perform() async throws -> some IntentResult {
        try await MessageService.send(to: recipient, content: message)
        return .result()
    }
}

Key points:

  • @EntitySchema and @IntentSchema help Siri understand your content and actions
  • The IndexedEntity protocol automatically compiles content into the Spotlight semantic index
  • System-defined schemas cover categories such as task management, photo editing, and communication

(16:27) The View Annotations API associates screen content with entities. When the user says “send this to Kevin,” the system can identify which image on screen “this” refers to.

import SwiftUI
import AppIntents

struct MessageListView: View {
    var messages: [MessageEntity]

    var body: some View {
        List(messages) { message in
            MessageRow(message: message)
                .entityAnnotation(message)
        }
    }
}

Key points:

  • .entityAnnotation(message) associates a list row with its corresponding MessageEntity
  • When users refer to screen content in natural language, the system can locate it accurately
  • Paired with App Intents, users do not need to learn specific commands

SwiftUI: less code, more capability

(29:12) SwiftUI gets substantial improvements this year in interaction, speed, and capability.

Drag-to-reorder now takes one line of code:

Grid {
    ForEach(origamiModels) { model in
        OrigamiCard(model: model)
            .reorderable()
    }
}
.reorderContainer()

Swipe actions are no longer limited to List:

ScrollView {
    ForEach(projects) { project in
        ProjectRow(project: project)
            .swipeActions {
                Button("Delete", role: .destructive) { delete(project) }
            }
    }
}
.swipeActionsContainer()

(32:40) Toolbar adds visibilityPriority and overflowMenu, keeping important buttons visible as the window shrinks while folding secondary buttons into a menu.

(33:53) Document-based apps get new infrastructure, with direct file URL reading and writing, partial updates, Observable configuration, and deep Swift concurrency integration.

(34:25) The Spatial Preview framework lets Mac apps stream 3D models live to Apple Vision Pro for spatial preview and editing.

Swift 6.4: smoother everyday coding

(38:41) Swift 6.4 focuses on removing everyday friction.

// Cross-platform availability declarations become much shorter
@available(anyAppleOS, introduced: 27.0)
func newFeature() { }

// Await directly inside defer
defer {
    await cleanupResources()
}

// Local warning control
#warning("Temporarily compatible with older versions", as: .warning)
#warning("New code must handle errors", as: .error)

Key points:

  • anyAppleOS replaces enumerating every platform version
  • await can be used directly inside defer
  • Warning levels can be controlled by region, which is especially useful during migrations

(39:55) The painful “The compiler is unable to type check this expression in reasonable time” error is greatly reduced in common scenarios: code either compiles directly or produces a more actionable error message.

Xcode 27: agentic coding end to end

(42:47) Xcode 27 is 30% smaller, exclusive to Apple silicon, and updates components automatically in the background. Settings sync through iCloud, so a new Mac can restore your development environment with one click. New projects no longer require a bundle ID; choose a template and go straight into the editor.

(45:46) The theme system is fully personalized. The entire IDE color palette can be customized, and each project can use a different theme.

(47:54) Device Hub replaces Simulator and unifies management of simulators and physical devices. It supports dynamic resizing of iOS apps, switching system settings, and launching and interacting with physical devices directly from the Mac.

(50:12) Agentic coding is the biggest highlight. Use the /plan command to have the agent propose a plan first, then implement after confirmation. The agent explores project structure, asks clarifying questions, generates code, and creates previews.

(53:18) The agent can also test automatically: clicking, swiping, and typing in Device Hub, then generating test reports and screenshots. Localization is also supported; the agent translates strings based on context rather than replacing words one by one.

(55:13) Crash reports in Organizer can be handed directly to the agent for analysis, localization, reproduction, fixing, and verification, with the whole flow automated.

Core Takeaways

  • Build a local “photo-to-customization” ecommerce app: Users take a photo of a corner in their home, and the app uses the multimodal capabilities of Foundation Models to recommend furniture that fits the scene. The on-device model handles image understanding, while the cloud model generates detailed recommendation copy. No cost under two million downloads.

  • Build a note app controlled by natural language: Expose actions such as “create note,” “search notes,” and “share note” through App Intents. A user can tell Siri, “Turn the meeting whiteboard I just photographed into notes and share them with the team,” and the system automatically recognizes the screen content and acts.

  • Build a fully local privacy-sensitive AI tool: Use Core AI to deploy a local document analysis model. Sensitive files such as contracts and medical records never leave the device, with no server dependency and no subscription cost.

  • Prototype a multilingual social app in three days with the Xcode agent: Start from a requirements description and let the agent generate the SwiftUI interface, Foundation Models integration, and multilingual support. The agent tests UI adaptation in different languages; the developer reviews and adjusts.

  • Build a creative tool with user-configurable AI agents: Use Dynamic Profiles so users can configure roles such as “creative assistant,” “proofreader,” and “translator,” each with different models and parameters, collaborating inside the same project.

Comments

GitHub Issues · utterances