WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Platforms State of the Union

Platforms State of the Union

WWDC 2024
Watch original video

Highlight

Apple Intelligence delivers a personal intelligence system on iPhone through a dual architecture of on-device foundation models and Private Cloud Compute, enabling language understanding, image generation, and cross-app actions.

Core Content

Developers have long faced a dilemma: generative AI keeps getting more capable, but large models are too slow on device and too risky in the cloud. User data—mail, photos, calendar—lives scattered across apps, so models lack context; Siri can invoke only limited actions, and when a user says “summarize this note and paste it into my email,” Siri can’t help.

Apple Intelligence answers with a dual on-device + cloud architecture. A quantized, adapter-specialized foundation language model runs on device for everyday tasks like summarization, proofreading, and email replies; more complex reasoning goes through Private Cloud Compute. PCC runs on a stripped-down iOS subset with no persistent storage, and server management tools are replaced to prevent privileged access. Devices establish end-to-end encrypted connections to PCC; request data isn’t retained after the response returns, and Apple can’t access it. Apple commits to publishing a virtual image of every PCC production build for security researchers to review.

The other thread is App Intents. Apple Intelligence organizes user information across apps with an on-device semantic index, then uses the App Intents Toolbox to understand what each app can do. Developers define Intents and Entities, and Siri routes user requests to the right cross-app actions—such as inserting a Notes summary into a Superhuman email draft.

Detailed Content

Apple Intelligence model architecture (02:50)

The on-device foundation model is a large language model (LLM). Apple makes it run on iPhone through three steps:

  1. Adapter fine-tuning—Each adapter is a small set of weights dynamically loaded onto the base model so the same model switches between tasks. Apple Intelligence includes a set of adapters fine-tuned for summarization, proofreading, email replies, and more.
  2. Quantization compression—16-bit models are compressed to an average of under 4-bit while preserving quality.
  3. Inference optimization—Speculative decoding, context pruning, and group query attention, tuned specifically for the Neural Engine.

Image generation uses a similar architecture: a diffusion model plus adapters, supporting different styles and Genmoji.

Writing Tools, Genmoji, Image Playground (09:02)

  • Apps that render text fields with standard UI frameworks automatically get Writing Tools. The new TextView delegate API lets you customize behavior, such as pausing sync while Apple Intelligence processes text.
  • Genmoji uses AttributedString (not plain text like regular emoji); set one attribute to let text views accept Genmoji input.
  • The Image Playground API provides a consistent image generation experience; images are created on device, so developers don’t need their own text-to-image service. In SwiftUI, integration takes just a few lines of code.

Siri and App Intents (12:06)

Siri gains two capabilities that benefit apps without extra adaptation: invoking app menu items (say “Show presenter notes” to trigger one); accessing text displayed in apps using the standard text system (see a reminder on screen saying “FaceTime him” and dial). With the new Spotlight API, App Entities can be included in the semantic index so Siri can retrieve content and perform actions across apps.

Xcode 16 and Swift Assist (21:13)

Xcode 16’s code completion engine is based on a model trained specifically for Swift and Apple SDKs, using project symbols for custom suggestions, running locally on Apple silicon. Swift Assist is a more powerful cloud model supporting natural-language code generation and references to project assets and symbols. Code isn’t stored on servers or used for training.

Swift 6 (27:45)

Swift 6 introduces compile-time data-race safety checks. The new Swift 6 language mode is optional and supports module-by-module migration. Swift Testing and Embedded Swift (for kernels and microcontrollers) also ship. The Swift repo moves to the new github.com/swiftlang organization.

SwiftUI and SwiftData (35:25)

SwiftUI adds the @Previewable macro, custom hover effects (visionOS), window customization APIs (macOS), and the Text Renderer API. UIKit gesture recognizers work in SwiftUI view hierarchies; SwiftUI animations can drive UIKit/AppKit views. SwiftData adds #Index, #Unique, custom data stores, and change history tracking.

visionOS 2 (56:30)

Volumes support user resizing and fixed/dynamic scaling modes; Ornaments can be placed freely along Volume edges; the new TabletopKit framework supports tabletop collaborative games; ARKit adds omnidirectional plane detection, Room Anchor, and Object Tracking API.

Game Porting Toolkit 2 (53:54)

Supports AVX2 and ray tracing evaluation; Metal adds resource management APIs to help DirectX developers port code; Xcode supports debugging raw HLSL shaders; Mac games can more easily port to iPad and iPhone.

Core Takeaways

  • What to do: Define App Intents and App Entities for core app actions Why it’s worth it: Apple Intelligence uses App Intents to understand what your app can do—the clearer the definitions, the better Siri routes requests to your app. Once App Entities are in the semantic index, Siri can retrieve your content across apps. How to start: Start with your 3–5 most common actions; define an App Intent for each and associated entities as App Entities, then expose them via the Spotlight API.

  • What to do: Integrate Writing Tools and Image Playground Why it’s worth it: Apps using the standard text system automatically get Writing Tools; one attribute enables Genmoji. Image Playground lets users create images in-app without building your own text-to-image service. How to start: Check whether text input uses standard UI frameworks; use the TextView delegate API to control sync behavior; add image generation in SwiftUI with imagePlaygroundSheet.

  • What to do: Prepare for Swift 6 migration Why it’s worth it: Swift 6’s compile-time data-race safety eliminates concurrency bugs at the root, with module-by-module migration so you don’t wait on dependencies. How to start: Enable Swift 6 language mode module by module in Xcode, fix data races from compiler diagnostics, and follow the migration guide on Swift.org.

  • What to do: Use Core ML Tools to optimize your own on-device models Why it’s worth it: Apple showed Mistral 7B with 4-bit quantization and stateful KV cache achieving 5Ă— faster inference and nearly 9Ă— lower peak memory—the same techniques apply to your models. How to start: Convert PyTorch models to Core ML with Core ML Tools, apply quantization and KV cache optimization, run in the Core ML framework, and test output quality.

Comments

GitHub Issues · utterances