WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Keynote

Keynote

WWDC 2024
Watch original video

Highlight

Apple Intelligence debuts as a personal intelligence system: on-device models handle text, diffusion models generate images, Private Cloud Compute extends compute on Apple silicon servers without retaining user data; Siri gains on-screen awareness and cross-app actions; ChatGPT (GPT-4o) integrates into Siri and Writing Tools.

Core Content

Every WWDC Keynote sets the direction for Apple platforms for the year. The 2024 theme is clear: make devices understand you.

At the operating system level, visionOS 2 turns 2D photos into spatial photos with machine learning, extends Mac Virtual Display to ultrawide (equivalent to two 4K displays side by side), adds train scenarios to Travel Mode, and provides TabletopKit APIs for tabletop collaboration apps anchored to flat surfaces. iOS 18 focuses on personalization—free placement of Home Screen icons, dark mode and tinting, fully customizable Control Center, and replaceable Lock Screen bottom buttons. Messages adds RCS, scheduled sending, and text effects. The Passwords app spins out from iCloud Keychain with cross-platform sync including Windows.

But what truly defines 2024 is Apple Intelligence. It runs as a personal intelligence layer embedded in the system. On-device language models handle text summarization, rewriting, and proofreading; diffusion models handle image generation and Genmoji. When tasks exceed device compute, Private Cloud Compute runs larger models on Apple silicon servers, retains no user data after processing, and server software images are auditable by independent experts. Siri’s upgrade builds on the same architecture: more natural conversational understanding, on-screen awareness, cross-app action orchestration, and third-party app integration via the App Intents framework.

Craig Federighi announced at the end that ChatGPT (GPT-4o) integrates into Siri and the Compose feature in Writing Tools. Users can use it for free without signing up; ChatGPT subscribers can connect accounts for paid features. User permission is requested before each information share. Apple also said other AI models will be integrated in the future.

Detailed Content

visionOS 2 (05:43)

  • 2D photos to spatial photos: Machine learning derives left and right eye views from a single 2D image, generating spatial photos with natural depth in one tap (08:15)
  • Mac Virtual Display ultrawide mode: Higher resolution and size, expandable to wraparound ultrawide equivalent to two 4K displays, relying on Mac-side dynamic foveated rendering for clarity (09:28)
  • Gesture enhancements: Raise hand and tap to open Home View, flip wrist to see time and battery, tap again to open Control Center (09:02)
  • TabletopKit: Quickly create apps anchored to flat surfaces, suited for board games and manufacturing workstations, with FaceTime spatial Personas (10:37)
  • Professional spatial video workflow: Canon spatial lens for EOS R7, Final Cut Pro editing, Vimeo app release (11:43)
  • Apple Immersive Video production: Partnership with Blackmagic Design using Blackmagic camera + DaVinci Resolve Studio + Apple Compressor (12:32)
  • Vision Pro market expansion: Launches June 28 in China, Japan, Singapore; July 12 in Australia, Canada, France, Germany, UK (13:55)

iOS 18 (14:34)

  • Home Screen customization: Icons placed freely along wallpaper, dark mode icons, tint mode auto-colors from wallpaper (14:52)
  • Control Center redesign: Multi-page groups (media playback, home control, etc.), Controls Gallery for third-party controls, Lock Screen bottom button and Action Button trigger custom controls (16:34)
  • Privacy enhancements: App Lock (Face ID/Touch ID protects individual apps, info hidden from search and notifications), hidden app folder, selective contact authorization, accessory pairing privacy (18:22)
  • Messages: RCS support, Send Later, text formatting (bold/italic/underline/strikethrough), text effects, any emoji and sticker Tapback, satellite messaging (iPhone 14 and later, end-to-end encrypted) (20:38)
  • Mail categorization: On-device categorization into Primary, Transactions, Updates, Promotions, with digest view grouping emails from the same merchant (22:55)
  • Maps: Topographic maps and hiking trails for 63 US national parks, offline save, turn-by-turn voice navigation (24:24)
  • Wallet: Tap to Cash (two phones tap to transfer Apple Cash without exchanging contact info), Apple Pay Online rewards redemption and installment payments (24:42)
  • Photos redesign: Unified view, Collections organized by theme, Carousel showing best content, customizable sorting and pinning (26:16)
  • Game Mode: Minimizes background activity for high frame rates, improves AirPods and wireless controller response (25:46)

AirPods and tvOS (30:45)

  • AirPods interaction: Nod yes / shake no to respond to Siri; AirPods Pro call Voice Isolation; personalized spatial audio API for games (31:04)
  • Apple TV InSight: Real-time actor character names and song info, add to Apple Music playlists (33:12)
  • Enhance Dialogue: Extended to TV speakers and receivers, using ML to enhance voice clarity (33:41)
  • 21:9 projector support: Display widescreen movies as the director intended (34:32)

watchOS 11 (36:02)

  • Training Load: Automatically estimates effort level (1-10) from heart rate, pace, elevation, and personal data, combined with workout duration to calculate training load (36:23)
  • Vitals App: Tracks heart rate, respiratory rate, wrist temperature during sleep, alerts on anomalies with correlated analysis (alcohol, elevation, illness, etc.) (38:10)
  • Smarter Smart Stack: Automatically adds precipitation alerts, translation widgets, and more; Live Activities come to Apple Watch (40:20)
  • Double Tap API: Developers can define double-tap actions, such as Sprout Baby Tracker one-tap baby sleep logging (41:26)

iPadOS 18 (43:05)

  • Floating tab bar: Navigation bar can morph into sidebar, supports custom frequent tabs (44:00)
  • Calculator + Math Notes: Apple Pencil handwritten equations auto-solve, results shown in handwriting style; supports variables and live updates; line summation; typing y= generates function graphs, hover to adjust variables for live preview (45:52)
  • Smart Script: On-device ML recreates your handwriting style, optimizing even fast messy handwriting into readable script in real time; pasted text becomes your handwriting style; scratch to erase; press and hold to make space for insertion (48:51)

macOS Sequoia (50:54)

  • iPhone Mirroring: Wirelessly control iPhone from Mac, iPhone stays locked with StandBy visible; iPhone notifications appear in Mac Notification Center; cross-device drag and drop (52:44)
  • Window tiling: Drag windows to edges for suggested tile positions, keyboard and menu shortcuts (55:29)
  • Passwords App: Unified management of passwords, verification codes, security alerts, synced across Mac/iPad/iPhone/Vision Pro/Windows (56:26)
  • Safari Highlights: ML auto-detects page info (addresses, summaries, people/music/TV links); Reader adds table of contents and summary; Viewer centers web video playback (57:13)
  • Game Porting Toolkit 2: Improved Windows compatibility and shader debugging tools; Xcode supports unified game code and shaders across devices (01:00:32)

Apple Intelligence (01:06:18)

  • Architecture: On-device large language models + diffusion models + on-device semantic index (organizes and retrieves cross-app information). On request, semantic index locates relevant personal data and passes it to generative models (01:12:02)
  • Private Cloud Compute: Complex requests sent to Apple silicon servers, data used only to fulfill the request, not retained after processing; iPhone/iPad/Mac encrypt and verify server software is publicly logged for audit (01:14:07)
  • Siri upgrade: New glowing edge UI; natural language understanding (corrects even verbal slips); conversational context continuity; text input support (double-tap bottom); product knowledge base answers thousands of how-to questions; future support for on-screen awareness and cross-app action orchestration (01:16:06)
  • Writing Tools: System-level Rewrite (friendly/professional/concise tone), Proofread (grammar + word choice + sentence structure), Summarize (TL;DR inserted at top of emails); third-party apps get it automatically (standard editable text views) (01:23:40)
  • Genmoji: Generate custom emoji from text descriptions in the keyboard, can be based on people in your photo library (01:28:24)
  • Image Playground: Choose concepts (theme/outfit/accessories/location) or enter descriptions, preview in seconds, three styles (Animation/Sketch/Illustration), runs on device; developers can integrate via API (01:29:17)
  • Image Wand: Circle rough sketches with Apple Pencil in Notes, Image Wand converts to polished images; circling blank areas infers from context (01:32:15)
  • Photos Clean Up: Auto-identifies and removes background distractions; natural language search for specific photo/video moments; descriptive Memory Movie generation (01:33:02)
  • Priority Notifications / Reduce Interruptions: Priority notifications pinned to top; new Focus mode selectively shows information needing immediate attention based on notification content (01:26:56)
  • ChatGPT integration: Siri requests permission then calls GPT-4o when needed; Compose writing tool can call ChatGPT to generate content and images; free without account, subscribers can connect accounts (01:36:39)
  • Supported devices: iPhone 15 Pro, iPad and Mac M1 and later; US English at launch, beta in fall, some features and languages rolling out over the next year (01:40:56)

Core Takeaways

  1. Add App Intents to your app: Apple Intelligence discovers and invokes app features through the App Intents framework. If your app already uses SiriKit, it automatically gets some enhancements; new projects should define actions directly with App Intents so Siri can execute commands like “apply a cinematic filter to yesterday’s photos in Darkroom.” Why it’s worth doing: When Siri cross-app orchestration launches, apps with App Intents will be prioritized for discovery and invocation. How to start: Map core app operations, wrap them with the AppIntent protocol, prioritizing search, edit, and share intents.

  2. Adapt to Writing Tools and Image Playground API: Standard editable text views get Writing Tools automatically with no extra development; Image Playground integrates in a few lines of code. Why it’s worth doing: Users will expect rewrite and summarize in Notes, Mail, and third-party apps; apps without support will feel incomplete. How to start: Confirm your app uses standard text views; call Image Playground API where image creation is needed.

  3. Revise privacy strategy for App Lock and selective contact authorization: iOS 18 lets users lock and hide apps and grant selective contact access. If your app relies on full contact access or background data reads, runtime permissions may be missing. Why it’s worth doing: When users lock an app, its info disappears from search and notifications, and your app may miss data it depends on. How to start: Audit dependencies on contacts, notifications, and background reads; design degraded experiences for partial authorization and App Lock states.

  4. Enter tabletop collaboration with visionOS TabletopKit: TabletopKit provides flat-surface anchoring APIs, paired with FaceTime spatial Personas—a low-barrier entry for board games, manufacturing workstations, and medical training apps. Why it’s worth doing: Vision Pro app ecosystem is still early; tabletop scenarios have less competition, and TabletopKit greatly lowers spatial anchoring development difficulty. How to start: Create a board game prototype with TabletopKit to validate flat anchoring and multi-persona interaction.

Comments

GitHub Issues · utterances