Wednesday@WWDC21

Highlight

WWDC21 Day 3 focuses on AR photogrammetry, Shortcuts Action design, UIKit style system, Apple Watch accessibility innovation, Vision Framework character segmentation, iCloud Keychain verification code generator, and discoverability design principles.

Core Content

Wednesday’s WWDC21 recap strung together a series of specific technical updates, each directly responding to real-world developer needs.

AR field: Object Capture allows developers to generate 3D models using dozens of photos, eliminating the need to hire outsourced modelers. Suitable for e-commerce product display, AR menu, virtual try-on and other scenarios.

Shortcuts and Siri: Action design guidelines tell you which application functions should be exposed and how to design parameters to allow users to use your application more naturally through voice and shortcut commands.

UI Framework Update: UIKit NewUIButton.ConfigurationandcontentConfigurationMake button and cell style configuration more declarative, and Interface Builder supports previewing the effects of different accessibility settings.

Accessibility Innovation: AssistiveTouch for Apple Watch uses a combination of sensors and machine learning to detect micro-movements in the arm, giving users with upper limb disabilities full control of the watch through gestures. This is a typical case of “designing for a few and benefiting everyone”.

Vision Framework: Person Segmentation API can separate characters from the background to realize virtual background and character cutout. At the same time, face detection supports wearing masks, and the posture index is changed from discrete values to continuous values.

Security Authentication: iCloud Keychain has a built-in TOTP verification code generator, supports one-click setup, auto-fill, and cross-device synchronization, and is more secure and reliable than SMS.

Design Principle: Discoverability design allows users to naturally discover application functions through visual cues, progressive disclosure, consistent interactions, and closed feedback loops.

Detailed Content

Object Capture: Generate 3D models from photos

（00:08）

Traditional 3D modeling requires professional software and skills, with a long cycle and high cost. Object Capture uses photogrammetry to recover 3D structures from photos.

The API entry isPhotogrammetrySession：

import RealityKit

let inputFolderUrl = URL(fileURLWithPath: "/tmp/Sneakers/", isDirectory: true)
let session = try! PhotogrammetrySession(
    input: inputFolderUrl,
    configuration: PhotogrammetrySession.Configuration()
)

Key points:

Photos can be taken with iPhone, iPad, DSLR or drone
HEIC format photos will automatically extract depth data -Supports four precision levels: Reduced, Preview, Reduced, and Raw

Handle asynchronous output streams:

Task {
    do {
        for try await output in session.outputs {
            switch output {
            case .requestProgress(let request, let fraction):
                print("Request progress: \(fraction)")
            case .requestComplete(let request, let result):
                if case .modelFile(let url) = result {
                    print("Request result output at \(url).")
                }
            case .processingComplete:
                print("Completed!")
            default:
                break
            }
        }
    } catch {
        print("Fatal session error! \(error)")
    }
}

Key points:

for try awaitContinue to receive messages until the session is released -.requestProgressCan drive progress bar -.requestCompleteReturn model file URL

Shortcuts Action Design

（00:13）

Good actions have five qualities: useful, modular, multimodal, clear, and discoverable.

Useful: Solve users’ actual problems, such as “adding events” to calendars and “sending messages” to emails.

Modular: The input and output design is universal, and the output of one Action can be used as the input of the next Action.

Multi-modal: The same Action can be used in Shortcuts drag, Siri voice, and system suggestions.

Clear: The name starts with a verb, and the parameter labels accurately describe the content.

Discoverable: Educate users within your app, donate intents to let the system learn, and provide example shortcuts.

UIKit style system

（00:22）

Introduced in iOS 15UIButton.Configuration：

var config = UIButton.Configuration.filled()
config.title = "Continue"
config.image = UIImage(systemName: "arrow.right")
config.imagePlacement = .trailing
config.cornerStyle = .capsule

let button = UIButton(configuration: config)

cellcontentConfiguration：

var content = cell.defaultContentConfiguration()
content.text = "Title"
content.secondaryText = "Subtitle"
content.image = UIImage(systemName: "star")

cell.contentConfiguration = content

Key points:

Configuration is a value type and needs to be reassigned after modification.
Automatically adapt to Dynamic Type and accessibility settings
Interface Builder can preview the effects of different accessibility settings

Apple Watch Accessibility

（00:37）

AssistiveTouch combines three sensors to detect arm micro-movements:

Accelerometer (detects arm movement)
Gyroscope (detects wrist rotation)
Heart rate sensor (detects changes in blood flow)

Gestures include pinching, double pinching, fisting, and double fisting, which are distinguished from daily movements through machine learning models.

Key points:

No new hardware required, can be achieved by combining existing sensors
Low-impact gesture design to reduce energy consumption
Four gestures for complete operation: navigation, selection, scroll, return

Vision frame character segmentation

（00:48）

Person Segmentation API：

let request = VNGeneratePersonSegmentationRequest()
let requestHandler = VNImageRequestHandler(url: imageURL, options: options)

try requestHandler.perform([request])

let mask = request.results!.first!
let maskBuffer = mask.pixelBuffer

Quality Level: -accurate: Highest quality, suitable for computational photography -balanced: Balanced, suitable for frame-by-frame video processing -fast: Fastest, suitable for real-time stream processing

Face detection supports wearing masks, and posture indicators (roll, yaw, pitch) are changed from discrete values to continuous values.

iCloud Keychain Verification Code Generator

（00:55）

iOS 15 builds the TOTP verification code generator into iCloud Keychain.

Support one-click settings:

// In the app
func openVerificationCodeSetup(url: URL) {
    if let scene = UIApplication.shared.connectedScenes.first as? UIWindowScene {
        scene.open(url, options: nil, completionHandler: nil)
    }
}

// On the web
// <a href="apple-otpauth://totp/Example?secret=...&issuer=example.com">Set up verification code</a>

Key points:

otpauth://URL plusapple-Prefix directly opens iCloud Keychain
The verification code is automatically filled in, consistent with the SMS verification code experience
Sync across devices, securely backed up in iCloud

Discoverability design

（01:07）

Discoverability design allows users to discover features naturally through four principles:

Visual cues: Use high-contrast colors for main operations, and use badges or animations to prompt new functions.

Progressive Disclosure: Show basic features first, with advanced features appearing when needed. Gestures such as long press and swipe reveal more actions.

Consistent interaction: Follow the system standard interaction mode, such as pull down to refresh and left swipe to delete.

Feedback Closed Loop: Every operation has instant feedback, allowing users to confirm “I did it right”.

Core Takeaways

E-commerce product visualization: Use Object Capture to generate 3D models for products and display them in AR Quick Look, allowing users to see the true size and details of the product at home.
Voice-first application process: Design Shortcuts Action for high-frequency operations, allowing users to complete tasks without opening the application, and combine with Siri to achieve full voice operations
Virtual background and character cutout: Use Vision’s Person Segmentation to implement virtual background for video calls and photo editing, without the need for a third-party SDK
Universal value of barrier-free design: AssistiveTouch was originally designed for users with upper limb disabilities, but its gesture control method can benefit all users, such as one-handed operation in sports scenes
More secure authentication process: Use iCloud Keychain verification code instead of SMS to reduce costs while improving security. End-to-end encryption ensures that the verification code will not be leaked.

Create 3D models with Object Capture — Detailed explanation of photogrammetry API
Design great actions for Shortcuts, Siri, and Suggestions — Action Design Guide
Build interfaces with style — UIKit new style system
Accessibility by design: An Apple Watch for everyone — Apple Watch accessibility story
Detect people, faces, and poses using Vision — Vision framework human segmentation
Secure login with iCloud Keychain verification codes — Verification code generator implementation
Discoverable design — Discoverable design principles

Highlight

Core Content

Detailed Content

Object Capture: Generate 3D models from photos

Shortcuts Action Design

UIKit style system

Apple Watch Accessibility

Vision frame character segmentation

iCloud Keychain Verification Code Generator

Discoverability design

Core Takeaways

Related Sessions

Comments