WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Day two recap

Day two recap

WWDCShazamKitScreen TimeStoreKitNearby Interaction
Watch original video

Highlight

The second day of WWDC21 focused on demonstrating voice recognition, ShazamKit custom audio matching, Screen Time API, StoreKit 2 and Nearby Interaction accessory support. Developers can integrate system-level identification, restriction, purchasing and spatial awareness capabilities into their own apps.

Core Content

The recap for day two is just over a minute long. It does not expand on the code, but strings together several technical directions of the day.

These directions all come from real development troubles.

If you want to recognize sounds in the environment, you previously had to train the model, process the audio window, and determine the confidence level yourself. Sound Analysis has a built-in classifier that can directly identify more than 300 sounds.

If you want to synchronize the mobile app with the video on the TV, you previously had to rely on the network, timecode or manual triggering. ShazamKit allows the app to listen to the audio and match it to a custom directory, and then display the content based on the matching offset.

If you want to control parental control, you couldn’t get Screen Time’s system restriction capabilities before. Screen Time API breaks restrictions, authorization, and device activity monitoring into three frameworks, allowing apps to work within privacy boundaries.

If you want to subscribe, you have to deal with it beforeSKProductsRequest, transaction status and receipt verification. StoreKit 2 turns products, purchases, transactions, and benefits into native Swift APIs.

You want your phone and accessories to be aware of each other’s location. Previously, Nearby Interaction was mainly used between Apple devices. iOS 15 starts supporting compatible third-party UWB accessories.

The value of this recap is to show the way. It tells you which sessions are worth continuing the next day and what types of product problems each new capability solves.

Detailed Content

Use Sound Analysis to identify environmental sounds

（00:07）

recap mentioned that the developers learned how to analyze and classify sounds. The corresponding session isDiscover built-in sound classification in SoundAnalysis。

Sound Analysis provides built-in sound classifiers. It can identify discrete sounds from a microphone, audio file or video file. Developers do not need to train custom models first.

func getListOfRecognizedSounds() throws -> [String] {
    let request = try SNClassifySoundRequest(classifierIdentifier: .version1)
    return request.knownClassifications
}

Key points:

  • getListOfRecognizedSounds()Returns the sound labels supported by the built-in classifier. -SNClassifySoundRequestis a sound classification request. -.version1Specifies to use the first version of the built-in classifier provided by the system. -knownClassificationsExpose a list of tags recognized by the classifier.

After getting the tags, you can classify the audio files.

let request = try SNClassifySoundRequest(classifierIdentifier: .version1)

let analyzer = try SNAudioFileAnalyzer(url: url)

var observer: SNResultsObserving // TODO

try analyzer.add(request, withObserver: observer)
analyzer.analyze()

Key points:

  • The first line creates the same built-in classification request. -SNAudioFileAnalyzer(url:)Read local audio files. -SNResultsObservingReceive classification results. -analyzer.addBind requests and observers to analyzers. -analyzer.analyze()Start performing analysis.

Use ShazamKit for custom audio matching

（00:17）

recap mentions that ShazamKit can recognize custom audio. The scenario it gives is very specific: the TV plays a video and the companion app on the phone displays the problem at the correct time.

In the corresponding session, Apple uses an educational video as an example. The App first generates a reference signature for the video and audio and puts it into a custom directory. When running, the App collects the audio stream from the microphone, matches the directory, and thenpredictedCurrentMatchOffsetFind the current video location.

// Matching signatures using SHSession
let session = SHSession()
session.delegate = self

let signatureGenerator = SHSignatureGenerator()
try signatureGenerator.append(buffer, at: nil)

let signature = signatureGenerator.signature()
session.match(signature)

Key points:

  • SHSessionResponsible for performing matching. -session.delegate = selfLet the current object receive the matching results. -SHSignatureGeneratorConvert audio buffer to signature. -append(buffer, at:)Write the captured audio to the signature generator. -signature()Generate matching audio signatures. -session.match(signature)Use this signature to initiate a match.

Custom catalogs can also carry metadata. After successful matching, delegate returnsSHMatch, App starts frommediaItemsRead information such as title, teacher, episode number, etc.

// Receiving matches via the session delegate
extension SongResultViewController: SHSessionDelegate {

    public func session(_ session: SHSession, didFind match: SHMatch) {

        guard let matchedMediaItem = match.mediaItems.first else {
            return
        }

        DispatchQueue.main.async {
            self.songView.titleLabel.text = matchedMediaItem.title
            self.songView.artistLabel.text = matchedMediaItem.artist
        }

    }
}

Key points:

  • SHSessionDelegateReceives ShazamKit’s match callback. -session(_:didFind:)Fired when a match is found. -match.mediaItems.firstGet the first matching media item. -matchedMediaItem.titleandartistIs the metadata saved in the directory. -DispatchQueue.main.asyncEnsure that interface updates occur on the main thread.

Building parental controls with Screen Time API

（00:37）

recap mentions that Nolan demonstrated the Screen Time API. The demo app is called Homework: Children need to use designated learning apps before they can lift restrictions on certain apps.

Screen Time API consists of three frameworks.

Managed Settings are responsible for setting restrictions, such as blocking apps, filtering web pages, and locking account settings. Family Controls is responsible for guardian authorization and privacy protection. Device Activity is responsible for triggering extension code based on time windows and usage thresholds.

import FamilyControls

Task {
    do {
        try await AuthorizationCenter.shared.requestAuthorization(for: .child)
    } catch {
        // Handle the error here.
    }
}

Key points:

  • FamilyControlsProvides authorization access to Screen Time API. -TaskUsed to call asynchronous authorization methods. -AuthorizationCenter.sharedIs a shared instance of the authorization center. -requestAuthorization(for: .child)Request to use Family Controls on children’s devices.
  • Enter in case of failurecatch, the App needs to handle authorization failures.

After authorization, the App can schedule device activity monitoring. Device Activity calls the extension when the time window starts, ends, or a threshold is reached.

let schedule = DeviceActivitySchedule(
    intervalStart: DateComponents(hour: 15, minute: 0),
    intervalEnd: DateComponents(hour: 18, minute: 0),
    repeats: true
)

let center = DeviceActivityCenter()
try center.startMonitoring(.daily, during: schedule)

Key points:

  • DeviceActivityScheduleDescribes a monitoring time window. -intervalStartis the window start time. -intervalEndis the window end time. -repeats: trueMeans repeat every day. -DeviceActivityCenterResponsible for starting and stopping monitoring. -startMonitoring(_:during:)Register a named monitoring task.

Simplify in-app purchases and subscriptions with StoreKit 2

（00:46）

recap mentioned that StoreKit 2 can quickly implement in-app subscriptions (in-app subscriptions). Corresponding to the session description, StoreKit 2 provides Swift native API for obtaining products, initiating purchases, processing transactions, and determining rights and customer status.

To obtain product information, you can directly callProduct.products(for:)。

let products = try await Product.products(for: ["fuel.small", "car.sports", "nav.pro"])

// Group by type
let consumables = products.filter { $0.type == .consumable }
let nonConsumables = products.filter { $0.type == .nonConsumable }
let subscriptions = products.filter { $0.type == .autoRenewable }

Key points:

  • Product.products(for:)Pull product information based on product ID. -try awaitIndicates that this is an asynchronous call that will throw an error. -products.filterGroup by product type. -.consumableIt is a consumable product. -.nonConsumableIt is a non-consumable product. -.autoRenewableIt is an auto-renewable subscription.

The purchase process also becomes an asynchronous call.

let result = try await product.purchase()

switch result {
case .success(let verification):
    let transaction = try checkVerified(verification)
    await transaction.finish()
case .userCancelled:
    break
case .pending:
    break
@unknown default:
    break
}

Key points:

  • product.purchase()Initiate a purchase. -.successReturn verification results. -checkVerifiedUsed to confirm that the transaction has been verified. -transaction.finish()Tells StoreKit that transaction processing is complete. -.userCancelledIndicates that the user cancels the purchase. -.pendingIndicates that the transaction is still awaiting approval or payment.

Use Nearby Interaction to connect third-party UWB accessories

（00:50）

recap mentions that Nearby Interaction can be used with third-party accessories. Corresponding to the session explanation, Nearby Interaction uses U1’s Ultra-Wideband capabilities to provide distance and direction updates to nearby devices.

Between two iPhones, the basic process is to create a session, set the delegate, create the configuration, and run the session.

let session = NISession()
session.delegate = self

let configuration = NINearbyPeerConfiguration(peerToken: peerDiscoveryToken)
session.run(configuration)

Key points:

  • NISessionIs the core object of Nearby Interaction. -session.delegate = selfReceive nearby object updates. -NINearbyPeerConfigurationFor two iPhones running the same app. -peerDiscoveryTokenFrom the peer device, it needs to be exchanged through its own connection channel. -session.run(configuration)Start spatial interaction.

After running, the delegate will receiveNINearbyObjectrenew. Objects contain distance, and some scenes also contain direction.

func session(_ session: NISession, didUpdate nearbyObjects: [NINearbyObject]) {
    guard let nearbyObject = nearbyObjects.first else { return }

    let distance = nearbyObject.distance
    let direction = nearbyObject.direction
}

Key points:

  • session(_:didUpdate:)Fires when a Nearby Interaction has a new measurement. -nearbyObjects.firstGet the most recently updated object. -distanceRepresents the distance to nearby objects. -directionRepresents directional information, availability depends on device and scenario.

Core Takeaways

1. Make an ambient sound reminder

  • What it does: Recognize doorbells, barking dogs, and baby cries, and get alerts on your Apple Watch or iPhone.
  • Why it’s worth doing: Sound Analysis’ built-in classifier can identify multiple types of discrete sounds, reducing the cost of training the model.
  • How ​​to start: Use firstSNClassifySoundRequest(classifierIdentifier: .version1)View the support tab, then connect to the microphone audio stream to turn high-confidence results into notifications.

2. Make a video synchronous answering app

  • What to do: The TV plays the course video, and the iPad automatically displays the corresponding question and answer input box.
  • Why it’s worth it: ShazamKit custom directories map audio locations to course content without relying on the same network or player interface.
  • How ​​to start: Generate a reference signature for the video and save the question offset; use it at runtimeSHSessionMatch the audio and then use the matching offset to find the current question.

3. Do a homework pattern

  • What to do: After the child completes the usage time of the learning app, the game app will be automatically unblocked.
  • Why it’s worth it: The Screen Time API can enforce throttling on children’s devices and respond to usage thresholds via Device Activity extensions.
  • How ​​to start: Apply for Family Controls capability first; useAuthorizationCenterrequest authorization; useDeviceActivityCenterCreate a schedule; use Managed Settings in the extension to update blocking rules.

4. Make a lightweight subscription function

  • What to do: Add a monthly subscription to the content app, and subscribed users will unlock premium columns.
  • Why it’s worth doing: StoreKit 2 puts product query, purchase and equity judgment into the Swift async/await process.
  • How ​​to get started: Configure subscription products in App Store Connect; useProduct.products(for:)Pull items; callproduct.purchase(); Use current rights and interests to determine whether to display advanced content.

5. Make an accessory finding experience

  • What: Point iPhone at a UWB-compatible accessory and display distance changes.
  • Why it’s worth it: Nearby Interaction supports compatible third-party hardware in iOS 15, and the app can get spatial awareness updates.
  • How ​​to get started: Read Nearby Interaction accessory specs; create in the appNISession;Exchange required configuration data with accessories;Update interface based on distance and direction.

Comments

GitHub Issues · utterances