WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Explore Packages and Projects with Xcode Playgrounds

Explore Packages and Projects with Xcode Playgrounds

WWDC 2020XcodePlaygroundsSwift Packages
Watch original video

Highlight

Xcode Playgrounds has always been a great tool for quickly experimenting with Swift code, but in the past there was a gap between it and actual projects - you couldn’t directly reference the code in the project in the playground.Xcode 12 bridges this gap.

Core Content

Playgrounds are best for quick experiments.When developers want to try out an API, verify a piece of Swift code, and write a working tutorial, they don’t want to build a complete sample app first.The problem is that the old Xcode Playground can easily stay in an isolated draft: the framework, package dependency, asset catalog, and Core ML model in the real project may not be directly available for running.

The changes in Xcode 12 come from the modern build system.After playgrounds are connected to the same set of build capabilities, Xcode knows how to build the Swift package, project target, and playground’s own sources/resources, and then expose these modules to the playground code.The session first uses NutritionFacts package to display runnable documents: the documents are written in the playground, the sample code can be executed, and the live view can be displayed on the right side.

The second scenario is closer to everyday projects.The Fruta project has NutritionFacts package dependency and a UtilityViews framework.After the SmoothieLab playground is opened, Xcode builds the target in the current scheme through the new Build Active Scheme option, allowing the playground to import frameworks and packages in the same workspace.This way, teams can put playgrounds into projects specifically for experimenting with recipes, charts, or other business logic.

The last demo addresses the resource issue.Xcode 12 playgrounds can handle resources like asset catalogs and ML models that require build steps.The speaker dragged Fruta’s ingredients asset catalog into playground resources and the YOLOv3 model. Xcode automatically compiled and generated the model.YOLOv3Swift class.Then use Vision to run object detection, and then use SwiftUI live view to view the detection results.

Detailed Content

Swift Package documents can be run directly

(00:30) Xcode 12 connects Playgrounds to modern build systems.This integration allows Xcode to build Swift packages and expose package modules to playground code.The documentation of NutritionFacts is placed in the Playgrounds folder of the package, and the user opens itPackage.swiftAfter that, you can directly run the sample code in the document.

(03:36) When the same package is used as a dependency of the Fruta project, the playground in the package can still be opened and run.(04:03) The difference is that the files in dependencies are read-only like other package files.This limitation is very important: it is suitable for browsing and learning the usage of dependencies, but not suitable for directly changing documents in dependency copies.

Build Active Scheme to let the playground use the project target

(04:35) A build is triggered when the SmoothieLab playground is opened.The reason is that Xcode 12 adds the Build Active Scheme option.When turned on, Xcode will automatically build all targets in the current scheme and provide the modules of these targets to the playground in the same workspace.

This setting directly affects the tool playground in the project.SmoothieLab in session uses the UtilityViews framework to draw nutrition charts, and can also access the NutritionFacts package.For something like a playground to work properly, you need to make sure two things are true: the target module is in the active scheme, or it is a dependency of the active scheme build target.

(06:27) Xcode 12 also puts the playground’s complete build log into the Report navigator.When the build fails, you can see which targets were built by the active scheme and which sources and resources were compiled by the playground support module.This is useful for troubleshooting “why this import is not visible in the playground”.

Playground resources support asset catalog

(08:42) Demonstrates copying the ingredients asset catalog from the Fruta project and dragging it into the playground’s resources folder.Xcode will compile the asset catalog, and the code will use the standard UIKit entry to load the image.

import UIKit
let image = UIImage(named: "ingredient/orange")

Key points:

  • UIImage(named:)Using the resource name in the asset catalog, there is no need to manually parse the file path.
  • "ingredient/orange"Corresponds to the folder and picture names in the asset catalog.
  • This code works because Xcode 12 first compiles the asset catalog into the playground support module.

Playground resources support Core ML models

(10:18) The speaker drags the YOLOv3 model into playground resources.Xcode automatically compiles the model and generates Swift classes based on the model.Playground code can instantiate this model just like project code.

import CoreML
let yoloModel = try YOLOv3(configuration: MLModelConfiguration()).model

Key points:

  • import CoreMLIntroducing the Core ML framework.
  • YOLOv3It is a Swift class generated after model compilation, and its name comes from the model class in the model details.
  • MLModelConfiguration()Use default model configuration.
  • .modelTake out the Core ML model object that Vision will wrap later.

Use Vision to turn the model into an object detection request

(10:54) After having the model and sample images, the session uses Vision to connect the two.VNCoreMLModelWrapping Core ML models,VNCoreMLRequestResponsible for performing object detection on images.

import UIKit
import CoreML
import Vision

let ingredientNames = [
    "banana",
    "orange",
    "almond-milk",
]

let yoloModel = try YOLOv3(configuration: MLModelConfiguration()).model
let model = try VNCoreMLModel(for: yoloModel)

let request = VNCoreMLRequest(model: model) {_,_ in }

Key points:

  • ingredientNamesLimit the three sample images to be tested in this playground.
  • VNCoreMLModel(for:)Convert Core ML models into models that Vision can execute.
  • VNCoreMLRequestCreate a reusable Vision request.
  • completion handler is initially left empty in the demo, the result will be retrieved laterrequest.resultsRead.

Run sample images in batches and organize the recognition results

(11:48) The next piece of code traverses the sample images, usingVNImageRequestHandlerExecute the request and thenVNRecognizedObjectObservationConvert to a data structure for SwiftUI display.

let results = ingredientNames.compactMap { ingredient -> ObjectDetectionResult? in

    guard let image = UIImage(named: "ingredient/\(ingredient)") else { return nil }

    let handler = VNImageRequestHandler(cgImage: image.cgImage!)
    try? handler.perform([request])
    let observations = request.results as! [VNRecognizedObjectObservation]

    let detectedObjects = observations.enumerated().map { (index, observation) -> RecognizedObject in

        // Select only the label with the highest confidence.
        let topLabelObservation = observation.labels[0]
        let objectBounds = VNImageRectForNormalizedRect(observation.boundingBox, Int(image.size.width), Int(image.size.height))

        return RecognizedObject(id: index, label: topLabelObservation.identifier, confidence: Double(topLabelObservation.confidence), boundingBox: objectBounds)
    }

    return ObjectDetectionResult(name: ingredient, image: image, id: ingredient, objects: detectedObjects)
}

results

Key points:

  • compactMapAllows to return when image loading failsnil, successful samples continue to enter the result array.
  • VNImageRequestHandler(cgImage:)Creates a Vision request executor with the current image.
  • handler.perform([request])Perform object detection on images.
  • request.resultsis converted to[VNRecognizedObjectObservation], each observation represents a detection object.
  • observation.labels[0]Take the label with the highest confidence.
  • VNImageRectForNormalizedRectConvert Vision’s normalized bounding box into image coordinates.

Use live view to directly view the test results

(12:33) Finally, the playground sets the SwiftUI visualizer to live view.The result area on the right will display the image, red bounding box and label confidence.In the demonstration, a banana was identified with 100% confidence, three oranges were also correctly framed, and almond milk was identified as a bottle.

import PlaygroundSupport

PlaygroundPage.current.setLiveView(
    RecognizedObjectVisualizer(withResults: results)
        .frame(width: 500, height: 800)
)

Key points:

  • PlaygroundSupportsupplyPlaygroundPage。
  • setLiveViewPlace the SwiftUI view in the playground’s live view area.
  • RecognizedObjectVisualizer(withResults:)Use the test results compiled in the previous step.
  • .frame(width: 500, height: 800)Fixed display size for visualization results.

Core Takeaways

  • What to do: Add a Swift packagePlaygroundsfolder and place the executable tutorial.Why it’s worth doing: Xcode 12 can run playgrounds in packages. People who read the documentation can immediately execute the sample code and view the live results.How ​​to start: Write the 2-3 most commonly used API calls as playground pages, ensuring that the example only relies on the public interface of the package itself.

  • What to do: Create a tool playground in the App project to debug business rules or design parameters.Why it’s worth doing: Build Active Scheme can expose the framework and package dependencies of the same workspace to the playground.How ​​to start: NewPlaygroundsfolder, create a playground, open the inspector to make sure Build Active Scheme is turned on, and then import the framework into the project.

  • What to do: Use the playground to quickly verify the resource naming and loading path of the asset catalog.Why it’s worth doing: Xcode 12 will compile playground resources,UIImage(named:)You can directly verify whether the images in the asset catalog can be loaded with the expected names.How ​​to start: Drag the target asset catalog into playground resources and write a line firstUIImage(named:), and then use Quick Look to examine the image.

  • What to do: Make an offline sample playground for Core ML or Vision features.Why it’s worth doing: Session demonstrates it.mlmodel, asset catalog, Vision request and SwiftUI live view are placed in the same playground to verify the model effect.How ​​to start: Prepare a small number of fixed test pictures, drag in the model resources, and useVNCoreMLRequestRun the test and use live view to display the results.

Comments

GitHub Issues · utterances