WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Explore ARKit 4

Explore ARKit 4

WWDC 2020ARKitAugmented Reality
Watch original video

Highlight

ARKit 4 adds geo-location anchors, LiDAR scene depth, improved ray casting, and facial tracking for wider device coverage, allowing AR content to bind real coordinates, read meter-level depth, and complete object placement faster.

Core Content

In the past, the biggest trouble when doing outdoor AR was coordinates. GPS can tell the app roughly where it is, but it cannot place a virtual sign stably in front of a building on the street corner. Developers also have to deal with the conversion between local AR coordinates and geographic coordinates, and virtual content may drift if the user takes a few steps.

ARKit 4 brings this problem into the framework. The developer gives the latitude and longitude and optional altitude,ARGeoTrackingConfigurationIt will combine high-precision map data from Apple Maps, camera images and device motion data to complete visual positioning. What the App obtains is a globally aware AR pose, and virtual content can be fixed at the location in the real world.

The pain points of indoor AR are also very straightforward. Previously, you had to wait for plane detection to stabilize, and featureless surfaces like white walls would slow down the placement process. ARKit 3.5 first uses LiDAR to bring scene geometry, and iOS 14 opens the Depth API. Developers can get the depth map of each frame, or let raycasting automatically use the depth or scene mesh to place objects on the real surface faster.

The main line of this Session is clear: ARKit 4 advances AR from “local coordinates in the room” to “position, depth, and surface in the real world.” Location anchors solve outdoor shared locations, Depth API solves real scene distance, raycasting solves placement speed, and Face Tracking extends facial AR to more A12 or newer chip devices.

Detailed Content

Geolocation anchor: first confirm the device and location

(06:58) Position anchor fromARGeoTrackingConfigurationstart. Session explicitly requires two things to be checked first: whether the device supports geo tracking, and whether the current location has enough map data for visual positioning.

// Check device support for geo-tracking
guard ARGeoTrackingConfiguration.isSupported else {
    // Geo-tracking not supported on this device
    return
}

// Check current location is supported for geo-tracking
ARGeoTrackingConfiguration.checkAvailability { (available, error) in
    guard available else {
        // Geo-tracking not supported at current location
        return
    }
    // Run ARSession
    let arView = ARView()
    arView.session.run(ARGeoTrackingConfiguration())
}

Key points:

  • ARGeoTrackingConfiguration.isSupportedBlocking out unsupported hardware first, Session mentioned that the location anchor requires an A12 Bionic or newer chip and GPS. -checkAvailabilityChecks if the current location has Apple Maps data for location.
  • After passing the inspection,ARViewThe session will runARGeoTrackingConfiguration().
  • ARKit will request camera and location permissions when the session starts, and should explain the error to the user if it fails.

Use ARGeoAnchor to bind real latitude and longitude

(08:38) Session uses the Ferry Building in San Francisco as an example. Created by developersCLLocationCoordinate2D, create againARGeoAnchor, and finally hang the RealityKit entity to this anchor point.

// Create coordinates
let coordinate = CLLocationCoordinate2D(latitude: 37.795313, longitude: -122.393792)

// Create Location Anchor
let geoAnchor = ARGeoAnchor(name: "Ferry Building", coordinate: coordinate)

// Add Location Anchor to session
arView.session.add(anchor: geoAnchor)

// Create a RealityKit anchor entity
let geoAnchorEntity = AnchorEntity(anchor: geoAnchor)

// Anchor content under the RealityKit anchor
geoAnchorEntity.addChild(generateSignEntity())

// Add the RealityKit anchor to the scene
arView.scene.addAnchor(geoAnchorEntity)

Key points:

  • CLLocationCoordinate2DUse at least 6 decimal places of precision, which Session uses to ensure content is placed exactly where it should be. -ARGeoAnchor(name:coordinate:)When only the latitude and longitude is passed, ARKit will use the map data to infer the ground elevation. -arView.session.add(anchor:)Let ARKit manage geo-anchors. -AnchorEntity(anchor:)Let RealityKit content follow this ARKit anchor. -generateSignEntity()Represents virtual signage to be displayed, and the actual app can be replaced with a mockup, instructional signage, or navigational prompt.

The anchor point direction is fixed, and the content transformation is handed over to the rendering layer.

（10:32）ARGeoAnchorThe coordinate axes are fixed to the orientation: the X-axis points east, the Z-axis points south, and the Y-axis points upward. The anchor point itself is immutable, rotating and raising the content is done on RealityKit entities.

// Create a new entity for our virtual content
let signEntity = generateSignEntity();

// Add the virtual content entity to the Geo Anchor entity
geoAnchorEntity.addChild(signEntity)

// Rotate text to face the city
let orientation = simd_quatf.init(angle: -Float.pi / 3.5, axis: SIMD3<Float>(0, 1, 0))
signEntity.setOrientation(orientation, relativeTo: geoAnchorEntity)

// Elevate text to 35 meters above ground level
let position = SIMD3<Float>(0, 35, 0)
signEntity.setPosition(position, relativeTo: geoAnchorEntity)

Key points:

  • generateSignEntity()Create the content entity to be displayed. -geoAnchorEntity.addChild(signEntity)Places the content in the local coordinate system of the geographic anchor. -simd_quatfRotate around the Y axis so the text faces the city. -setOrientation(_:relativeTo:)Set the orientation relative to a geographic anchor point. -SIMD3<Float>(0, 35, 0)Elevate the signage by 35 meters for long-distance identification.

User clicks can also generate geographical coordinates

(14:08) Geographic coordinates do not necessarily come from a database. After the user taps the screen, the App can check the geographical location from the ARKit world coordinates and save it as a new location anchor.

let session = ARSession()
let worldPosition = raycastLocationFromUserTap()
session.getGeoLocation(forPoint: worldPosition) { (location, altitude, error) in
    if let error = error {
        ...
    }
    let geoAnchor = ARGeoAnchor(coordinate: location, altitude: altitude)
}

Key points:

  • raycastLocationFromUserTap()Represents the AR world coordinates obtained from the user’s click position. Session mentioned that it can come from a raycast or a point on the plane. -getGeoLocation(forPoint:)Convert ARKit world points to geolocation and elevation.
  • in the callbackerrorTo handle, incorrect anchor points cannot be saved when positioning fails. -ARGeoAnchor(coordinate:altitude:)Suitable for saving user-created points of interest.

Depth API: Read meter-level depth and confidence per frame

(20:32) On LiDAR devices, ARKit 4 is exposedsceneDepthframe semantics. When enabled, eachARFrameAll can be readARDepthData, which containsdepthMapandconfidenceMap。

// Enabling the depth API

let session = ARSession()
let configuration = ARWorldTrackingConfiguration()

// Check if configuration and device supports .sceneDepth
if type(of: configuration).supportsFrameSemantics(.sceneDepth) {
    // Activate sceneDepth
    configuration.frameSemantics = .sceneDepth
}
session.run(configuration)

...

// Accessing depth data
func session(_ session: ARSession, didUpdate frame: ARFrame) {
    guard let depthData = frame.sceneDepth else { return }
    // Use depth data
}

Key points:

  • ARWorldTrackingConfiguration()Still the entrance to normal world tracking. -supportsFrameSemantics(.sceneDepth)Check if the device supports LiDAR scene depth. -configuration.frameSemantics = .sceneDepthTurn on depth output. -frame.sceneDepthreturnARDepthData, where the unit of depth pixels is meters.
  • Session mentioned that the depth map has the same aspect ratio as the camera image, but the resolution is smaller.

Character occlusion and depth can be shared

(21:12) If the App has been usedpersonSegmentationWithDepth,supportsceneDepthThe device will automatically provide scene depth, and Session explains that this will not cause additional power consumption.

// Using the depth API alongside person occlusion

let session = ARSession()
let configuration = ARWorldTrackingConfiguration()

// Set required frame semantics
let semantics: ARConfiguration.FrameSemantics = .personSegmentationWithDepth

// Check if configuration and device supports the required semantics
if type(of: configuration).supportsFrameSemantics(semantics) {
    // Activate .personSegmentationWithDepth
    configuration.frameSemantics = semantics
}
session.run(configuration)

Key points:

  • personSegmentationWithDepthIt is the frame semantic required for character occlusion. -supportsFrameSemantics(semantics)Still a device capability check before enabling.
  • After turning it on, the App can continue to use character occlusion and read it on supported devicessceneDepth.
  • This is suitable for placing virtual objects between people and the real environment to enhance the occlusion relationship.

Raycasting: Replace old hit-testing with queries

(25:41) ARKit 4 recommends using raycasting for object placement. On LiDAR devices, raycasting will use scene depth or scene geometry to give faster results on surfaces that lack features such as white walls.

let session = ARSession()
hitTest(point, types: [.existingPlaneUsingGeometry,
                       .estimatedVerticalPlane,
                       .estimatedHorizontalPlane])

let query = arView.makeRaycastQuery(from: point,
                                    allowing: .estimatedPlane,
                                    alignment: .any)

let raycast = session.trackedRaycast(query) { results in
   // result updates
}

Key points:

  • The first half shows the oldhitTestWriting methods usually require developers to filter multiple results themselves. -makeRaycastQuery(from:allowing:alignment:)Use a query to describe the ray, target surface, and alignment. -.estimatedPlaneAllows ARKit to estimate the surface based on its current understanding. -.anySurfaces with different orientations such as horizontal and vertical are allowed. -trackedRaycastThe results will be continuously called back as ARKit updates its understanding of the scene.

Core Takeaways

  • What to do: Make an outdoor landmark navigation app. Users will see floating signboards when they walk near the building. Why it’s worth doing: Location anchors can bind content to latitude and longitude, and ARKit is responsible for visual positioning and local coordinate fusion. How ​​to start: UseARGeoTrackingConfiguration.checkAvailabilityCheck the location before usingARGeoAnchor(name:coordinate:)Place each landmark.

  • What to do: Make an AR road sign system for a store or exhibition hall, marking the entrance, service desk and exit in the real space. Why it’s worth doing:ARGeoTrackingStatusIt will tell you whether you are currently initializing, positioning, or already positioned. It is suitable for guiding the user to lift the device and look towards the building. How ​​to start: After starting geo tracking, continue to monitor the status, and only display navigation content when it is localized and the accuracy reaches the standard.

  • What to do: Make a LiDAR room scan preview, convert the depth of each frame into a point cloud, and display the real scene in color. Why it’s worth doing:frame.sceneDepthProviding depth map and confidence map, the example of Session is to back-project depth into 3D point cloud. How ​​to start: Enable.sceneDepth,existsession(_:didUpdate:)read inARDepthData, filtering low-quality points by confidence.

  • What to do: Make a more reliable AR furniture placement function, which can be quickly placed by clicking on a white wall or empty space. Why it’s worth doing: ARKit 4’s raycasting will take advantage of scene depth or scene geometry when available, reducing custom filtering behind legacy hit-testing. How ​​to start: UsearView.makeRaycastQuery(from:allowing:alignment:)Create a query and use itsession.trackedRaycastTrack placement results.

  • What to do: Make a face sticker or live emoticon character feature to reach more devices without TrueDepth. Why it’s worth it: ARKit 4 extends Face Tracking to devices with front-facing cameras using the A12 Bionic or newer chip. How ​​to get started: Check out Face Tracking by device capabilities, using face anchors, face geometry and blendshapes to drive virtual content.

Comments

GitHub Issues · utterances