WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Discover ARKit 6

Discover ARKit 6

WWDC 2022ARKitRealityKitSpatial Computing
Watch original video

Highlight

ARKit 6 adds 4K 30fps video, running high-resolution photos, HDR, low-level AVCaptureDevice configuration, stable plane anchors, Motion Capture improvements and more Location Anchors support, allowing AR apps to get both higher image quality and finer camera control.

Core Content

AR apps always have to make trade-offs between image quality, tracking, and performance. Film and television previews, virtual production and photogrammetry require more pixels; games and interactive experiences require higher frame rates; scanning and post-processing also hope to obtain the original information of the camera.

The updates to ARKit 6 focus on this trade-off. It provides 4K video and high-resolution single-frame photos for high-definition scenes, HDR for high-contrast scenes, and exposes apps that need to customize image styles.AVCaptureDevice, while continuing to advance planar anchors, body tracking and geographical anchors.

This talk is not just about APIs. It first explains how ARKit used binning (pixel merging) to downgrade camera images to HD in exchange for 60fps and lower memory usage; and then explains how ARKit 6 skips this step on new hardware and allows apps to use 4K images directly. (02:25)

For developers, the key judgment is very practical: if the app is used for film and television, virtual production, object capture or photography, priority should be given to new image quality capabilities; if the app is a game or an interactive experience that relies on high refresh rates, HD 60fps may still be more suitable. (06:26)

Detailed Content

4K Video Mode: Prepare for pixel-hungry scenes

(04:10) ARKit 6 can skip binning and directly use the 3840×2160 image area to output 4K video at 30fps. RealityKit will still handle scaling, cropping, and rendering; apps that write their own Metal renderers can continue to useARSession.currentFrame.capturedImageGet the screen.

The speech mentioned that to enable 4K, you must first query whether the configuration supports 4K video format, then assign it to the configuration and run the session. (04:49) The following is the minimum call method organized according to the API name in the verbatim draft:

let config = ARWorldTrackingConfiguration()

if let format = ARWorldTrackingConfiguration.recommendedVideoFormatFor4KResolution {
    config.videoFormat = format
}

session.run(config)

Key points:

  • ARWorldTrackingConfiguration()Create a world tracking configuration. -recommendedVideoFormatFor4KResolutionReturns 4K video format when the device and configuration support 4K. -if letHandle devices that don’t support 4K; lecture notes will return if they don’t support itnil。
  • config.videoFormat = formatSwitch the session to 4K video format. -session.run(config)Start an AR session with the updated configuration.

4K mode is available on iPhone 11 and later, as well as iPad Pro with the M1 chip. Its output is 3840×2160, 30fps. When the iPad is displayed full screen, the final rendering may look more magnified because the 16:9 frame is cropped on both sides. (05:25)

There’s another memory rule when using 4K: don’t hold onto it for long periods of timeARFrame. 4K frames take up more memory, and retaining them for too long will prevent the system from releasing the image, which may cause rendering frames to drop, and ultimatelyARCameratracking state returns limited. You can check whether too many images are being held via the Console warning. (05:53)

High-resolution background photos: take a single photo without interrupting the ARSession

(07:30) ARKit 6 allows apps to request a high-resolution photo on demand while an ARSession is running. The video stream is not interrupted and there is no need to start another oneAVCaptureSession. On the iPhone 13, this photo uses the full 12 million pixels of the wide camera.

This is straightforward for Object Capture. The app can use AR to superimpose shooting guidance on real objects, and then trigger high-resolution photos at appropriate angles to provide better input to the photogrammetry algorithm. (09:06)

First select a video format that supports high-resolution capture:

if let hiResCaptureVideoFormat = ARWorldTrackingConfiguration.recommendedVideoFormatForHighResolutionFrameCapturing {
    // Assign the video format that supports hi-res capturing.
    config.videoFormat = hiResCaptureVideoFormat
}
// Run the session.
session.run(config)

Key points:

  • recommendedVideoFormatForHighResolutionFrameCapturingQuery the video format that supports high-resolution background photos. -if letAvoid forcing configuration on devices that do not support this capability. -config.videoFormat = hiResCaptureVideoFormatMake the current configuration use a format that supports hi-res capture. -session.run(config)Start or restart the session for this format to take effect.

When the user taps the screen, scans to the target angle, or other events in the app occur, trigger a single frame capture:

session.captureHighResolutionFrame { (frame, error) in
   if let frame = frame {
      saveHiResImage(frame.capturedImage)
   }
}

Key points:

  • captureHighResolutionFrameTriggers an out-of-band high-resolution image capture.
  • completion handler returns asynchronouslyARFrameanderror。
  • if let frame = frameConfirm that the capture was successful before accessing the image. -frame.capturedImageare high-resolution images to be saved or sent for subsequent processing. -saveHiResImageRepresents the App’s own saving logic; in real projects, it is necessary to write or process as soon as possible to avoid holding large images for a long time.

HDR: Let high-contrast environments retain more details

(11:22) HDR (High Dynamic Range) captures a wider color range and maps it to the display device. Comparative explanation of the backyard scene used for speeches: After turning on HDR, the wooden fence in the dark and the clouds in the bright can retain more details.

Enable HDR To check if the current video format is supported:

if (config.videoFormat.isVideoHDRSupported) {
    config.videoHDRAllowed = true
}
session.run(config)

Key points:

  • config.videoFormat.isVideoHDRSupportedQuery whether the current video format supports HDR.
  • The talk notes that currently only non-binned video formats support HDR. -config.videoHDRAllowed = trueAllow ARKit to use HDR. -session.run(config)Run the session with HDR settings.

HDR comes with a performance overhead. It should only be turned on in high-contrast environments or when image quality requirements are clear, such as outdoor architectural previews, film and television storyboards, and AR shooting apps that need to retain sky and shadow details. (12:22)

AVCaptureDevice: Control exposure, white balance and focus

(12:29) Some AR Apps require manual control of camera settings, such as exposure, white balance, or triggering focus events. ARKit 6 allows configuration to access the underlying layer corresponding to the primary cameraAVCaptureDevice。

if let device = ARWorldTrackingConfiguration.configurableCaptureDeviceForPrimaryCamera {
   do {
      try device.lockForConfiguration()
      // configure AVCaptureDevice settings
      …
      device.unlockForConfiguration()
   } catch {
      // error handling
      …
   }
}

Key points:

  • configurableCaptureDeviceForPrimaryCameraReturns a configurable underlyingAVCaptureDevice。
  • if let deviceHandle situations where the device or configuration does not support direct access to the camera. -try device.lockForConfiguration()Lock before modifying camera settings.
  • The annotation position can be configured with exposure, white balance, etc.AVCaptureDeviceset up. -device.unlockForConfiguration()Release the configuration lock after the modification is completed. -catchProcessing lock or configuration failure.

Exercise restraint here. ARKit’s camera footage is both a rendering background and an input for scene analysis. Settings such as strong overexposure can affect ARKit’s analysis of the scene and reduce output quality. (12:58)

ARKit 6 will also work on everyARFrameExpose EXIF ​​tags. They contain information such as white balance, exposure, etc., and can be used for post-processing or recording shooting conditions. (13:29)

Plane Anchors: Anchor point direction is more stable, geometric changes enter ARPlaneExtent

(13:53) In iOS 15, when the plane geometry is updated, the rotation of the plane anchor will also be updated accordingly. This will cause the visual content attached to the anchor to rotate accordingly.

In iOS 16, plane anchor and plane geometry updates are decoupled. The plane can continue to expand and update the geometry, but the anchor rotation remains unchanged. All plane geometry information entersARPlaneExtent,inrotationOnYAxisRepresents the rotation of the geometry on the Y axis. (14:39)

The RealityKit visualization code given in the speech is as follows:

// Create a model entity sized to the plane's extent.
let planeEntity = ModelEntity(
    mesh: .generatePlane (
        width: planeExtent.width, 
        depth: planeExtent.height),
    materials: [material])

// Orient the entity.
planeEntity.transform = Transform(
    pitch: 0, 
    yaw: planeExtent.rotationOnYAxis, 
    roll: 0)

// Center the entity on the plane.
planeEntity.transform.translation = planeAnchor.center

Key points:

  • ModelEntityCreate a solid body to display the plane. -.generatePlane(width:depth:)according toplaneExtent.widthandplaneExtent.heightGenerate a planar mesh. -materials: [material]Assign a material to this plane. -Transform(pitch:yaw:roll:)Set the entity direction. -yaw: planeExtent.rotationOnYAxisUse the geometry’s own Y-axis rotation instead of relying on anchor rotation changes. -planeEntity.transform.translation = planeAnchor.centerMove the entity to the center coordinate of the plane anchor.

Each time the plane is updated, width, height, center androtationOnYAxis. To enable this new behavior, the project deployment target needs to be set to iOS 16. (16:16)

Motion Capture and Location Anchors: Tracking continues to expand

(16:34) Motion Capture adds left and right ear joints to the 2D skeleton and improves overall pose detection. On iPhone 12 and newer models, the latest iPad Pro and M1 iPad Air, the 3D skeleton has less jitter and better temporal consistency; tracking is also more stable when people are occluded or close to the camera. To take advantage of these improvements, the deployment target needs to be set to iOS 16.

(17:29) Location Anchors added more regions. The new areas listed in the speech include Vancouver, Toronto, Montreal, and Singapore in Canada, seven metropolitan areas in Japan including Tokyo, and Melbourne and Sydney in Australia; Auckland, Tel Aviv, and Paris will also be supported later.

The usability check entry given in verbatim isARGeoTrackingConfiguration.checkAvailability：

ARGeoTrackingConfiguration.checkAvailability(at: coordinate) { available, error in
    if available {
        // Create ARGeoAnchor-based content at this coordinate.
    }
}

Key points:

  • checkAvailability(at:)Used to check whether a coordinate supports Location Anchors. -coordinateIs the location where the app wants to place or validate geo-AR content.
  • completion handler returns the availability of this coordinate. -availableWhen true, create a newARGeoAnchorof AR content.
  • This paragraph is a call example organized based on the verbatim description; specific error handling and anchor point creation should be completed according to project requirements.

Core Takeaways

  • Make an AR framing and reshooting tool: Let users go to the shooting location of historical photos, and then use AR to guide the composition. ARKit 6’s high-resolution background photos can save high-resolution photos without stopping the session; the entrance iscaptureHighResolutionFrame。

  • Object Capture Wizard: Shows angle, distance and missed shot tips around real objects. ARKit is responsible for overlaying the 3D UI, high-resolution photos provide better input for photogrammetry, and then hands the results to RealityKit or Object Capture processes.

  • Virtual Production Previewer: Use 4K AR background to synthesize virtual props or scenes on site. Query the 4K video format first, and then decide whether to use 4K 30fps or HD 60fps based on device performance.

  • Make an AR camera with controllable image style: PassedconfigurableCaptureDeviceForPrimaryCameraAdjust exposure, white balance or focus events. Suitable for brand filters, exhibitions, or film and television AR apps that require uniform tones.

  • Stable plane annotation tool: Using iOS 16ARPlaneExtentUpdate the size and rotation so that the desktop, wall or floor annotation will not be rotated by anchor rotation when the plane is expanded.

Comments

GitHub Issues · utterances