WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Create parametric 3D room scans with RoomPlan

Create parametric 3D room scans with RoomPlan

WWDC 2022RoomPlanARKitSpatial Computing
Watch original video

Highlight

Praveen from Apple’s prototyping team and Kai from the video engineering team jointly introduce the new RoomPlan framework. RoomPlan uses ARKit and machine learning algorithms to scan a room on a LiDAR-equipped device, automatically detecting walls, windows, openings, doors, and important objects in the room (fireplaces, couches, tables, cabinets, etc.).

Core Content

When doing interior design, property showings, or furniture previews, developers often need a room model. Previously, this often meant handling cameras, depth data, geometry reconstruction, object recognition and 3D export yourself. Each step requires specialized machine learning or computer vision capabilities.

RoomPlan puts this process into a framework. It runs on an iPhone or iPad equipped with LiDAR and uses the spatial understanding capabilities and machine learning models provided by ARKit to generate a parametric 3D model of the room. (01:01)

Parameterization is the key word of this speech. RoomPlan’s output contains semantic structure rather than an incomprehensible grid. It identifies walls, doors, windows, openings, and room-defining objects such as sofas, tables, beds, cabinets, etc., and organizes them intoCapturedRoomstructure. (01:21)

Apple offers a two-level entrance. (02:57)

The first layer isRoomCaptureView. It’s the out-of-the-box scanning interface responsible for live contours, bottom 3D preview and text guidance. Suitable for apps that want to quickly access the scanning experience.

The second layer isRoomCaptureSession. It exposes real-time parameterized data and user instructions. Suitable for apps that need to customize the AR interface and overlay scanning results into their own visualization system.

Detailed Content

Use RoomCaptureView to quickly access scanning

（03:30）RoomCaptureViewis aUIViewsubclass. It handles scan feedback in world space, real-time room model generation, coaching, and user prompts.

Access scanning only needs to keep a view reference and a session configuration, and then start or stop the capture session. (04:36)

import UIKit
import RoomPlan

class RoomCaptureViewController: UIViewController {

    var roomCaptureView: RoomCaptureView
    var captureSessionConfig: RoomCaptureSession.Configuration

    private func startSession() {
        roomCaptureView.captureSession.run(configuration: captureSessionConfig)
    }

    private func stopSession() {
        roomCaptureView.captureSession.stop()
    }
}

Key points:

  • import RoomPlanIntroducing the RoomPlan framework. -RoomCaptureViewHosts the scanning interface provided by Apple. -RoomCaptureSession.ConfigurationSave the configuration for this scan. -run(configuration:)Starts scanning and passes configuration to the internal capture session. -stop()The scan ends, after which the framework can enter the processing and results presentation phase.

The value behind this code is saving yourself from having to write an entire scanning UI. RoomCaptureView displays detected walls, windows, doors, openings and object outlines during the scan, and also displays the scan progress with an underlying 3D model. (04:01)

Process scan results and export USDZ

(05:00) If the App needs to take over the result processing, it can be achievedRoomCaptureViewDelegate. The speech showed two callbacks: one to decide whether to display the post-processing results, and one to get the finalCapturedRoom。

import UIKit
import RoomPlan

class RoomCaptureViewController: UIViewController {
    func captureView(shouldPresent roomDataForProcessing: CapturedRoomData, error: Error?) -> Bool {
        // Optionally opt out of post processed scan results.
        return false
    }

    func captureView(didPresent processedResult: CapturedRoom, error: Error?) {
        // Handle final, post processed results and optional error.
        // Export processedResults
        try? processedResult.export(to: destinationURL)
    }
}

Key points:

  • CapturedRoomDataIt is data obtained by scanning and waiting for post-processing. -captureView(shouldPresent:error:)returnfalse, the App can choose not to display the post-processing result interface provided by the framework. -CapturedRoomis the post-processed parametric room model. -export(to:)Export the results to a specified URL, which can be used in USDZ workflows.
  • in codedestinationURLIt needs to be prepared by the app itself, such as writing the file path in the application sandbox.

This step allows RoomPlan to tap into existing 3D toolchains. It was mentioned in the second half of the speech that the exported USD or USDZ can be used to view the hierarchy, size and position in tools such as Cinema 4D, and can also be entered into the rendering process of real estate, e-commerce, tools and interior design applications. (12:12)

Customize the scanning experience with RoomCaptureSession

(05:40) When you need to customize the visual presentation, you can use the Data API directly. This process is divided into three steps: scan, process, and export.

Used during scanning phaseRoomCaptureSession. It also provides the underlyingARSession, allowing the app to run its ownARViewDraw planes and object bounding boxes in . (06:50)

import UIKit
import RealityKit
import RoomPlan
import ARKit

class ViewController: UIViewController {
    @IBOutlet weak var arView: ARView!
    var previewVisualizer: Visualizer!

    lazy var captureSession: RoomCaptureSession = {
        let captureSession = RoomCaptureSession()
        arView.session = captureSession.arSession
        return captureSession
    }()

    override func viewDidLoad() {
        super.viewDidLoad()
        captureSession.delegate = self
        // set up previewVisualizer
    }
}

Key points:

  • RealityKitsupplyARView, used to host custom AR images. -RoomCaptureSession()Create a low-level scan session. -captureSession.arSessionExpose the AR session used by RoomPlan. -arView.session = captureSession.arSessionLet the custom ARView use the same AR session. -captureSession.delegate = selfHave the current controller receive real-time room updates and scan commands.

The core difference here is control. RoomCaptureView gives you a complete interface; RoomCaptureSession gives you data and events. A custom app can draw RoomPlan’s real-time results into its own brand interface, or place it in the same scene as existing AR content.

Receive real-time room models and scan commands

（07:40）RoomCaptureSessionDelegateTwo types of real-time feedback are provided:CapturedRoomupdate andInstructioninstruction.

extension ViewController: RoomCaptureSessionDelegate {

    func captureSession(_ session: RoomCaptureSession,
                        didUpdate room: CapturedRoom) {
        previewVisualizer.update(model: room)
    }

    func captureSession(_ session: RoomCaptureSession,
                        didProvide instruction: Instruction) {
        previewVisualizer.provide(instruction)
    }
}

Key points:

  • didUpdate room:Called when a room change is detected. -CapturedRoomis real-time parametric room data that can be used to update the preview model. -previewVisualizer.update(model:)Represents the App’s own visual logic. -didProvide instruction:Provides user guidance during the scanning process. -InstructionDistance covered, scan speed, lighting adjustments, and texture areas of concern.

Instructions are important. The scanning quality of RoomPlan is related to the way people move, distance, speed, and light. The framework exposes this feedback, and developers can turn it into UI copy, icons, or voice prompts. (08:33)

Use RoomBuilder to generate the final model

(09:04) After the scan is completed,RoomBuilderResponsible for processingCapturedRoomData, generate the finalCapturedRoom. The speech examples used.beautifyObjectsoptions.

import UIKit
import RealityKit
import RoomPlan
import ARKit

class ViewController: UIViewController {

    @IBOutlet weak var arView: ARView!
    var previewVisualizer: Visualizer!

    // set up RoomBuilder
    var roomBuilder = RoomBuilder(options: [.beautifyObjects])
}

Key points:

  • RoomBuilderIt is the entry point for post-processing of scanned data. -options: [.beautifyObjects]Use the object beautification options provided by RoomPlan. -RoomBuilderUsually andRoomCaptureSessionused together. -previewVisualizerResponsible for presenting the final model after processing.

whenRoomCaptureSessionWhen stopped or an error occurs, the delegate receivesCapturedRoomDataand optional errors. The final room model can then be generated asynchronously using Swift concurrency. (09:30)

extension ViewController: RoomCaptureSessionDelegate {
    func captureSession(_ session: RoomCaptureSession,
                        didEndWith data: CapturedRoomData, error: Error?) {
        if let error = error {
            print("Error: \(error)")
        }

        Task {
            let finalRoom = try! await roomBuilder.capturedRoom(from: data)
            previewVisualizer.update(model: finalRoom)
        }
    }
}

Key points:

  • didEndWith data:Triggered when scanning stops or an error occurs. -CapturedRoomDataSave sensor scan data for post-processing. -TaskCreate an asynchronous context. -await roomBuilder.capturedRoom(from: data)Process scan data asynchronously. -finalRoomis finalCapturedRoom, can be used for display, analysis or export.

The presentation explains that this process usually only takes a few seconds. It uses Swift async/await, so post-processing is not written as a synchronous process that blocks the UI. (09:45)

Understand the structure of CapturedRoom

(10:17) The output of RoomPlan is a parameterized structure. top levelCapturedRoomDepend onSurfaceandObjectcomposition.

public struct CapturedRoom: Codable, Sendable {
    public let walls: [Surface]
    public let doors: [Surface]
    public let windows: [Surface]
    public let openings: [Surface]
    public let objects: [Object]

    public func export(to url: URL) throws

    // Surface definitions ...

    // Object definitions ...
}

Key points:

  • CapturedRoomfollowCodable, convenient for coding and persistence. -SendableIndicates that it can be passed safely across concurrency boundaries. -walls、doors、windows、openingsAllSurfacearray. -objectsSave the 3D objects in the room, described in the lecture as cuboids. -export(to:)Export parameterized results as USD or USDZ data.

SurfaceContains architectural elements such as walls, doors, windows, openings, etc. The speech mentioned that it has surface-related properties, such as radius, starting and ending angles, and four sides; as well as size, confidence, 3D transform matrix, and unique identifier. （10:26）

ObjectIndicates furniture categories, such as table, bed, sofa. It also comes with dimensions, confidence, 3D transform matrix and unique identifier. (10:49)

Scanning conditions will affect the results

(13:09) RoomPlan is best suited for a single residential room. The upper limit of the room given in the presentation was 30 feet by 30 feet, approximately 9 meters by 9 meters.

Light can also affect scanning. Apple recommends at least 50 lux, which is about the typical brightness of a home living room at night. Hardware-wise, RoomPlan supports all LiDAR-equipped iPhone and iPad Pro models. (13:38)

Some environments present challenges: full-height mirrors, glass, high ceilings, and very dark surfaces. The reasons come from LiDAR expected output, LiDAR scan distance, and video stream quality respectively. (14:02)

You can also prepare the room before scanning. Opening curtains increases natural light and reduces window occlusion; closing doors reduces the probability of scanning areas outside the room. High-precision scenarios should put these preparation steps into the pre-scan prompts. (14:27)

Finally, there is power and heat dissipation. Apple recommends avoiding repeated scans, or single scans that are longer than 5 minutes. Scanning for long periods of time can cause fatigue, power consumption, and thermal issues, which in turn affects the app experience. (15:16)

Core Takeaways

  • What to do: Add an automatic room modeling portal to the interior design app. Why it’s worth it: RoomPlan recognizes walls, windows, doors, openings and key furniture, and design tools can be based directly onCapturedRoomMake wall color changes, area estimates and layout previews. How ​​to start: Use firstRoomCaptureViewComplete the scanning loop, and thencaptureView(didPresent:error:)Read inCapturedRoom。

  • What to do: Generate room 3D models and USDZ files for real estate agency tools. Why it’s worth doing: The presentation explicitly mentioned real estate apps that allow agents to capture floor plans and 3D models of listings. How ​​to start: Called after the scan is completedCapturedRoom.export(to:), save the result to the App file directory, and then upload it to the business backend or local preview.

  • What to do: Make a custom scan boot interface. Why it’s worth doing:InstructionFeedback on distance, speed, lighting and texture areas will be given, and the app can turn it into guidance that better fits the tone of the product. How ​​to get started: UseRoomCaptureSessionDelegate,existdidProvide instruction:Update your own prompt component.

  • What to do: Combine e-commerce furniture previews with real room structures. Why it’s worth doing: The walls, openings, and furniture locations output by RoomPlan can provide spatial context for product placement. How ​​to start: UseRoomCaptureSessionGet real timeCapturedRoom, put the product model into the scanned room structure in the RealityKit scene.

  • What to do: Design a room preparation process for high-precision scanning. Why it’s worth doing: The talk gives limits on lighting, mirrors, glass, doors, and scan duration. Early booting can reduce failed scans. How ​​to start: At startuprun(configuration:)Add a checklist beforehand: open the curtains, close the door, remind to avoid mirrors and glass, and remind a single scan to be within 5 minutes.

Comments

GitHub Issues · utterances