Qualities of great AR experiences

Highlight

AR experiences are designed very differently than 2D interfaces. Session first provides a judgment framework: What scenarios are suitable for AR? The answer lies in four directions - real-size display (how big the dinosaur is), physical space participation (wall color change preview), 3D visualization (furniture placement, glasses try-on), action simplification (scanning is measurement).

---

Core Content

AR is most vulnerable to abuse. Putting an ordinary list, form, or purchase process into the camera screen will usually only make people more tired. Users have to lift their phones, move their bodies, and understand the relationship between virtual objects and the real environment.

This Session first gives the judgment criteria. AR is suitable for four types of tasks: showing true dimensions, allowing physical space to participate in the experience, using 3D to help understand or try out objects, and using the real world as input to reduce operations.

This judgment is very important. Monster Park uses dinosaurs to give people a direct sense of their size. Color Snap turns your wall into a canvas for color testing. Warby Parker lets users try glasses on their faces. IKEA Place uses ARKit’s built-in lights and shadows to bring furniture into a room. Measure App points the camera at the person and displays the height.

Once you decide you want to make AR, the design problem becomes another set of constraints: the environment may be too dark, the surface may not have texture, the user’s arms may tire, virtual objects may run off the screen, and text may be difficult to read in a 3D world.

Therefore, a good AR experience is not about moving a 2D interface into space. It teaches users to prepare the environment, keep important text in screen space, support movement with sound, tactile and visual feedback, and control the experience within a minute or two.

Detailed Content

1. First determine whether AR brings real value

(01:26) The first judgment criterion given by Session is whether AR can provide a real representation. The size of a dinosaur, the dimensions of furniture, and trying on glasses are all difficult to accurately convey using words or flat images alone.

The requirements can be written as this judgment list:

• Real Size: Does the user need to feel the proportion of the object to the space he or she is in? Monster Park’s T. rex example falls into this category.
• Physical Space Engagement: Do users need to make decisions in their own rooms, walls, or desktops? Color Snap’s wall color test falls into this category.
• 3D Visualization: Does the user need to understand or evaluate an object from multiple perspectives? Warby Parker and IKEA Place fall into this category.
• Action Simplification: Can users directly use the real world as input? Measure App’s height measurement falls into this category.

Key points:

• Real size solves the “imagination cost”. Users don’t have to convert numbers into physical sensations.
• Physical space is involved in solving the “loss of context”. What users see is their own environment.
• 3D visualization addresses “Buying or Judgment Confidence”. Users can rotate, approach, and compare.
• Action simplification solves the “input cost”. Camera movement becomes the interaction itself.

2. Use boot sequences to deal with environmental issues

(06:18) AR relies on the real environment. The user may be standing next to glass, in front of a white wall, or in a poorly lit room. Session uses Mission to Mars’ onboarding instructions: Before starting, quickly tell users how to get a better AR session.

This guide contains three facts:

• Find a safe location. AR can be very immersive, and wayfinding-type experiences especially don’t make people stare at the screen for long periods of time.
• Find a textured surface. AR performs poorly on glass or smooth white surfaces.
• Find a bright space. AR works easier when there’s enough light.

Key points:

• Keep the lead short. Users come to experience AR, not read instructions.
• Safety reminders should be placed at the beginning. It is part of the boundary of experience.
• For LiDAR-enabled devices, some surface texture alerts can be reduced since LiDAR can alleviate some difficult environments.
• The goal of guidance is to let users know what to do next, without explaining technical principles.

3. Place text and buttons in screen space

(07:33) Screen space is the 2D layer overlaid on top of the camera view. Session explicitly recommends that text and buttons be placed first in screen space, rather than in the 3D world.

The reason is straightforward. The camera image will move, the real background color is uncontrollable, and 3D text will be affected by angle and distance. When users read on handheld devices, they need stable, high-contrast, and sufficiently large text.

You can use these rules when designing:

• Description text is placed in screen space.
• Buttons are placed in screen space.
• High contrast text and buttons overlaid on the 3D scene.
• If text must be bound to an object in the world, try to make it billboarded and keep it facing or parallel to the screen.
• Increase text contrast, font size, and background to maintain accessibility.

Key points:

• The main screen of AR is 3D, but readability still relies on 2D interface principles.
• Text can update as the camera moves, but it doesn’t have to rotate with the 3D object. -Screen space is suitable for carrying high-frequency information such as “what to do now”.

4. Provide instant feedback for continuous movement

(08:32) AR is not a static interface. Users move their phones and sometimes their bodies. The design must tell users when to move, how to move, and what happens after they move.

Session suggests using visuals and sounds to provide real-time feedback. Even if the action occurs outside the current field of view, the user can understand the state of the experience through feedback. Captions should be short, scannable, and appear as needed.

The example of DoodleLens is very specific: it uses an animation of the phone panning back and forth to tell the user how to aim the doodle. This animation is bound to the App scene and is easier to understand than general text.

Key points:

• Do not display all mobile instructions at once. Appears as per current stage.
• Animations are better for explaining gestures and body movements than long sentences.
• Sound and visual feedback can connect to actions happening off-screen.
• You can use the system’s built-in coaching animations, or you can provide guidance for your own app that is more suitable for the scene.

5. Address ergonomics, field of view and depth

(09:19) The interaction cost of AR is higher than that of ordinary apps. Users have to extend their arms, keep their eyes at a distance from the screen, and touch with their thumbs. Session therefore emphasizes that the interface should be suitable for one hand, suitable for long-arm reading, and suitable for completion in a short time.

Buttons should be large, icons should be high-contrast, and there should be few interaction steps. DoodleLens places the buttons at the bottom of the screen, making them easily reachable by your thumb.

(09:52) Handheld devices also have a limited field of view. Large virtual objects may not fit on the screen, and an object may be off-screen. The AR Quick Look example allows for zooming on an object using pinch gestures and gives haptic feedback when it exceeds 100%. The RoomPlan example provides a small 3D model preview at the bottom to help the user know what has been scanned.

(10:55) The sense of depth also needs to be designed. Object size, perspective, shadows, lighting, texture details, and occlusion relationships will all affect whether users believe that virtual objects are in real space. Session specifically mentions occlusion. In AR Quick Look, the virtual aircraft is partially blocked by a wooden block on the table, so the user can judge that it is behind the wooden block.

Key points:

• The AR button is prioritized where it is easily accessible to the thumb.
• Allow scaling when the object is too large.
• When the target leaves the field of view, use sound, touch, indicators or mini-maps to help the user find their way back.
• Shadows, lighting and occlusion are not decorations, they directly affect spatial judgment.
• A single AR experience is best limited to one or two minutes; longer experiences should provide chapters or rest points.

Core Takeaways

1. Make a “space verification before purchase” function

• What to do: Let users place furniture, lighting, and fitness equipment into their rooms, checking size, placement, and style.
• Why it’s worth doing: Session mentioned that IKEA Place uses ARKit’s built-in lighting and shadows to make virtual furniture fall more naturally in the room.
• How ​​to get started: First support the placement, rotation and scaling of a single USDZ model, then add shadows, lighting and screen space descriptions.

2. Create a “realistic surface color test” function

• What to do: The user points at the wall or object surface, selects the color and previews the effect directly.
• Why it’s worth it: Color Snap’s example shows that users can see the results from different angles in their own rooms, which is more accurate than looking at a color chart or sample image.
• How ​​to start: First do the minimal process of wall selection and color coverage, then add lighting cues, guidance on re-selecting surfaces, and before-and-after comparisons.

3. Make a “Short Scan Checklist”

• What to do: Break down the RoomPlan or similar scanning process into short steps: find bright spots, move device, confirm scan range, preview results.
• Why it’s worth doing: Session repeatedly emphasized that AR requires environmental guidance, motion feedback and limited field of view assistance.
• How ​​to get started: Place instructions in screen space, animate how to move, and show the scanned area at the bottom or a simplified 3D preview.

4. Make a “lightweight reality input” tool

• What to do: Point the camera at real objects to automatically generate measurements, recognition or shortcut operations.
• Why it’s worth doing: Measure App’s height measurement demonstrates the value of AR as a supplementary function: a small amount of UI, direct actions, and the information disappears after reading it.
• How ​​to get started: Select a single input target, such as height, package size, device label, and overlay the result in screen space with short text.

Related Sessions

• Discover ARKit 6 — ARKit 6 provides camera, tracking, planar anchor, and position anchor capabilities and is the underlying technology theme to achieve high-quality AR experiences.
• Create parametric 3D room scans with RoomPlan — RoomPlan converts a real room into a parametric 3D scan, which is directly related to the spatial guidance, scanning feedback, and field of view restrictions mentioned in this field.
• Bring your world into augmented reality — This speech shows how Object Capture and RealityKit can bring real objects into AR, which is an implementation-side supplement to the design principles of this field.
• Understand USD fundamentals — USD is an important format for Apple’s 3D and AR content pipeline and is suitable for continuing to understand how AR assets are organized and delivered.