Fix failures faster with Xcode test reports

Highlight

Xcode 15 redesigned the Test Report, which reduces the time to locate the root cause of test failures from manual log review to just a few minutes through automatic identification of failure patterns, interactive UI test playback, and heat map views.

Core Content

Your project has thousands of tests running in multiple language configurations and multiple devices. A certain CI build fails and you are faced with a huge test matrix - which failure is a common problem? Which one is device specific? Where to start?

The Test Report of Xcode 15 is designed to solve this pain point.

(00:19) Test Report uniformly displays the results of local testing, Xcode Cloud CI testing and remote machine testing. Regardless of test size, from a single debug test to thousands of CI tests, all are organized in the same interface.

Test structure review

(01:03) Xcode’s test hierarchy from bottom to top is: Test Method (single test method) → Test Class (test class) → Test Bundle (test package, Unit or UI) → Test Plan (test plan). Test Plan can contain multiple Bundles and configure multiple running conditions (Configuration) and devices (Run Destination).

(03:11) The speaker took multi-language support as an example: Test Plan is configured with English, French, Japanese and other language configurations, and runs on three devices: iPhone 14, iPhone 14 Pro, and iPad. Each combination of “test method + configuration + equipment” generates a Test Method Run, forming a complete test result matrix.

Test Summary: See the overall situation clearly at a glance

(05:57) Test Summary provides an overall overview of the test run:

Test environment information: Displays the running configuration, device list, test plan characteristics (such as the number of repeated tests, performance indicator collection)
Heat Map: Use colors to visually display which combinations passed and which failed in a multi-device and multi-configuration matrix.
Failure Quick Entry: List all failed tests directly, no need to expand the hierarchical search

(07:05) Heatmap colors and counts help quickly compare performance between different runs. If a device fails in all configurations, the problem is most likely in the device environment rather than the code logic.

Insights: Automatically discover failure patterns

(06:11) Insights is the core highlight of Test Report, which automatically analyzes test results on all configurations and devices to discover hidden patterns:

Common Failure Patterns

(06:32) Xcode automatically groups tests with similar failure messages. In the speaker’s example, 3 tests were grouped together due to similar failure messages. After clicking Insight you can see: common failure messages, which tests were involved, and on which configurations and devices it occurred.

(08:01) This kind of grouping directly gives the repair priority - solving a common problem may fix multiple tests.

Longest Test Runs

(06:42) Lists the longest tests to help identify performance bottlenecks. This is the most effective place to start when your overall test suite is slow.

Test Details: Exclusive debugging space for a single test

(08:12) Click a Test Method Run to enter the Test Details view, which is a debugging space focused on a single test result.

For Unit Test:

Comparison of running results across all configurations and devices
Failure message and call stack (Call Stack)
The entry in the call stack can jump directly to the source code

For UI Test, debugging capabilities are stronger:

(09:50) The Test Details of UI Test contains three areas:

Test Activity

A timeline format shows each event of the test from start to finish. The top is the starting point of the test, the bottom is the end point, and each row in the middle represents an operation event.

Automation Explorer

(10:17) Video playback area linked to Test Activity. Click on any event on the timeline, and Automation Explorer displays a screenshot of the corresponding moment, allowing you to completely replay the entire test process.

Scrubber (Timeline)

(10:25) The linear timeline of the test run, marked with clicks, slides, rotations and other interactive events. Failure icons mark the precise location where the failure occurred, helping to quickly locate the problem moment.

(11:46) Click the bounding box on the UI element to view the identifier (Identifier) and hierarchy information (Hierarchy) of the element. In the speaker’s example, the “Account” button is clearly visible, but the test cannot find it - by looking at the element information, it is found that the label is written incorrectly.

Detailed Content

Configure multi-language test matrix

(02:19) Set different languages and regions in Test Plan’s Configurations:

Test Plan: Backyard Birds
├── Configuration: English (US)
├── Configuration: French (FR)
├── Configuration: Japanese (JP)
└── Run Destinations: iPhone 14, iPhone 14 Pro, iPad

Key points:

Each Configuration will run the complete test suite independently
Combine multiple Run Destination to form a complete test coverage matrix
Suitable for verifying the correctness of internationalization, localization, and regional formats (date, number, currency)

Use Insights to prioritize common issues

(07:34) When you see Insights displaying “3 tests share similar failure messages”, the process flow is:

Click Insight to view common failure messages
Check which test methods and classes are involved
See on which configurations/devices it occurs
If it fails across all configurations → code logic problem
If it only fails on a specific configuration → localization or environment issue

(08:07) The speaker’s 3 failed tests all returned exceptions because of the same API. Fixing one can solve the 3 test failures at the same time.

UI testing and debugging practice

(09:39) Debugging steps when UI Test fails:

Click the failed item in Test Summary to enter Test Details
Enter the Activities tab by default
Find the failure event marked red in the timeline
View screenshots of failed moments in Automation Explorer
Check the interaction sequence before Scrubber confirms failure
Click the bounding box of the target UI element to view the identifier and accessibility information
Compare the query conditions in the code with the actual element attributes

(12:12) The speaker found that the accessibility label of the button was written as “Account” in the code, but the actual element displayed “Settings” - the label mismatch causedXCTAssertElement not found.

Unified experience of local and CI

(12:27) Test Report provides a consistent interface across the Xcode IDE and Xcode Cloud. Problems that recur during local debugging can be analyzed in the same way in CI; failures discovered by CI can also be checked locally using the same tool chain.

Core Takeaways

1. Establish multi-configuration CI test access control

What to do: Configure a multi-language, multi-device Test Plan in the Xcode Cloud workflow, and run it automatically for each PR
Why it’s worth doing: Insights’ Common Failure Patterns can point out common problems as soon as a failure occurs and prevent problems from being merged into the main branch.
How to start: Add Language/Region Configuration in Test Plan and associate the Test Plan in Xcode Cloud workflow

2. Use heat maps to identify equipment-specific problems

What to do: Check the heat map in the Test Summary regularly to look for patterns of “systematic failure of a device”
Why it’s worth doing: Device-specific issues are usually environment or adaptation issues and have nothing to do with code logic. The color distribution of the heat map can be distinguished at a glance.
How to start: Configure multiple Run Destination in Test Plan, and view the columns of the heat map (device dimension) after CI is run.

3. Optimize slow tests to improve CI efficiency

What to do: Use Longest Test Runs Insight to find the longest tests and optimize or split them accordingly
Why it’s worth doing: CI build time directly affects development efficiency. If a test that takes 30 seconds is split into three 10-second tests, running in parallel can significantly shorten the total time.
How to get started: View the Longest Test Runs list in Insights and perform performance analysis on the top tests

4. Use UI Test playback to locate occasional bugs

What to do: For UI Tests that are prone to occasional failures, use Automation Explorer’s video playback and Scrubber timeline to analyze the pre-failure operations.
Why is it worth doing: Occasional bugs are the most difficult to reproduce. Test Report automatically records the complete testing process, eliminating the need to reproduce manually.
How to start: Make sure the test steps are clear in UI Test. After failure, view them frame by frame on the Activities tab of Test Details.

What’s new in Xcode 15 — An overview of new features in Xcode 15, including the context of Test Report
Xcode Cloud — CI/CD continuous integration, Test Report usage scenarios in the cloud
Demystify SwiftUI performance — Performance testing and analysis methods
Beyond the basics of structured concurrency — Testing strategies for concurrent code