WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Embrace Expected Failures in XCTest

Embrace Expected Failures in XCTest

Developer ToolsXCTest
Watch original video

Highlight

Apple provides XCTExpectFailure in XCTest, which allows known test failures to continue and be marked as expected failures in reports, thereby reducing noise and preserving the signal of new problems.

Core Content

A test that was passing yesterday started failing today. There is value in failing the first time. It may indicate that the product has a bug, it may indicate that the test was written incorrectly, or it may indicate that the dependent framework or system behavior has changed.

The problem is, the team won’t necessarily be able to fix it right away. Each subsequent CI run will continue to report the same failure. The red result is still there, but it no longer provides new information. Real new failures will be buried within old failures.

There were two common processing methods in the past. The first is to disable testing in the test plan or scheme. The code still compiles, but the tests don’t execute and the problem is not visible in the report. The second is to use XCTSkip. The test will be executed to the skip point and recorded in the report, but the code after the skip point will not run and subsequent information will be lost.

The tool introduced by Apple in this session is XCTExpectFailure. It is suitable for handling failures that have been registered and cannot be repaired for the time being. The test runs as normal, a hit failure will be reported as an expected failure, and the test suite containing it will show a mixed status of passing. You’ll still see the problem and discover new failures faster.

Known failures still need to be linked to defects

XCTExpectFailure accepts a failure reason. The talk suggested putting the URL of the bug tracking system into this string. Xcode’s test report will recognize the URL and display a jump button when hovering to facilitate returning to the problem record from the test report.

The smaller the scope, the less noise

XCTExpectFailure has two uses. Stateful usage affects failure after the call. Scoped usage wraps potentially failing code into a closure, and failures outside the closure are reported as usual. It is more suitable to use scoped writing method in shared test tool functions, because it reduces the impact on the outer test state.

Strict mode helps you delete expired tags

The default behavior is strict. If you declare an expected failure, but there is no failure this time, XCTest will generate a new failure called unmatched expected failure. This will alert you: the underlying problem may have been fixed and the XCTExpectFailure should be removed.

For flaky tests, strict mode creates additional noise. XCTExpectedFailure.Options provides isStrict, which can be turned off. The Swift version also supports passing strict: false directly in XCTExpectFailure.

Detailed Content

1. Use XCTSkip to configure restrictions

(03:31) XCTSkip is suitable for expressing configuration constraints. The example in the talk is that the test only supports iPads. The test will be executed until this line and will be skipped if the device is not an iPad.

try XCTSkipUnless(UIDevice.current.userInterfaceIdiom == .pad, "Only supported on iPad")

Key points:

  • try：XCTSkipUnlessA skip exception will be thrown and the test method needs to handle it. -XCTSkipUnless(...): Skip the current test when the condition is false. -UIDevice.current.userInterfaceIdiom == .pad: Limit the test running environment to iPad. -"Only supported on iPad": Write the skip reason for the test report.

This API solves environment mismatches. It is not suitable for handling known failures because the test code after the skip point will not continue to run.

2. Mark known failures with XCTExpectFailure

(04:31) XCTExpectFailure is used to manage registered failures. After the call, the test continues execution and hit failures appear as expected failures.

XCTExpectFailure("<https://dev.myco.com/bugs/4923> myValidationFunction is returning false")

Key points:

  • XCTExpectFailure(...): Declares that the next failure in the current test is a known failure. -"<https://dev.myco.com/bugs/4923> ...": The failure reason contains a bug link, and the Xcode report can jump to the issue tracking system. -myValidationFunction is returning false: The reason text describes the specific failure point to avoid leaving only a context-free mark.

In the lecture, after adding this line, the red X for the failed test turns into a gray X. The test suite that contains it turns into a green dash, indicating that it passed but contained skip or expected failure.

3. Use scoped writing to limit the scope of influence

(07:14) If only certain lines will trigger known failures, put them in closures. When a failure occurs outside the closure, XCTest still handles it as a normal failure.

XCTExpectFailure("<https://dev.myco.com/bugs/4923> fix myValidationFunction") {
    XCTAssert(myValidationFunction())
}

Key points:

  • XCTExpectFailure(... ) { ... }: Creates an expected failure scope that only covers the code inside the closure. -"<https://dev.myco.com/bugs/4923> fix myValidationFunction": Log issue links and fix targets. -XCTAssert(myValidationFunction()): The true assertion is still executed and is captured by the closure scope on failure.
  • Code outside the closure: If a failure occurs, it will be reported as a normal failure.

The talk also explains that XCTExpectFailure supports nesting. Failure will first match the nearest call point, and then pass to the outer call if the matcher rejects it. The use of closure writing in the shared test library can reduce the impact on the caller’s test.

4. Use issueMatcher to accurately match failure types

(08:34) By default, any failure within the scope will be caught. In order to discover new problems, you can set issueMatcher for options and only accept XCTIssue that meets the conditions.

let options = XCTExpectedFailure.Options()
options.issueMatcher = { issue in
    return issue.type == .assertionFailure
}

XCTExpectFailure("<https://dev.myco.com/bugs/4923> fix myValidationFunction", options: options)

Key points:

  • let options = XCTExpectedFailure.Options():Create an expected failure configuration object. -options.issueMatcher = { issue in ... }: Set a failure matcher for the configuration object. -issue:XCTest passed inXCTIssue, containing failure details. -return issue.type == .assertionFailure: Only treat assertion failures as expected failures. -XCTExpectFailure(..., options: options): Pass matching rules to expected failure statements.

If a matcher rejects a failure, the failure is not treated as an expected failure. In this way, old questions continue to reduce noise, and new questions can still turn the test red.

5. Close expected failure by platform

(09:03) Some failures only occur on specific platforms. The example given in the talk is that the test passes on macOS and fails on iOS. Expected failure assertions on some platforms can be turned off via options.isEnabled.

let options = XCTExpectedFailure.Options()
#if os(macOS)
options.isEnabled = false
#endif

XCTExpectFailure("<https://dev.myco.com/bugs/4923> fix myValidationFunction", options: options) {
    XCTAssert(myValidationFunction())
}

Key points:

  • let options = XCTExpectedFailure.Options(): Prepare configuration object. -#if os(macOS): Only perform configuration within the macOS build branch. -options.isEnabled = false: Turn off this expected failure statement on macOS. -#endif: End platform conditional compilation. -XCTExpectFailure(..., options: options) { ... }: Use configuration to run assertions within a closure. -XCTAssert(myValidationFunction()):Failure will only be caught on platforms where expected failure is enabled.

This way of writing is suitable for deterministic environmental differences, such as differences in platforms, device types, or system versions.

6. Turn off strict behavior for unstable tests

(10:39) XCTExpectFailure defaults to strict. XCTest reports unmatched expected failure when an expected failure is declared but not caught. This behavior is suitable for stable failures because it prompts you to remove expired tags.

let options = XCTExpectedFailure.Options()
options.isStrict = false

XCTExpectFailure("<https://dev.myco.com/bugs/4923> fix myValidationFunction", options: options) {
    XCTAssert(myValidationFunction())
}

Key points:

  • let options = XCTExpectedFailure.Options():Create a configuration object. -options.isStrict = false: Turn off strict behavior. -XCTExpectFailure(..., options: options) { ... }: Run expected failure scope with non-strict configuration. -XCTAssert(myValidationFunction()): When the assertion passes this time, the test still passes.

Swift also provides direct parameter writing.

XCTExpectFailure("<https://dev.myco.com/bugs/4923> fix myValidationFunction", strict: false) {
    XCTAssert(myValidationFunction())
}

Key points:

  • strict: false: Turn off strict behavior directly without creating options separately.
  • Closure internal assertion: Failure is handled as expected, and unmatched expected failure is not produced when passed.

The talk places this usage in nondeterministic failure scenarios, including timing issues, ordering dependencies, and concurrency bugs.

Core Takeaways

  • What to do: Add a bug link to each known failing test. Why it’s worth doing: The reason of XCTExpectFailure can contain a URL, and the Xcode test report can jump back to the problem system from expected failure. How ​​to start: Change the temporary comment in the failing test to XCTExpectFailure("<bug URL> failure reason").

  • What: Change the known failures in the shared test tool function to scoped. Why it’s worth doing: Closure scope only catches inner failures, new failures in outer tests will continue to turn red. How ​​to start: Use XCTExpectFailure("reason") { ... } to wrap known failed assertions.

  • What to do: Create a non-strictly expected failure list for flaky tests. Why it’s worth doing:isStrict = falseAvoid unmatched expected failure when passing by accident. How ​​to start: Create related testsXCTExpectedFailure.Options(),set upoptions.isStrict = false, and then pass it toXCTExpectFailure。

  • What: Mark only specific failure types as expected failures. Why it’s worth doing:issueMatcherCan be filteredXCTIssue, old assertion failures are denoised, new crashes or other failures are still exposed. How ​​to get started: Setupoptions.issueMatcher, start withissue.type == .assertionFailureSuch rules begin.

  • What to do: Split the expected failure strategy by platform. Why it’s worth doing: In the speechoptions.isEnabledSupports turning off expected failures on some platforms, suitable for tests that fail on iOS and have been fixed on macOS. How ​​to start: Use#if os(macOS)or other platform condition settingsoptions.isEnabled = false。

Comments

GitHub Issues · utterances