WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Author fast and reliable tests for Xcode Cloud

Author fast and reliable tests for Xcode Cloud

Developer ToolsTestingXcode Cloud
Watch original video

Highlight

Xcode Cloud supports running test suites in parallel on multiple platforms and multiple system versions, but when the test volume expands, unreliable tests will drag down the CI process. Apple PassedXCTSkip、XCTExpectFailure, test repeated execution, parallel testing and timeout control mechanisms to help developers write tests that are both reliable and fast.

Core Content

Is the test environment unreliable? Start with setup

When you run the test locally, everything works fine, but when you run the test on CI, it fails. The problem is that Xcode Cloud uses a brand new simulator, and the time zone, Locale, network permissions, and preset data may be different from the local one.

Apple’s suggestion is: put all state preparations insetUp()Here, not heretearDown()Here to clean up for the next test. Each test is independently responsible for its own initial state.

（03:37）

When the conditions are not met, skip the test

Some tests only run in specific environments. For example, the ordering interface in the production environment cannot be actually called, otherwise real orders will be generated.XCTSkipConditions can be checked at runtime and tests skipped.

（06:00）

Known failed tests, do not mute them

During maintenance of external services, related tests will fail. Disabling tests directly will lose coverage, useXCTExpectFailureFailures can be marked as “expected failures” and the test suite as a whole still passes while keeping track of the problem.

（10:02）

Parallel test acceleration CI

Xcode Cloud tests multiple platforms in parallel by default. You can also enable “Execute in parallel” in the test plan to allow multiple test targets and test classes to be executed concurrently on the server’s multi-cores.

（16:37）

Prevent runaway tests from bringing down the entire suite

If a test falls into an infinite loop or waits for a timeout, the entire CI process will be stuck. Turn on “Test Timeouts” in the Configurations of the test plan and set the maximum execution time of a single test (default 600 seconds). The timeout will automatically mark failure and continue to execute subsequent tests.

（17:52）

Detailed Content

Use async setUp to prepare test data

var truck: Truck!

override func setUp() async throws {
    let directoryURL = FileManager.default.temporaryDirectory
    let fileName = UUID().uuidString
    let fileURL = directoryURL.appendingPathComponent(fileName, isDirectory: false)
    let data = await mockDonutMenuData()
    try data.write(to: fileURL)
    truck = Truck(menuURL: fileURL)
}

Key points:

  • setUp()marked asasync throws, supports asynchronous data preparation and error throwing
  • usetemporaryDirectoryAvoid polluting the real file system -UUID().uuidStringEnsure each test uses a separate file name to avoid conflicts between parallel tests
  • insetUp()Complete all state initialization in , does not depend ontearDown()clean up

Use environment variables to control the test process

var truck: Truck!

func testOrderDonut() throws {
    let host = ProcessInfo.processInfo.environment["BASE_URL"]
    try XCTSkipIf(host == "prod.example.com")

    let expectation = XCTestExpectation(description: "Order donut")
    truck.order(with: .sprinkles, host: host) { error, donut in
        XCTAssertTrue(donut.hasSprinkles)
        expectation.fulfill()
    }
    wait(for: [expectation], timeout: 5)
}

Key points:

  • ProcessInfo.processInfo.environmentRead environment variables -XCTSkipIfSkip the test when the condition is met, marked as skipped
  • Environment variables need to be added in Xcode CloudTEST_RUNNER_prefix, such asTEST_RUNNER_BASE_URL, the runtime prefix will be automatically stripped
  • No prefix is required when configuring environment variables in Test Plan

Use async/await instead of expectation timeout

func testOrderDonut() async throws {
    let host = ProcessInfo.processInfo.environment["BASE_URL"]
    try XCTSkipIf(host == "prod.example.com")

    let donut = try await truck.orderDonut(with: .sprinkles, host: host)
    XCTAssertTrue(donut.hasSprinkles)
}

Key points:

  • Test methods are marked asasync throws, call the asynchronous API directly
  • no longer neededXCTestExpectationandwait(for:timeout:)- There is no timeout limit, the test will wait for the asynchronous operation to complete
  • The code is more concise and avoids timeout failures caused by slow servers.

Mark expected failure

func testOrderDonut() async throws {
    let host = ProcessInfo.processInfo.environment["BASE_URL"]
    try XCTSkipIf(host == "prod.example.com")

    XCTExpectFailure("https://dev.myco.com/bug/98 Donut ordering service is down")
    let donut = try await truck.orderDonut(with: .sprinkles, host: host)
    XCTAssertTrue(donut.hasSprinkles)
}

Key points:

  • XCTExpectFailureReceives a description string, usually a bug link
  • Failure of the wrapped assertion will be logged as “expected failure”
  • The entire test suite is still considered passed to avoid known issues interfering with CI results.
  • After the service is restored, the test will automatically return to normal passing status

Configure Pull Request dedicated test plan

// Select a subset in the Tests tab of the Test Plan
// For example: select only unit tests plus critical UI tests

Key points:

  • Create a dedicated “Pull Requests” test plan that only contains core tests
  • Set Start Condition to “Pull Request Changes” in Xcode Cloud Workflow
  • Add Build action and Test action, select Pull Requests test plan
  • Create another scheduled Workflow to run the complete test suite

Core Takeaways

  • Create layered test plans for different environments: core tests are quickly verified on PR, and full tests are scheduled to run in the background. Entrance: Xcode -> Product -> Test Plan -> New, Xcode Cloud -> Manage Workflows.

  • Control test behavior with environment variables: Configure in Xcode Cloud Workflow or Test PlanTEST_RUNNER_*Environment variables, passed in test codeProcessInfoRead and implement a set of test codes to adapt to multiple development/test/production environments.

  • Migrate asynchronous tests to async/await: Will be based onXCTestExpectationThe callback test is rewritten asasync throwsTest method, eliminate timeout parameters, and reduce flaky tests caused by network fluctuations.

  • Use XCTExpectFailure for known issues: do not disable or skip tests that fail due to external dependencies, useXCTExpectFailurePackages that both preserve test coverage and automatically restore it when issues are fixed.

  • Set a single test timeout to prevent CI from getting stuck: Set a reasonable execution time limit in Test Plan -> Configurations -> Test Timeouts to prevent a single out-of-control test from dragging down the entire CI process.

Comments

GitHub Issues · utterances