WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Debug and profile agentic app experiences with Instruments

Debug and profile agentic app experiences with Instruments

InstrumentsFoundation ModelsAgentic AppsLLM DebuggingDynamic Instructions
Watch original video

Highlight

Xcode 27’s Foundation Models Instrument visualizes prompts, tool calls, instruction switches, and token usage for LLM agent apps directly on a timeline, helping developers locate silent failures, instruction mismatches, and performance bottlenecks.

Core Content

Three hurdles in LLM development

(01:58)

In traditional code, input A deterministically produces output B. But when building agentic apps with the Foundation Models framework, developers face three challenges that do not exist in traditional development.

First, probabilistic output. Send the same prompt to an LLM twice, and it may return completely different answers. That means standard unit tests break down: you cannot simply assert that the output equals a fixed string. You can only evaluate answer quality and intent.

Second, communication between models. Complex features often require multiple models to work together. For example, in a recipe app, one model identifies ingredients from a photo, while another generates a recipe from the recognition result. If anything goes wrong while data moves between models, the whole chain breaks.

Third, observability. When one step in a multi-model pipeline fails, it is hard to tell exactly where the failure happened. You need to see each step’s input, the model’s decision process, and the output.

(03:08)

The basic flow of an LLM app is simple: the user sends a prompt, the model runs inference, and it returns an answer. But many scenarios need a tool-call loop: the user sends a prompt, the model runs inference and calls a tool, the tool performs an action, and the model uses the result to generate the final answer. This loop can repeat multiple times.

Each step adds latency, and each step is another failure point. Understanding this loop is the prerequisite for reading every piece of data in the Foundation Models Instrument.

A real debugging case

(04:02)

The presenter is building a craft journal app with an interactive brainstorming feature: the user and model chat to refine a creative idea, and once the idea is confirmed, the app automatically generates a detailed tutorial. The feature uses two sets of instructions: one for brainstorming and one for tutorial generation. The brainstorming instructions include two tools: GenerateCraftIdeaTool and SwitchToTutorialModeTool.

The presenter runs the app. The model proposes several craft ideas, and “Paper Butterfly” is selected. But instead of switching into tutorial mode, the model keeps generating more ideas. The feature is stuck.

(05:02)

Open Xcode, choose Product > Profile, select the Foundation Models template in the template chooser, and click Record. The instrument captures prompt and response data from the device. This data may contain sensitive information. It is disabled by default in production and only enabled during tracing, so trace files must be handled carefully.

Six lanes on the timeline

(06:36)

The Foundation Models Instrument timeline has six lanes:

  • Instructions lane: shows how long a set of instructions and tools is active. One set of instructions can span multiple requests.
  • Model Inference lane: yellow bars show time spent processing the input prompt, while orange bars show time spent generating the response.
  • The remaining lanes provide an overview of session structure and latency.

Below the timeline is the tree detail view, which organizes all logs from one recording hierarchically: session -> request -> model inference -> instructions -> prompt -> response.

(06:59)

Looking at the Instructions lane, the whole session activated only one set of instructions. But the feature design needs two: brainstorming and tutorial generation. That means the handoff step is broken.

Expand the tree view. Session 1 has two requests. The first request was triggered by the prompt “Please generate 3 craft ideas” and contains two model inferences plus several tool calls. Click a model inference node, and the inspector on the right shows summaries of the instructions, prompt, and response.

(08:54)

Select the Instructions node. The inspector shows that this set of instructions is bound to only one tool. The prompt mentions switchToTutorialMode, but the toolset does not configure this tool. Without it, the app cannot switch from brainstorming mode to tutorial mode, and the user is trapped in the loop.

The harder part is that this is a silent failure. The model keeps accepting input and calling tools, and no error is thrown. Without Instruments, this kind of bug is almost impossible to find.

Fix and verify

(09:38)

Return to Xcode and inspect the definition of BrainstormDynamicInstructions. The prompt mentions SwitchToTutorialMode, but the toolset only registers GenerateCraftIdeasTool. Add SwitchToTutorialModeTool to the toolset, rebuild, and record again with Instruments.

Run the app again, choose “necklace,” and the UI successfully switches to tutorial mode. The model generates a complete tutorial.

(10:37)

Back in Instruments, the Instructions lane now shows two independent sets of instructions: brainstorming instructions first, then tutorial-generation instructions, exactly matching the feature design.

In the tree view, the first set of instructions now contains both generateCraftIdea and switchToTutorialMode. The second model inference in the second request triggers a switchToTutorialMode tool call, passing the selected craft as an argument. The next request correctly switches to the tutorial-generation instructions and passes the selected craft as context.

Three performance metrics

(12:07)

The model inference node in Instruments shows three key performance metrics:

  • Time to First Token (TTFT): the time from sending the prompt until the model starts generating the first token. High TTFT means the user is staring at a blank screen. Reduce it by shortening the prompt.
  • Tokens per Second (TPS): the overall response generation speed. Use it to benchmark different prompt configurations and catch performance regressions after changes.
  • Total Latency: the total time from sending the request to receiving the complete response. This is the number users feel most directly. Reduce perceived latency by using streaming and showing partial results early.

The info column in the tree view automatically marks anomalous nodes: errors, long durations, and large token counts. Request 1’s first model inference took longer than expected. Open it and inspect token usage and duration breakdowns to find the starting point for optimization.

Details

Debugging Foundation Models apps with Instruments

(05:02)

Open the project in Xcode, choose Product > Profile (or press Cmd+I), select the Foundation Models template in the template chooser, and click Record. The app launches on the connected device, and Instruments starts capturing data.

At the top of the timeline are tracks, and each track contains multiple lanes that visualize activity levels or regions. Below the timeline is the detail view, which shows summary information for the current selection. Click a bar in the timeline or a row in the detail view, and the inspector on the right shows details for the selected item.

The tree view is the core tool for troubleshooting. It organizes all logs hierarchically:

Session
  └── Request
        └── Model Inference
              ├── Instructions
              ├── Prompt
              └── Response (or Error)

Each level can be expanded and collapsed. Click any node, and the inspector shows the complete content for that node.

Finding silent failures caused by instruction mismatches

(08:54)

The bug in the demo is a classic instruction-tool mismatch. The prompt text in BrainstormDynamicInstructions tells the model it can call switchToTutorialMode, but the toolset is missing that tool:

// Error: the prompt mentions switchToTutorialMode, but the toolset does not register it
DynamicInstructions {
    "...call switchToTutorialMode when ready..."
} toolset: {
    GenerateCraftIdeaTool()  // Only one tool is registered
}

The model sees the instruction in the prompt, realizes it has no corresponding tool to call, and keeps generating text instead of switching modes. No error, no exception, just behavior that does not match expectations.

The fix is to complete the toolset:

// Correct: prompt and toolset are strictly aligned
DynamicInstructions {
    "...call switchToTutorialMode when ready..."
} toolset: {
    GenerateCraftIdeaTool()
    SwitchToTutorialModeTool()  // Add the missing tool
}

Key points:

  • The toolset parameter of DynamicInstructions determines which tools the model can actually call
  • Tool names mentioned in the prompt must have corresponding registrations in toolset
  • The LLM does not throw a Swift error when a tool is not registered; it silently continues generating text
  • The Instructions lane in Instruments shows at a glance how many instruction sets were active in a session
  • The Instructions node in the tree view shows the actual bound tool list, which you can compare with the prompt to locate mismatches

Optimizing performance: how to use the three metrics

(12:07)

The inspector for a model inference node includes two visual sections: Duration and Token Usage.

Time to First Token (TTFT) measures the time from sending the prompt until the model outputs the first token. The amount of computation in this phase depends on prompt length: the longer the prompt, the more expensive the prefill phase. The optimization direction is to simplify the prompt and remove unnecessary context.

Tokens per Second (TPS) measures response generation speed. This metric is sensitive to changes in model configuration and prompt versions, making it a useful benchmark anchor. After changing a prompt or switching models, compare TPS to catch performance regressions quickly.

Total Latency is the end-to-end waiting time. In agentic scenarios with multi-step tool calls, this number can grow quickly. Apple’s recommendation is to use streaming to reduce perceived latency:

let session = LanguageModelSession(instructions: myInstructions)

// Use streaming responses to show partial results early
let responseStream = session.streamResponse(to: userPrompt)

var isFirstToken = true
for try await partialResponse in responseStream {
    if isFirstToken {
        // Record TTFT; at this point, the user has already seen the first character
        recordTTFTMetric()
        isFirstToken = false
    }
    // Update the UI in real time and use streaming output to hide total latency
    updateUI(with: partialResponse)
}

Key points:

  • streamResponse(to:) returns an asynchronous sequence that can be consumed token by token
  • Update the UI when the first partial response arrives to greatly reduce perceived TTFT
  • Without streaming, complex agentic flows may leave users facing a blank screen for several seconds
  • Token usage data in Instruments is the starting point for performance optimization; it tells you where time and resources are being spent

Privacy notes

(05:16)

The Foundation Models Instrument captures full prompt and response content, which may contain sensitive user information. Logging is disabled by default in production and only enabled during trace recording. Treat trace files as sensitive data. Do not casually export or share them from environments that contain real user privacy data.

Key Takeaways

1. Add an AI assistant to an existing app and use Instruments to debug tool-call loops

  • What to build: Integrate a Foundation Models-based intelligent assistant into an existing app so it can call app features such as search, filtering, and content creation.
  • Why it is worth building: Most apps already have rich features, but users have to click through layers of UI to find them. An agentic assistant can translate “help me find the sunset photos I took last week” into a sequence of tool calls.
  • How to start: Define instructions and a toolset with LanguageModelSession, observe tool-call order and latency in Instruments, and make sure every tool is called correctly without falling into a loop.

2. Build multi-step workflows and manage context with instruction switching

  • What to build: Design a feature that requires multi-stage reasoning, such as “analyze document -> extract key points -> generate summary -> translate into Chinese.” Use different instructions for each stage.
  • Why it is worth building: One giant prompt can sharply increase TTFT and make the model lose track in complex tasks. Staged processing reduces complexity at each step and lets you observe timing for each stage in Instruments.
  • How to start: Define separate DynamicInstructions for each stage, then trigger instruction switching through tool calls when a stage completes. Verify switch timing in the Instructions lane in Instruments.

3. Use streaming plus placeholder UI to improve perceived latency for LLM features

  • What to build: Change all LLM interactions from “wait for the complete response before display” to “show the first token as soon as it arrives, then keep updating.”
  • Why it is worth building: Total latency in multi-step agentic flows can reach several seconds or even more than ten seconds. Streaming turns TTFT from “user waiting time” into “time when the user has already seen content.”
  • How to start: Replace response(to:) with streamResponse(to:), consume the asynchronous sequence with for try await, and add a typewriter effect in the UI so users can see content being generated.

4. Establish performance benchmarks for LLM features

  • What to build: Establish baseline TTFT, TPS, and Total Latency values for each LLM-driven feature, and monitor metric changes in CI.
  • Why it is worth building: Small prompt changes can significantly affect performance. Without baseline data, performance regressions are often discovered only after user complaints.
  • How to start: Record traces for typical scenarios in Instruments and write down the three metrics for key model inference nodes. After each prompt or model-configuration change, record again and compare metrics.

5. Design fallback strategies for LLM features

  • What to build: When model inference takes too long or token usage is too high, automatically switch to a lighter model or shorter prompt.
  • Why it is worth building: The info column in Instruments marks nodes with long durations and large token counts. This data can define fallback trigger conditions.
  • How to start: Monitor TTFT from streamResponse in code. If it exceeds a threshold, cancel the current request and retry with a simplified prompt. Instruments helps identify which scenarios have prompts most likely to time out.

Comments

GitHub Issues · utterances