WWDC Quick Look 💓 By SwiftGGTeam
中文 EN 日本語
Debug GPU-side errors in Metal

Debug GPU-side errors in Metal

MetalDeveloper Tools
Watch original video

Highlight

Xcode 12 adds Enhanced Command Buffer Errors and Shader Validation to Metal. Developers can locate GPU-side errors to encoder, draw call, Metal functions, and shader line numbers in some scenarios.

Core Content

The trouble with Metal is that a lot of bugs happen on the GPU. When the CPU-side API is used incorrectly, Metal API Validation will provide the function entry, error type, call stack and source code location. Shader errors on the GPU side often manifest themselves simply as command buffer failures, corrupted graphics, missing shadows, or occasional data contamination.

Such errors include global memory or threadgroup memory out of bounds, access to nil texture, invalid argument buffer resource resident, long running or infinite loop. In the past, I often saw a general command buffer error. There may be hundreds of encoders in a command buffer, and developers can only narrow down the range manually.

Xcode 12 adds two layers of information to the Metal debugging process. The first layer is Enhanced Command Buffer Errors, which breaks down command buffer errors to the encoder level and tells you which encoders are completed, pending, affected, faulted or unknown. The second layer is Shader Validation, which inserts the Metal shader on the GPU, detects undefined behavior, and locates the error to draw call, Metal function, and GPU backtrace. When the conditions are met, the file and line number will also be given.

These two layers of tools correspond to a clear troubleshooting path: first detect the error, then locate the scope, then classify, and finally repair the code. Enhanced Command Buffer Errors is suitable for normal use in development and QA. In low-overhead scenarios, it can also be brought online to collect telemetry. Shader Validation is more expensive and suitable for Xcode debugging, automated testing, and pre-release verification.

Detailed Content

Let command buffer report encoder level status

(03:40) Enhanced Command Buffer Errors are enabled through the new descriptor API. The key switch iserrorOptions = .encoderExecutionStatus。

let desc = MTLCommandBufferDescriptor()
desc.errorOptions = .encoderExecutionStatus
let commandBuffer = commandQueue.makeCommandBuffer(descriptor: desc)

Key points:

  • MTLCommandBufferDescriptor()Create a command buffer description object. -desc.errorOptions = .encoderExecutionStatusRequires Metal to log encoder execution status when an error occurs. -makeCommandBuffer(descriptor:)Use this description object to create a command buffer, and subsequent errors will bring encoder information.

(03:55) After the error occurs, the encoder information is placed inNSError.userInfoofMTLCommandBufferEncoderInfoErrorKeyinside. eachMTLCommandBufferEncoderInfoAll come with label, debug signposts and error state.

if let error = commandBuffer.error as NSError? {

    if let encoderInfos =
        error.userInfo[MTLCommandBufferEncoderInfoErrorKey]
        as? [MTLCommandBufferEncoderInfo] {

        for info in encoderInfos {
            print(info.label + info.debugSignposts.joined())
            if info.errorState == .faulted {
                print(info.label + " faulted!")
            }
        }
    }
}

Key points:

  • commandBuffer.error as NSError?Get the error object after the command buffer is completed. -MTLCommandBufferEncoderInfoErrorKeyCorresponding to a set of encoder info, arranged in recorded order. -info.labelandinfo.debugSignpostsRely on the debugging flag that developers usually add to the encoder. -info.errorState == .faultedIndicates that this encoder directly causes command buffer fault. -affectedRepresentatives may have been affected by the glitch, but Metal cannot confirm it is the source.

This mechanism first solves the problem of “where to start looking”. The video clearly recommends that development and QA phases enable it for every command buffer. Because it is integrated directly into Metal, the overhead is low, and it can also be brought to the user device after performance evaluation, using telemetry to gradually narrow down the scope of the problem.

Use Shader Validation to find undefined behavior in shaders

(07:08) Shader Validation is a verification layer that runs on the GPU. It instruments the Metal shader and detects operations that cause undefined behavior. When a problem is detected, the tool will prevent the operation and generate logs to help locate draw calls, Metal functions, and specific shader lines in some scenarios.

The range it can detect includes:

  • Device memory and constant memory are accessed out of bounds.
  • Threadgroup memory out-of-bounds access.
  • Call texturing functions on null texture object.

The example in the video is typical. An out-of-bounds read sometimes falls into unallocated virtual memory, triggering a command buffer fault; sometimes it falls into an adjacent buffer, reading incorrect data or corrupting another buffer. In the latter case, there may be no command buffer error, but only intermittent rendering errors. This is the value of Shader Validation: even without fault, some undefined behaviors can be discovered.

(09:47) When using it in Xcode, open Diagnostics in scheme settings and check Shader Validation in the Metal area. Xcode will turn on Enhanced Command Buffer Errors for all command buffers at the same time. Then add Metal diagnostics breakpoint. When the program hits shader validation error, it will stop and display GPU and CPU backtrace.

In the demo, the Ray Tracing example has missing shadows and screen lines. When Shader Validation is enabled, Xcode flags out-of-bounds memory reads in the Metal shader. The developer followed the GPU backtrace back to the calling point and found that when converting two-dimensional coordinates to one-dimensional array index,gridXMultiply by the width and correct togridY * width + gridXPost-rendering resumes.

Enable validation outside Xcode and read logs

(15:05) Automated tests or non-Xcode runtime environments can also enable the verification layer.MTL_DEBUG_LAYERSet to a non-zero value to turn on API Validation.MTL_SHADER_VALIDATIONSet to a non-zero value to turn on Shader Validation. The two variables can be set at the same time or used separately, but they must be created earlier than the Metal device; after the device is created, these values ​​will be locked, and subsequent modifications in this process will not take effect.

(15:39) command buffer now haslogsproperty. It is only valid after the command buffer is completed, so the reading logic must be put into the completion handler.

commandBuffer.addCompletedHandler { (commandBuffer) in
    for log in commandBuffer.logs {
        let encoderLabel = log.encoderLabel ?? "Unknown Label"
        print("Faulting encoder \"\(encoderLabel)\"")
        guard let debugLocation = log.debugLocation,
              let functionName = debugLocation.functionName
        else {
            return
        }
        print("Faulting function \(functionName):\(debugLocation.line):\(debugLocation.column)")
    }
}

Key points:

  • addCompletedHandlerMake sure the code runs after the command buffer completes. -commandBuffer.logsMay contain multiple Shader Validation errors. -log.encoderLabelAssociate errors back to the encoder. -log.debugLocationProvides the source code location when the library is compiled from source or with debug symbols. -functionName、line、columnHave automated test logs point directly to shader functions and locations.

(15:40) The same type of information will also enter the system log. You can uselog streamFilter Metal GPU debug logs.

log stream --predicate "subsystem = 'com.apple.Metal' and category = 'GPUDebug'"

Key points:

  • subsystem = 'com.apple.Metal'Just look at the Metal subsystem. -category = 'GPUDebug'Filter GPU debug related logs.
  • The log will contain information such as process name, error type, error details, and Metal file name and line number.

Costs to know before turning on Shader Validation

(17:17) Shader Validation will increase pipeline compilation time. The video recommends using asynchronous compilation methods to spread compilation across multiple threads to reduce waiting during development.

(17:34) Metal library must have debug symbols. Xcode debug scheme will handle it automatically; when calling Metal frontend manually, you need to add-g. When compiling a library from source at runtime, it is recommended to use#linedirective, giving backtracer a recognizable file name.

(18:26) Shader Validation is a process-level switch. When turned on, all Metal commands in the process, including UI rendering, will go through the verification layer. It will bring high performance and memory overhead. Video is recommended for development and QA, and is not recommended for users.

(18:57) After turning on, some query results may change. Computing pipelinemaxTotalThreadsPerThreadgroupandthreadExecutionWidthAll need to be read again. Some instrumentation can be turned off through environment variables. For example, when you have checked for null texture yourself, you can setMTL_SHADER_VALIDATION_TEXTURE_USAGE = 0Turn off texture usage instrumentation, but will give up the corresponding detection capabilities.

Core Takeaways

  • Make a Metal error telemetry channel: used in QA buildsMTLCommandBufferDescriptorEnable encoder execution status and upload the label, signposts and device information of the faulted encoder together. This turns “occasional GPU crashes” into an aggregated problem list.

  • Integrate Shader Validation into rendering regression testing: Set in the graphics test task of CIMTL_DEBUG_LAYER=1andMTL_SHADER_VALIDATION=1, read after running the critical rendering pathcommandBuffer.logs. Test failure information directly outputs faulting encoder, function, line and column.

  • Establish a naming convention for each render pass: Set a stable label for the command encoder and insert debug signposts in complex passes. Enhanced Command Buffer Errors and Xcode Metal Debugger will reuse these identifiers, and fault reports can be directed to business modules.

  • Preserve debug symbols mode for shader compilation pipeline: Enable Metal debug symbols uniformly for debugging and QA builds. Runtime compiled shader source code plus#linedirective, allowing Shader Validation’s backtrace to point to the real file or generator output location.

  • Make a pre-release GPU undefined behavior checklist: Cover out-of-bounds device memory, threadgroup memory, null texture, argument buffer resource persistence and timeout. Enhanced Command Buffer Errors is responsible for encoder-level positioning, and Shader Validation is responsible for undefined behavior that can be discovered by instrumentation.

Comments

GitHub Issues · utterances