WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Discover Metal 4

Discover Metal 4

Metal 4GPUMetalFXMachine Learning
Watch original video

Highlight

Metal 4 swaps the command queue, command buffer, and encoders for a new MTL4 family of objects. Command buffers are decoupled from queues, so you can encode in parallel across threads. New mechanisms — MTL4ArgumentTable, residency sets, placement sparse resources, the Barrier API, and MTL4Compiler — round out the design, and tensors and machine learning encoders join as first-class resources.

Core content

The biggest pain point in old Metal apps is that the binding model cannot keep up with modern rendering’s resource scale. A single scene packs thousands of materials, geometry buffers, and textures into a fixed number of bind points. Each draw call resets bindings on the CPU, and the encoding thread quickly becomes the bottleneck. Apple eased this once with bindless: stuff every resource into an argument buffer and bind one object at a time. But binding management, residency, shader compilation, and ML operations stayed scattered, and the code was tedious to write.

Metal 4’s answer is to tear the chain apart and rebuild it. On the encoding side, MTL4CommandQueue and MTL4CommandBuffer arrive, and the command buffer is decoupled from the queue — you ask the device for a buffer and submit it to any queue, which gives you parallel encoding across threads (02:22). Encoders are merged too: the new unified compute encoder handles compute, blit, and acceleration structure builds in one place (02:53); MTL4RenderCommandEncoder carries an attachment map, so you can switch color attachments inside the same encoder and skip repeated allocation (03:06). MTL4CommandAllocator manages the command buffer’s memory directly — you are in charge (03:44).

For resource management, the binding table becomes MTL4ArgumentTable, sized on demand; a bindless scene needs only one buffer binding (05:43). Residency goes through residency sets — attach a set of resources to an MTL4CommandQueue once, and every command submitted to that queue sees them automatically (06:55). After Remedy integrated residency sets in Control Ultimate Edition, residency overhead dropped sharply, and memory use went down too when ray tracing was off (07:22). When you need more than physical memory holds, use placement sparse resources — declare the resource without paging, then allocate pages from a placement heap on demand (08:32). Concurrent synchronization moves to the Barrier API, stage to stage, matching the barrier concept in DirectX and Vulkan (09:02).

The most interesting change is that ML becomes a first-class citizen. Metal 4 adds the tensor resource type with arbitrary dimensions (14:22), and ships a machine learning command encoder that drops a whole CoreML network into a command buffer as one command, sharing barrier sync with rendering commands (15:05). Small networks can run inline in a shader through Metal performance primitives — for neural material evaluation, sampling the latent texture, running inference, and shading all fuse into one dispatch (16:01).

Detailed content

Command encoding: new objects and parallel encoding

Metal 4’s encoding model rests on a set of new objects: MTL4CommandQueue, MTL4CommandBuffer, MTL4CommandAllocator, MTL4RenderCommandEncoder, and the new unified compute encoder. The existing MTLDevice stays — no need to swap the device object (02:08).

// Pseudocode: all API names are explicitly mentioned in the transcript.
let queue = device.makeMTL4CommandQueue()
let allocator = device.makeMTL4CommandAllocator()
let cmdBuffer = device.makeMTL4CommandBuffer(allocator: allocator)

let renderEncoder = cmdBuffer.makeMTL4RenderCommandEncoder(descriptor: rpDesc)
// The attachment map allows switching color attachments within the same encoder.
renderEncoder.setAttachmentMap(mapA)
renderEncoder.drawPrimitives(...)
renderEncoder.setAttachmentMap(mapB)
renderEncoder.drawPrimitives(...)
renderEncoder.endEncoding()

queue.commit([cmdBuffer])

Key points:

  • MTL4CommandQueue: new queue object, decoupled from the command buffer, usable from many threads at once.
  • MTL4CommandAllocator: manages command buffer memory directly, so the framework no longer allocates implicitly.
  • makeMTL4CommandBuffer(allocator:): command buffers come from the device, not the queue, so you can encode in parallel and decide which queue to submit to later (02:29).
  • MTL4RenderCommandEncoder plus attachment map: switch color attachments inside one render encoder, skipping a second encoder for a different output (03:16).

Binding: MTL4ArgumentTable

The argument table is declared per stage and can be shared across stages. In a bindless setup, one buffer binding is enough (06:00).

Residency: residency sets

Configure a residency set once and reuse it. When residency rarely changes, fill the set at app launch; for streaming, update it on a separate thread in parallel with encoding (07:09).

Sparse resources: placement sparse

A placement sparse resource is declared without pages; pages come from a placement heap at runtime, on demand (08:43). Note: the mapping operation needs an MTL4CommandQueue, while the old render queue is still an MTLCommandQueue — sync the two queues with an MTLEvent (21:26).

Synchronization: the Barrier API

Barriers go from stage to stage. Example: dispatch a grayscale conversion shader in a compute encoder, and the next render encoder’s fragment stage needs to read it — that calls for a dispatch-to-fragment barrier (10:01). See the official sample “processing a texture in a compute function” for the full code.

Compilation: MTL4Compiler and flexible render pipeline states

MTL4Compiler is split off from the device, giving you direct control over when CPU-side compilation runs. It inherits the QoS of the calling thread, and during concurrent multi-threaded compilation, requests from higher-priority threads jump the queue (11:45). Flexible render pipeline states let multiple pipelines that differ only in color state share one compiled Metal IR — build an unspecialized pipeline, then specialize on demand (12:41).

Machine learning: tensors, ML encoder, shader ML

Big networks go through the ML command encoder: convert a CoreML package to a Metal package with the Metal toolchain, hand it to the encoder, and share barriers with rendering commands (15:32). Small networks go through shader ML: use Metal performance primitives to run matrix and convolution math inside the shader, and the OS compiler inlines and tunes the code for the current device (17:00).

MetalFX: new frame interpolation and denoising

On top of the existing temporal upscaling, MetalFX this year adds frame interpolation (synthesize an in-between frame to push refresh rate) and a denoising upscaler (upscale and denoise a low-sample-count ray-traced result at the same time) (18:24, 18:53).

Debug tools and getting started

Xcode 26 ships a Metal 4 project template — pick Metal 4 when you create a new Game project (22:50). API/Shader Validation, Metal Debugger, Metal Performance HUD, and Metal System Trace all support Metal 4.

Core takeaways

  • What to do: move your existing Metal app’s shader compilation path to MTL4Compiler.

    • Why it pays off: lowest risk, direct gain. MTL4Compiler is independent of the device, inherits the calling thread’s QoS, and prioritizes high-priority requests during concurrent compilation, which cuts startup hitches noticeably.
    • How to start: ask the device for an MTL4Compiler and route every makePipelineState call through the new entry point. Leave shaders and pipeline descriptors alone; only swap the compilation entry.

  • What to do: replace scattered useResource / makeResident calls with residency sets.

    • Why it pays off: Remedy’s measurements on Control Ultimate Edition show a clear drop in memory use when ray tracing is off and a parallel cut in residency CPU overhead. Integration cost is low.
    • How to start: group resources by lifetime (permanent, level, streaming) and build one residency set per group. Fill the permanent group at app launch and attach it to MTL4CommandQueue; update the streaming group on the loading thread.

  • What to do: convert textures that exceed physical memory into placement sparse.

    • Why it pays off: one asset covers more device tiers. Low-end machines keep only low-LOD pages resident; high-end machines keep everything. No need to ship multiple asset sets.
    • How to start: begin with the largest textures (terrain virtual texture, UI atlas). Declare them as placement sparse, build a placement heap, and let a visibility script allocate and release pages. Map on the MTL4CommandQueue and sync with the old MTLCommandQueue using an MTLEvent.

  • What to do: combine MetalFX temporal upscaling with frame interpolation to hit high refresh.

    • Why it pays off: rendering at 1/4 pixels and inserting one frame, in theory, cuts render load by 7/8. For a game targeting 120Hz, this is the highest-leverage step.
    • How to start: wire up MetalFX temporal upscaling first (start at 2x), confirm there are no obvious artifacts, then add frame interpolation. Watch real frame rate and latency live with Metal Performance HUD.

  • What to do: use Shader ML and Metal performance primitives to render neural materials.

    • Why it pays off: latent textures are far smaller than a traditional PBR texture set, cutting texture bandwidth. Inference, sampling, and shading fuse into one dispatch, with no round-trips through device memory.
    • How to start: train a small material decoder in PyTorch or CoreMLtools and export a CoreML package. Call it from the shader through Metal performance primitives, feed the sampled latent in as a tensor, and output albedo, normal, and roughness.

Comments

GitHub Issues · utterances