Harness Apple GPUs with Metal

Highlight

Apple explains how Tile Based Deferred Rendering, Tile Memory, Memoryless Render Targets, Tile Shaders, Argument Buffers, and Indirect Command Buffers allow Metal applications to reduce bandwidth, reduce memory usage, and move more complex rendering loops to the GPU side for execution.

Core Content

The performance bottleneck of many graphics programs often comes from memory transfer, and the shader itself may not be the main cause. Traditional rendering pipelines are used to writing G-Buffer, MSAA sampling results, and temporary color attachments back to external memory, and then reading them back in the next pass. The more complex the picture, the easier it is for bandwidth to become a bottleneck.

The premise of Apple GPUs is different. The CPU and GPU share system memory. The GPU also has an on-chip Tile Memory, but no independent video memory. The path chosen by Apple is Tile Based Deferred Rendering (TBDR, tile-based deferred rendering): first divide the geometry into tiles, and then complete pixel processing in each tile. As long as the intermediate results are kept in Tile Memory, many reads and writes don’t need to happen.

The value of this session is to put several of Metal’s APIs back into the context of the hardware.loadAction、storeAction, memoryless storage, programmable blending, tile shader, imageblock, argument buffer, indirect command buffer all have hardware background. Together they serve one goal: access less system memory and hand over parallelizable decision-making to the GPU.

Detailed Content

TBDR splits rendering into tiling and rendering

(01:50) A render pass on Apple GPU is divided into two stages. The Tiling phase processes the geometry of the entire pass and puts the transformed primitives into the corresponding tiles. The Rendering phase performs load actions, rasterization, visibility calculations, fragment shading, and store actions tile by tile.

Workflow example:

• When entering the render pass, first determine whether each attachment requires the previous round of content.
• No attachment of old content is required, initialize Tile Memory with clear.
• At the end of the pass, only the results that will be displayed or sampled later are saved.
• Only the depth, template or intermediate color data of this pass is served and can be discarded after completion.

Key points:

• The cost of the load action comes from moving the attachment content into Tile Memory.
• The cost of the store action comes from writing the Tile Memory contents back to system memory.
• This set of judgments does not change the picture result, but changes how much external bandwidth is required for a render pass.

(03:57) The speech repeatedly emphasized that what the application can explicitly control is the load/store action. Only load the required data and save only the data that will be used later. For TBDR, this directly determines how many transfers occur between Tile Memory and system memory.

HSR calculates visibility first, then runs the fragment shader

(04:39) Hidden Surface Removal (HSR, hidden surface removal) relies on the on-chip depth buffer to record the frontmost primitive of each pixel before the fragment shader runs. Even if the opaque triangle is submitted back-to-front, the GPU can get the visible layer first and then run the shader only on the final visible pixels.

Workflow example:

• Submit opaque geometry first.
• Submit alpha test, discard or depth feedback related geometry again.
• Finally submit translucent geometry.
• Avoid staggering submission of opaque mesh and non-opaque mesh.

Key points:

• HSR will first record the front primitive of each pixel and then run the fragment shader.
• translucent primitives force the GPU to flush overwritten pixels, and premature commits can increase ineffective shading.
• There is no strict ordering between opaque meshes, grouping by visibility state is more important.

(07:27) Overdraw here refers to the number of times each pixel triggers the fragment shader. In order to reduce overdraw, the practice given by session is to group by visibility state: opaque, alpha test/discard/depth feedback, translucent. There is no strict ordering required between opaque meshes, but do not interleave with non-opaque meshes.

Programmable Blending and memoryless target avoid intermediate textures from landing

(09:27) Algorithms such as full-screen fog effects and deferred lighting often require custom mixing. The traditional approach is to write out multiple attachments and use them as texture samples in subsequent passes. TBDR exposes programmable blending, allowing fragment shaders to directly access pixel data in Tile Memory to merge multiple render passes.

Concept example:

• Deferred lighting first obtains the surface properties, and then uses these properties to calculate lighting.
• The traditional multi-pass method will write out the attribute attachment and then sample it in the next pass.
• Programmable Blending allows the fragment stage to directly access the data of the same pixel in Tile Memory.
• In this way, the originally separated passes can be merged and the load/store of the intermediate attachment can be reduced.

Key points:

• Session explicitly classifies global fog and deferred lighting as scenes that require custom blending.
• The scope of Programmable Blending is the current pixel, which is suitable for single-pixel level blending and light accumulation.
• It benefits from eliminating the need for external write-out and read-back of intermediate render targets.

(10:28) If some render targets are only used within Tile Memory, they can be set to memoryless storage. This way Metal does not allocate persistent system memory for these temporary attachments.

Workflow example:

• List G-Buffer attachments in deferred renderer.
• Mark attachments that are only used within the current render pass.
• These attachments do not need to be retained in system memory.
• The final color or the results that need to be read in subsequent passes need to be persisted.

Key points:

• Memoryless Render Targets target temporary data that only exists in Tile Memory.
• What it reduces is the memory footprint because external storage is no longer allocated for these temporary render targets.
• If a subsequent pass needs to sample a certain result, it does not belong to the memoryless scenario described by this session.

MSAA resolution is completed from Tile Memory

(11:31) Apple GPU’s MSAA (Multisample Antialiasing) will save multiple samples on the chip and resolve during tile flush. The conclusion given by session is very straightforward: multi-sampling textures usually do not need to be stored, and only the resolve results need to be written to the final texture.

Workflow example:

• Use MSAA in the render pass to generate multiple samples.
• Multiple samples are saved in Tile Memory.
• Directly resolve to the final result during tile flush.
• Only the resolved texture is saved, not the complete multisample attachment.

Key points:

• session explicitly says MSAA Resolve always happens from Tile Memory.
• Multisampling attachment is transient data in this process.
• Combine it with memoryless storage to save bandwidth and footprint at the same time.

Imageblocks and Tile Shaders allow compute to enter the middle of the render pass

(15:53) Starting with A11, Apple GPUs have added on-chip Imageblock and Tile Compute. Imageblock is a two-dimensional data structure in Tile Memory and can be accessed by fragment function or kernel function. Tile Shader is a compute kernel that can be dispatched in the render pass, and is interleaved with draw calls in the order of API submission.

Workflow example:

• First obtain the image data corresponding to the tile in the render pass.
• Use Imageblock to represent two-dimensional data in Tile Memory.
• Insert Tile Shader dispatch between draw calls.
• Tile Shader is interleaved with draw calls in the order of submission, and synchronization boundaries are established with the preceding and following draw calls.

Key points:

• The value of Imageblock is that the GPU knows it is processing image data.
• Tile Shader can access Imageblock in mid-render pass.
• This allows some steps that would otherwise require additional compute passes to be completed within the same render pass.

(19:28) Tile Shading can be used to accumulate light culling, G-Buffer usage and lighting into a single pass. The original external texture reading and writing between the three passes has become interleaved rendering and calculation within Tile Memory.

Argument Buffers and Indirect Command Buffers put draw decisions on the GPU

(20:47) The traditional render loop consists of the CPU traversing the scene, reading the occlusion information written by the GPU of the previous frame, doing frustum cropping and LOD selection, and then encoding the draw command. There are synchronization points here: the CPU has to wait for the GPU results, and the GPU has to wait for the CPU to submit the next batch of draws.

Workflow example:

• Organize the mesh, material, and model relationships in the scene into a data structure that the GPU can traverse.
• The GPU first traverses the scene and renders the occluder.
• The GPU then traverses the scene and performs culling and LOD selection.
• The GPU finally submits the draw commands for visible objects to complete the final rendering.

Key points:

• Argument Buffers are responsible for keeping complex scene data in a structure accessible to the GPU.
• Indirect Command Buffers are responsible for letting the GPU generate its own draw commands.
• This path removes the synchronization point where the CPU reads the occlusion data of the previous frame and then encodes the draw command.

(22:45) The GPU-driven render loop of the session is three steps: the GPU traverses the scene to render the occluder, the GPU then traverses the scene to do culling and LOD selection, and finally the GPU renders the final scene. The CPU no longer decides each draw call on a frame-by-frame basis.

Core Takeaways

1. What to do: Add memoryless G-Buffer to existing deferred renderer. Why it’s worth doing: The session clearly points out that the cost of deferred rendering comes from the writing and sampling of intermediate textures, and the memoryless render target can reduce the footprint. How ​​to start: Check the life cycle of each G-Buffer attachment and only set attachments used within the same render passMTLStorageMode.memoryless, the final color attachment is retained.store。

2. What to do: Group the rendering queue by visibility state. Why it’s worth doing: HSR is very efficient for opaque geometry, transparency and discard will trigger flush, and incorrect ordering will increase overdraw. How ​​to start: Before submitting the draw, split the mesh into three groups: opaque, alpha test/discard/depth feedback, and translucent. First ensure that opaque and non-opaque are not interlaced.

3. What to do: Change the MSAA intermediate sampling texture to a transient resource. Why it’s worth doing: Apple GPU’s MSAA resolve is done from Tile Memory, there is no need to write the full multisample texture back to system memory. How ​​to start: Let the multisample texture use memoryless storage and set it in the render pass.resolveTexture, store action uses.multisampleResolve。

4. What to do: Experiment with tile-based light culling for multi-light scenes. Why it’s worth doing: Tile Shading can process the light list by tile in the render pass, combining multiple passes into a process that saves bandwidth. How ​​to start: First organize the light index list required by each tile, and then use the tile shader to read the G-Buffer data in the imageblock for lighting accumulation.

5. What to do: Migrate culling and LOD selection of large scenes to the GPU. Why it’s worth doing: Argument Buffers allow the GPU to traverse complex scene data, and Indirect Command Buffers allow the GPU to encode draw commands, which can reduce CPU/GPU synchronization points. How ​​to start: Convert mesh, material, and model flatten into argument buffer array, use compute pass to write to indirect command buffer, and then execute these commands in render pass.

Related Sessions

• Optimize Metal apps and games with GPU counters — Learn to use Xcode 12’s GPU counters to verify whether TBDR optimization really reduces bottlenecks.
• Bring your Metal app to Apple silicon Macs — Continue to talk about Apple GPU’s TBDR architecture and Metal adaptation from the perspective of Mac porting.
• Optimize Metal Performance for Apple silicon Macs — Further performance tuning for the Metal workload of Apple silicon Macs.
• Build customized ML models with the Metal Performance Shaders Graph — Learn how Metal compute power can serve machine learning graph and tensor workloads.