gecko-dev/gfx/doc/AdvancedLayers.md

Advanced Layers

Advanced Layers is a new method of compositing layers in Gecko. This document serves as a technical overview and provides a short walk-through of its source code.

Overview

Advanced Layers attempts to group as many GPU operations as it can into a single draw call. This is a common technique in GPU-based rendering called "batching". It is not always trivial, as a batching algorithm can easily waste precious CPU resources trying to build optimal draw calls.

Advanced Layers reuses the existing Gecko layers system as much as possible. Huge layer trees do not currently scale well (see the future work section), so opportunities for batching are currently limited without expending unnecessary resources elsewhere. However, Advanced Layers has a few benefits:

• It submits smaller GPU workloads and buffer uploads than the existing compositor.
• It needs only a single pass over the layer tree.
• It uses occlusion information more intelligently.
• It is easier to add new specialized rendering paths and new layer types.
• It separates compositing logic from device logic, unlike the existing compositor.
• It is much faster at rendering 3d scenes or complex layer trees.
• It has experimental code to use the z-buffer for occlusion culling.

Because of these benefits we hope that it provides a significant improvement over the existing compositor.

Advanced Layers uses the acronym "MLG" and "MLGPU" in many places. This stands for "Mid-Level Graphics", the idea being that it is optimized for Direct3D 11-style rendering systems as opposed to Direct3D 12 or Vulkan.

LayerManagerMLGPU

Advanced layers does not change client-side rendering at all. Content still uses Direct2D (when possible), and creates identical layer trees as it would with a normal Direct3D 11 compositor. In fact, Advanced Layers re-uses all of the existing texture handling and video infrastructure as well, replacing only the composite-side layer types.

Advanced Layers does not create a LayerManagerComposite - instead, it creates a LayerManagerMLGPU. This layer manager does not have a Compositor - instead, it has an MLGDevice, which roughly abstracts the Direct3D 11 API. (The hope is that this API is easily interchangeable for something else when cross-platform or software support is needed.)

LayerManagerMLGPU also dispenses with the old "composite" layers for new layer types. For example, ColorLayerComposite becomes ColorLayerMLGPU. Since these layer types implement HostLayer, they integrate with LayerTransactionParent as normal composite layers would.

Rendering Overview

The steps for rendering are described in more detail below, but roughly the process is:

1. Sort layers front-to-back.
2. Create a dependency tree of render targets (called "views").
3. Accumulate draw calls for all layers in each view.
4. Upload draw call buffers to the GPU.
5. Execute draw commands for each view.

Advanced Layers divides the layer tree into "views" (RenderViewMLGPU), which correspond to a render target. The root layer is represented by a view corresponding to the screen. Layers that require intermediate surfaces have temporary views. Layers are analyzed front-to-back, and rendered back-to-front within a view. Views themselves are rendered front-to-back, to minimize render target switching.

Each view contains one or more rendering passes (RenderPassMLGPU). A pass represents a single draw command with one or more rendering items attached to it. For example, a SolidColorPass item contains a rectangle and an RGBA value, and many of these can be drawn with a single GPU call.

When considering a layer, views will first try to find an existing rendering batch that can support it. If so, that pass will accumulate another draw item for the layer. Otherwise, a new pass will be added.

When trying to find a matching pass for a layer, there is a tradeoff in CPU time versus the GPU time saved by not issuing another draw commands. We generally care more about CPU time, so we do not try too hard in matching items to an existing batch.

After all layers have been processed, there is a "prepare" step. This copies all accumulated draw data and uploads it into vertex and constant buffers in the GPU.

Finally, we execute rendering commands. At the end of the frame, all batches and (most) constant buffers are thrown away.

Shaders Overview

Advanced Layers currently has five layer-related shader pipelines:

• Textured (PaintedLayer, ImageLayer, CanvasLayer)
• ComponentAlpha (PaintedLayer with component-alpha)
• YCbCr (ImageLayer with YCbCr video)
• Color (ColorLayers)
• Blend (ContainerLayers with mix-blend modes)

There are also three special shader pipelines:

• MaskCombiner, which is used to combine mask layers into a single texture.
• Clear, which is used for fast region-based clears when not directly supported by the GPU.
• Diagnostic, which is used to display the diagnostic overlay texture.

The layer shaders follow a unified structure. Each pipeline has a vertex and pixel shader. The vertex shader takes a layers ID, a z-buffer depth, a unit position in either a unit square or unit triangle, and either rectangular or triangular geometry. Shaders can also have ancillary data needed like texture coordinates or colors.

Most of the time, layers have simple rectangular clips with simple rectilinear transforms, and pixel shaders do not need to perform masking or clipping. For these layers we use a fast-path pipeline, using unit-quad shaders that are able to clip geometry so the pixel shader does not have to. This type of pipeline does not support complex masks.

If a layer has a complex mask, a rotation or 3d transform, or a complex operation like blending, then we use shaders capable of handling arbitrary geometry. Their input is a unit triangle, and these shaders are generally more expensive.

All of the shader-specific data is modelled in ShaderDefinitionsMLGPU.h.

CPU Occlusion Culling

By default, Advanced Layers performs occlusion culling on the CPU. Since layers are visited front-to-back, this is simply a matter of accumulating the visible region of opaque layers, and subtracting it from the visible region of subsequent layers. There is a major difference between this occlusion culling and PostProcessLayers of the old compositor: AL performs culling after invalidation, not before. Completely valid layers will have an empty visible region.

Most layer types (with the exception of images) will intelligently split their draw calls into a batch of individual rectangles, based on their visible region.

Z-Buffering and Occlusion

Advanced Layers also supports occlusion culling on the GPU, using a z-buffer. This is disabled by default currently since it is significantly costly on integrated GPUs. When using the z-buffer, we separate opaque layers into a separate list of passes. The render process then uses the following steps:

1. The depth buffer is set to read-write.
2. Opaque batches are executed.,
3. The depth buffer is set to read-only.
4. Transparent batches are executed.

The problem we have observed is that the depth buffer increases writes to the GPU, and on integrated GPUs this is expensive - we have seen draw call times increase by 20-30%, which is the wrong direction we want to take on battery life. In particular on a full screen video, the call to ClearDepthStencilView plus the actual depth buffer write of the video can double GPU time.

For now the depth-buffer is disabled until we can find a compelling case for it on non-integrated hardware.

Clipping

Clipping is a bit tricky in Advanced Layers. We cannot use the hardware "scissor" feature, since the clip can change from instance to instance within a batch. And if using the depth buffer, we cannot write transparent pixels for the clipped area. As a result we always clip opaque draw rects in the vertex shader (and sometimes even on the CPU, as is needed for sane texture coordiantes). Only transparent items are clipped in the pixel shader. As a result, masked layers and layers with non-rectangular transforms are always considered transparent, and use a more flexible clipping pipeline.

Plane Splitting

Plane splitting is when a 3D transform causes a layer to be split - for example, one transparent layer may intersect another on a separate plane. When this happens, Gecko sorts layers using a BSP tree and produces a list of triangles instead of draw rects.

These layers cannot use the "unit quad" shaders that support the fast clipping pipeline. Instead they always use the full triangle-list shaders that support extended vertices and clipping.

This is the slowest path we can take when building a draw call, since we must interact with the polygon clipping and texturing code.

Masks

For each layer with a mask attached, Advanced Layers builds a MaskOperation. These operations must resolve to a single mask texture, as well as a rectangular area to which the mask applies. All batched pixel shaders will automatically clip pixels to the mask if a mask texture is bound. (Note that we must use separate batches if the mask texture changes.)

Some layers have multiple mask textures. In this case, the MaskOperation will store the list of masks, and right before rendering, it will invoke a shader to combine these masks into a single texture.

MaskOperations are shared across layers when possible, but are not cached across frames.

BigImage Support

ImageLayers and CanvasLayers can be tiled with many individual textures. This happens in rare cases where the underlying buffer is too big for the GPU. Early on this caused problems for Advanced Layers, since AL required one texture per layer. We implemented BigImage support by creating temporary ImageLayers for each visible tile, and throwing those layers away at the end of the frame.

Advanced Layers no longer has a 1:1 layer:texture restriction, but we retain the temporary layer solution anyway. It is not much code and it means we do not have to split TexturedLayerMLGPU methods into iterated and non-iterated versions.

Texture Locking

Advanced Layers has a different texture locking scheme than the existing compositor. If a texture needs to be locked, then it is locked by the MLGDevice automatically when bound to the current pipeline. The MLGDevice keeps a set of the locked textures to avoid double-locking. At the end of the frame, any textures in the locked set are unlocked.

We cannot easily replicate the locking scheme in the old compositor, since the duration of using the texture is not scoped to when we visit the layer.

Buffer Measurements

Advanced Layers uses constant buffers to send layer information and extended instance data to the GPU. We do this by pre-allocating large constant buffers and mapping them with MAP_DISCARD at the beginning of the frame. Batches may allocate into this up to the maximum bindable constant buffer size of the device (currently, 64KB).

There are some downsides to this approach. Constant buffers are difficult to work with - they have specific alignment requirements, and care must be taken not too run over the maximum number of constants in a buffer. Another approach would be to store constants in a 2D texture and use vertex shader texture fetches. Advanced Layers implemented this and benchmarked it to decide which approach to use. Textures seemed to skew better on GPU performance, but worse on CPU, but this varied depending on the GPU. Overall constant buffers performed best and most consistently, so we have kept them.

Additionally, we tested different ways of performing buffer uploads. Buffer creation itself is costly, especially on integrated GPUs, and especially so for immutable, immediate-upload buffers. As a result we aggressively cache buffer objects and always allocate them as MAP_DISCARD unless they are write-once and long-lived.

Buffer Types

Advanced Layers has a few different classes to help build and upload buffers to the GPU. They are:

• MLGBuffer. This is the low-level shader resource that MLGDevice exposes. It is the building block for buffer helper classes, but it can also be used to make one-off, immutable, immediate-upload buffers. MLGBuffers, being a GPU resource, are reference counted.
• SharedBufferMLGPU. These are large, pre-allocated buffers that are read-only on the GPU and write-only on the CPU. They usually exceed the maximum bindable buffer size. There are three shared buffers created by default and they are automatically unmapped as needed: one for vertices, one for vertex shader constants, and one for pixel shader constants. When callers allocate into a shared buffer they get back a mapped pointer, a GPU resource, and an offset. When the underlying device supports offsetable buffers (like ID3D11DeviceContext1 does), this results in better GPU utilization, as there are less resources and fewer upload commands.
• ConstantBufferSection and VertexBufferSection. These are "views" into a SharedBufferMLGPU. They contain the underlying MLGBuffer, and when offsetting is supported, the offset information necessary for resource binding. Sections are not reference counted.
• StagingBuffer. A dynamically sized CPU buffer where items can be appended in a free-form manner. The stride of a single "item" is computed by the first item written, and successive items must have the same stride. The buffer must be uploaded to the GPU manually. Staging buffers are appropriate for creating general constant or vertex buffer data. They can also write items in reverse, which is how we render back-to-front when layers are visited front-to-back. They can be uploaded to a SharedBufferMLGPU or an immutabler MLGBuffer very easily. Staging buffers are not reference counted.

Unsupported Features

Currently, these features of the old compositor are not yet implemented.

• OpenGL and software support (currently AL only works on D3D11).
• APZ displayport overlay.
• Diagnostic/developer overlays other than the FPS/timing overlay.
• DEAA. It was never ported to the D3D11 compositor, but we would like it.
• Component alpha when used inside an opaque intermediate surface.
• Effects prefs. Possibly not needed post-B2G removal.
• Widget overlays and underlays used by macOS and Android.
• DefaultClearColor. This is Android specific, but is easy to added when needed.
• Frame uniformity info in the profiler. Possibly not needed post-B2G removal.
• LayerScope. There are no plans to make this work.

Future Work

• Refactor for D3D12/Vulkan support (namely, split MLGDevice into something less stateful and something else more low-level).
• Remove "MLG" moniker and namespace everything.
• Other backends (D3D12/Vulkan, OpenGL, Software)
• Delete CompositorD3D11
• Add DEAA support
• Re-enable the depth buffer by default for fast GPUs
• Re-enable right-sizing of inaccurately sized containers
• Drop constant buffers for ancillary vertex data
• Fast shader paths for simple video/painted layer cases

History

Advanced Layers has gone through four major design iterations. The initial version used tiling - each render view divided the screen into 128x128 tiles, and layers were assigned to tiles based on their screen-space draw area. This approach proved not to scale well to 3d transforms, and so tiling was eliminated.

We replaced it with a simple system of accumulating draw regions to each batch, thus ensuring that items could be assigned to batches while maintaining correct z-ordering. This second iteration also coincided with plane-splitting support.

On large layer trees, accumulating the affected regions of batches proved to be quite expensive. This led to a third iteration, using depth buffers and separate opaque and transparent batch lists to achieve z-ordering and occlusion culling.

Finally, depth buffers proved to be too expensive, and we introduced a simple CPU-based occlusion culling pass. This iteration coincided with using more precise draw rects and splitting pipelines into unit-quad, cpu-clipped and triangle-list, gpu-clipped variants.