Realtime Graphics World

// Render Pipeline

The GPU render pipeline transforms 3D geometry into 2D pixels through a sequence of programmable and fixed-function stages. Every frame: vertices → triangles → fragments → pixels. The programmer controls the vertex shader and fragment shader; the hardware handles rasterization, depth test, and blending.

realtime render pipeline — animated data flow

// pipeline_speed 0.60

Vertex Shader

Transforms each vertex position from object space through world → view → clip space. Runs once per vertex on the GPU. Outputs gl_Position and varyings.

Rasterization

Fixed-function hardware converts triangles into fragments (potential pixels). Interpolates vertex attributes (UVs, normals) across the triangle surface.

Fragment Shader

Computes final pixel color. Texture sampling, lighting calculations, procedural effects. Runs once per fragment — the bottleneck for complex scenes.

Output Merge

Depth test rejects occluded fragments. Alpha blending composites transparency. Stencil operations mask regions. Final pixel written to framebuffer.

// GPU Architecture

Modern GPUs are massively parallel processors with thousands of shader cores executing the same program on different data (SIMD). NVIDIA Ampere/Ada, AMD RDNA3, and Apple M-series GPUs differ in microarchitecture but share the same shader execution model.

GPU compute unit visualization

// compute_units 64

// occupancy 75%

Shader Cores (ALUs)

Each SM (NVIDIA) or CU (AMD) contains 64–128 shader processors. RTX 4090: 16,384 CUDA cores across 128 SMs. Peak: 82.6 TFLOPS FP32.

VRAM & Bandwidth

GPU memory bandwidth is the primary bottleneck for texture-heavy scenes. RTX 4090: 1008 GB/s. Texture caches (L1/L2) reduce bandwidth pressure.

Warp / Wavefront

32 threads (NVIDIA warp) or 64 threads (AMD wavefront) execute in lockstep. Divergent branching serializes execution — minimize branching in shaders.

Ray Tracing Cores

Dedicated BVH traversal hardware (RTX). Handles triangle-ray intersection at hardware speed. Enables real-time reflections, shadows, and GI.

// Frame Timing

Real-time graphics demands consistent frame delivery. At 60fps, each frame has 16.67ms budget. CPU submits draw calls; GPU executes them in parallel. GPU timing tools (NVIDIA NSight, RenderDoc) reveal which passes consume the most time.

frame timeline — GPU pass breakdown

// target_fps 60 fps

// scene_complexity 50%

CPU Frame Time

Game logic, animation, physics, draw call submission. CPU sends command buffer to GPU. Goal: ≤8ms for 60fps split between CPU and GPU.

GPU Frame Time

Shadow map rendering, G-buffer pass, lighting pass, post-processing. GPU stalls occur when VRAM bandwidth or shader ALU are saturated.

V-Sync & Tearing

V-Sync locks frame presentation to display refresh, preventing tearing at cost of latency. NVIDIA G-Sync / AMD FreeSync enable variable refresh rate.

GPU-Driven Rendering

Multi-Draw Indirect (MDI) lets GPU fetch draw calls from VRAM buffer, reducing CPU-GPU synchronization. Enables 10–100× draw call count increase.

// Lighting Systems

Real-time lighting has evolved from per-vertex Gouraud shading to physically-based deferred rendering with ray-traced global illumination. Each technique trades accuracy for performance within the frame budget.

realtime lighting simulation — move cursor

// light_count 8

// roughness 0.40

Deferred Shading

Geometry pass writes material data to G-buffer (albedo, normal, roughness, depth). Lighting pass reads G-buffer — decouples light count from geometry complexity.

PBR / Cook-Torrance

Physically-based BRDF. Microfacet model: GGX distribution, Smith geometry function, Fresnel-Schlick approximation. Energy-conserving, artistically predictable.

Lumen (UE5)

Software and hardware ray-traced global illumination. Radiance cache for diffuse GI. Screen space for short-range reflections. Lumen proxy meshes for SDF tracing.

SSAO / GTAO

Screen Space Ambient Occlusion darkens corners and crevices. Ground Truth AO (UE5) uses temporal accumulation for higher quality at similar cost.

// Render Engines

Real-time render engines embed decades of rendering research into accessible pipelines. The gap between real-time and offline rendering has nearly closed — UE5 Nanite + Lumen at 60fps produces imagery indistinguishable from path-traced renders to most viewers.

engine capability comparison

Unreal Engine 5

Nanite virtualized geometry, Lumen GI, Chaos physics. Industry standard for virtual production, games, and architectural visualization.

Unity HDRP

High Definition Render Pipeline. Deferred rendering, volumetric fog, TAA. Excellent multi-platform targeting. C# scripting with Burst compiler.

Godot 4

Open source. Vulkan backend, Global Illumination via SDFGI and Voxel GI. GDScript + C# + C++. Lightweight, no licensing fees.

TouchDesigner

Real-time node-based. GLSL shaders, TOP/SOP pipelines, Vulkan backend (TD2023+). The live performance and installation standard.

// Procedural Rendering

Procedural techniques generate content algorithmically rather than storing it explicitly. Noise-based terrain, signed distance fields for geometry, procedural textures, and L-system vegetation reduce storage requirements while enabling infinite variation.

procedural generation — realtime

// noise_octaves 4

// frequency 0.30

SDF Rendering

Signed Distance Fields represent geometry as distance functions. Ray marching through SDF enables complex implicit surfaces with zero triangles. Used in Unreal Nanite, font rendering, and AO.

Compute Shaders

General-purpose GPU computation. Particle systems, fluid simulation, physics, and procedural mesh generation all happen on GPU via compute passes.

Noise Functions

Perlin, Simplex, Worley, FBM. Basis for terrain, clouds, organic textures. GPU-optimized implementations run millions of noise samples per frame.

Mesh Shaders

DX12 Ultimate / Vulkan feature replacing vertex + geometry shaders. Amplification shader culls meshlets; mesh shader generates geometry. Powers Nanite.

// Practitioners

These engineers, researchers, and artists define the field — advancing the science of real-time rendering while making it accessible to creative practitioners.

🔬

John Carmack

id Software / Meta

BSP trees, portal rendering, lightmapping, MegaTexture. Invented much of what modern engines treat as given. Quake's renderer remains a study in ingenuity.

🏗️

Tim Sweeney

Epic Games / UE5

Designed Unreal Engine's architecture. Pushed virtual production. Nanite and Lumen represent his vision of closing the gap with offline rendering.

📐

Brian Karis

Epic Games

Authored UE4's PBR shading model (Real Shading in Unreal Engine 4, SIGGRAPH 2013). Lumen GI architecture. TAA temporal anti-aliasing systems.

🌊

Inigo Quilez

Meta / Shadertoy

SDF ray marching, smooth minimum operators, procedural noise. Co-founded Shadertoy. His GLSL work defined creative coding aesthetics for a generation.

⚡

Graham Sellers

AMD / OpenGL

OpenGL SuperBible author. GPU-driven rendering advocacy. Persistent buffers, bindless textures — modern OpenGL techniques for reducing CPU overhead.

🎮

Naty Hoffman

SIGGRAPH / Industry

Physics-based rendering theory. Co-wrote "Physically Based Rendering" with Matt Pharr and Greg Humphreys. Consultant to major game studios.

🔭

Alex Evans (Mmalex)

Media Molecule / Sony

SDF-based rendering in Dreams (PS4). Traced a path from GPU voxel research to a consumer game with fully sculpted real-time geometry. SIGGRAPH 2015.

🌐

Patrick Cozzi

Cesium

3D Tiles specification, glTF format. Real-time geospatial rendering at planetary scale. Making GPU-accelerated earth visualization accessible via WebGL/WebGPU.

// Glossary

Vertex Shader: Programmable GPU stage that transforms vertex positions from object space to clip space. Runs once per vertex. Outputs interpolated varyings to fragment stage.
Fragment Shader: Computes per-pixel color. Also called pixel shader. The primary tool for visual effects, texturing, and lighting in real-time rendering.
Rasterization: Conversion of vector geometry (triangles) to discrete pixels. Determines which screen pixels a triangle covers and interpolates vertex attributes.
Deferred Shading: Multi-pass technique storing geometry data in a G-buffer then computing lighting in a full-screen pass. Allows many lights at low cost.
PBR: Physically Based Rendering. Microfacet BRDF model that conserves energy and produces consistent results under all lighting conditions.
G-Buffer: Geometry Buffer. Set of textures storing albedo, normal, roughness, metalness, and depth for deferred shading. Typically 4–6 render targets.
Draw Call: CPU command to GPU to render a set of geometry. Minimizing draw calls reduces CPU overhead. GPU-driven rendering submits draws from GPU memory.
TAA: Temporal Anti-Aliasing. Accumulates sub-pixel samples across multiple frames to achieve smooth edges. Industry standard. Causes ghosting on fast motion.
Nanite: UE5 virtualized geometry system. Streams and renders as many triangles as there are pixels — no LOD transitions or polygon budgets for static meshes.
Lumen: UE5 dynamic global illumination. Combines ray-traced surface cache, radiance cache, and screen-space reflections for real-time GI at 60fps.
Compute Shader: GPU shader stage for general-purpose computation. No fixed-function pipeline. Used for physics, particles, post-processing, and procedural generation.
SDF: Signed Distance Field. Stores the nearest surface distance per voxel. Used for font rendering, soft shadows, AO, and implicit surface ray marching.
VRAM: Video RAM on the GPU. Stores textures, meshes, framebuffers, and shader data. Limited resource — typically 8–24GB. VRAM overflow causes stuttering.
Warp: NVIDIA term for 32 threads executing in lockstep on a SM. AMD equivalent: wavefront (64 threads). Divergent code paths serialize warp execution.
BVH: Bounding Volume Hierarchy. Tree structure for accelerating ray-geometry intersection. Required for ray tracing. Built on GPU for dynamic scenes.
DLSS / FSR: NVIDIA DLSS and AMD FSR use AI/spatial upscaling to render at lower resolution and reconstruct higher-resolution output. 2× or 4× performance gain.
Mesh Shader: Next-gen graphics API feature replacing vertex + geometry shaders. Amplification shader selects meshlets; mesh shader computes final geometry.
Overdraw: Rendering the same pixel multiple times due to overlapping transparent geometry. Major performance issue. Sort back-to-front or use OIT techniques.
Occlusion Culling: Skipping draw calls for geometry hidden behind other objects. Hardware occlusion queries, hierarchical Z-buffer, and HZB culling reduce wasted GPU work.
WebGPU: Modern successor to WebGL. Direct access to Vulkan/Metal/D3D12 from browser. Enables compute shaders and GPU-driven rendering on the web.

REALTIME GRAPHICS