Open worlds: terrain, foliage, sky and streaming

Code: crates/sparklight-world (the world), a world hook in sparklight-3d (fog and aerial perspective in the render pipeline hooks, crates/sparklight-3d/src/atmosphere.rs, shaders/hooks.wgsl), crates/sparklight-scene3d/src/open_world.rs (the world inside a Space3D), crates/sparklight-control/src/commands/space3d/terrain.rs (the commands). Sample: samples/world. Tests: crates/sparklight-world/tests/world_goldens.rs (goldens goldens/3d/world_*.png), crates/sparklight-control/tests/world.rs. Design background: docs/AAA_ROADMAP.md (terrain, foliage, sky, fog; scene streaming).

An open world is data: a WorldDef (JSON) plus an edit log. The same definition and the same edits rebuild the same world, bit for bit (hash checked). Nothing in it is a mesh somebody modelled: the ground comes from noise, roads and lakes carved into it; materials come from rules on height and slope; trees, grass and rocks come from scatter rules; the sky from an atmosphere model and a clock.

Five-minute workflow

spark call space3d.new '{}'
spark call terrain.create '{"path": "samples/world/world.json"}'            # a world from a file (or {"definition": {...}})
spark call terrain.height '{"x": 0, "z": 0}'                                # ground, slope, layers, road, water
spark call terrain.raise '{"x": 100, "z": 40, "radius": 30, "amount": 12}'  # edit: a hill (raise lower smooth flatten paint hole)
spark call world.stream.drive '{"path": [[-1500,0,-1200],[0,0,0],[1500,0,1000]], "speed": 40}'   # headless drive: cells stream
spark call world.time '{"hours": 18.8}'                                     # dusk
spark call space3d.capture '{"eye": [0,60,0], "target": [200,40,150], "tier": "high"}'
spark call space3d.scenario.run '{"path": "samples/world/scenarios/drive.scenario3d.json"}'

In Rust: World::new(def) (CPU: terrain field and splat control map), world.attach_renderer(&mut renderer, "high"), then per frame world.update(&camera, Some(&mut physics), Some(&mut renderer)) returns the draw items (renderer.scene.items = ...) and puts the sun and fog into the renderer. A game that owns a Space3D calls space.set_open_world(def); the scene steps stream cells, and space.world_render(...) (the windowed player and space3d.capture call it) adds the draw items.

Coordinates and sizes

X east, Z south, Y up, metres. The world is the square [-size/2, size/2]^2 centred on the origin, cut into square cells (cell_size, 256 m by default). Height samples sit on a grid of terrain.base_spacing (1 m); size must be a whole number of cells and of spacings. 4 km x 4 km at 1 m is the sample; the control map (materials) is control_resolution texels per side over the whole world (2048 = 2 m per texel for 4 km). Worlds up to 262 km wide validate; the control map and the generation time grow with the area, so a very large world wants a larger base_spacing and a larger control_resolution budget.

The definition (WorldDef)

Field	Meaning
name, seed, size, cell_size	identity, the seed of every noise, the side of the world and of a cell (metres)
terrain	base_spacing, node_resolution (quads per mesh node, power of two 8..128), base_height, hills (hill_amplitude, hill_scale, hill_octaves), mountains (mountain_*: ridged noise where a continental noise is high), edge_height (a mountain wall towards the world’s edge), flat_radius and flat_blend (a flat start area), control_resolution
layers	up to 8 material layers: name, albedo, variation, roughness, pattern (soft, grain, rock, pebbles, smooth), relief, tile_size, height_blend, and a generator rule (height window, slope window with soft edges, noise). Layer 0 takes what the others leave
roads	polylines points: [[x, z], ...], width, shoulder, smoothing, layer: the ground is graded onto a smoothed profile along the line
lakes	round basins: center, radius, depth, level (default: just under the ground at the centre), layer (shore material). A water surface is drawn with the shader library’s water shader
scatter	rules, see below
objects	models placed by hand (model, position, yaw, scale, on_ground), owned by the cell they stand in
atmosphere	the sky model, the clock, fog, the volumetric tier
wind	direction (degrees), strength and speed multipliers for grass, bushes and trees
streaming	load_radius, unload_radius, collider_radius, cells_per_frame, uploads_per_frame, upload_bytes_per_frame, threads, install_latency_frames

WorldDef::validate (run by terrain.create) gives coded errors with JSON paths: E_WORLD_SIZE, _SPACING, _NODE, _CELL, _CONTROL, _LAYERS, _ROAD, _LAKE, _STREAMING, _SCATTER, _ATMOSPHERE, E_WORLD_JSON (unknown fields are rejected).

Terrain: chunks, continuous LOD, no cracks

The ground is a quadtree of square nodes (CDLOD, Strugar 2010). Every node, whatever its size, is the same grid of 33 x 33 vertices; a node of level l spans 32 m * 2^l, so vertices are 1 m * 2^l apart. The camera picks the nodes: a node is drawn when the camera is farther from its bounding box than scale * span of the level below, closer it is replaced by its four children (lod::select; scale is the quality tier’s LOD scale, 2.0 low to 4.5 ultra). The tree has 8 levels for a 4 km world.

Continuous: each vertex carries in its UV the height it would have on the next coarser grid and the parity of its grid position (mesh.rs). The terrain shader (vertex_offset of shaders/terrain.wgsl) moves the vertex onto the coarser grid by its distance to the camera over the last 35 % of the node’s range, so a level never pops. Crack free: where a level-l node meets a level-l+1 node the morph is complete (the neighbour is selected only beyond the range at which level l is fully morphed, and the node’s bounding box distance is a lower bound of every vertex’s distance), so the two share their edge vertices exactly. Proven three ways: a unit test builds the neighbours and compares the morphed border with the coarse border vertex for vertex (and, as the negative control, shows that without the morph they differ); a selection test checks that adjacent nodes never differ by more than one level; and the GPU test lod_has_no_cracks_without_skirts renders the terrain without skirts in front of a magenta background from five very different cameras and finds no background pixel below the horizon. Skirts (a strip hanging below every node edge, on by default) are only a safety net for sub-texel rounding.

Normals come from central differences at the node’s own cell size; meshes are built on background threads from a Terrain snapshot (reference counted, so edits do not wait for them) and uploaded within the per-frame budget (uploads_per_frame, upload_bytes_per_frame). A node whose mesh is not ready is covered by its nearest resident ancestor, so there are no holes while detail streams in. Unused meshes are evicted least recently used first above cache_bytes (256 MB).

Materials (splat map). The generator writes a control map: per texel the weights of up to 8 layers (bytes summing to 255) from the layer rules, roads and lake shores, and a hole flag. The terrain shader samples it by world position and blends the layers: height blending (the relief of each layer decides which pokes through), triplanar projection on slopes steeper than about 0.25 (1 - n.y), a fine detail pass close to the camera, large-scale colour variation against visible tiling, and a fade to each layer’s mean colour with distance. Layer textures are generated tileable atlases (textures.rs: albedo with height in alpha, relief with roughness in alpha, 4 x 2 tiles of 256 with a 16 texel wrap gutter); TerrainGpu::set_atlas replaces them with real texture sets.

Holes are bits per finest cell (terrain.hole): the shader discards the flagged texels (a HOLES variant of the material, switched on only while holes exist) and the collider drops the cells.

Colliders. One rapier heightfield per cell, built from the same samples and the same triangulation (the anti-diagonal of each cell) as the finest mesh, so a body rests on the surface it is drawn on (tested: collider and surface agree within 1 cm). ShapeDesc::Heightfield of sparklight-physics-fx takes rows, cols, heights, size and per-cell holes. Terrain bodies carry the user data TERRAIN_USER_DATA | cell.

Editing (data)

An edit is a record: {"op": "raise" | "lower" | "smooth" | "flatten" | "paint" | "hole", "x", "z", "radius", ...} with falloff (share of the radius over which the brush fades), amount, strength, height, layer, erase, fill; or {"op": "scatter_remove", "ids": [{"rule", "gx", "gz"}, ...]} (see Foliage and scatter). terrain.raise and friends build the record; terrain.edit takes it as it is; terrain.edits returns the log; terrain.undo / terrain.redo step through it. Heights are stored as sparse deltas over the generated ground (blocks of 64 x 64 samples), paint is stored in the control map (painted texels keep their paint when the ground under them changes), holes as bits. An edit that changes the shape of the ground drops the cells, mesh nodes and grass patches under it; they rebuild from the new ground (colliders included, and the bodies resting on them are woken). Replaying the log on a fresh world gives the same hash (tested), and the log is saved with the scene or by terrain.save.

Foliage and scatter

A scatter rule (scatter[]): kind (grass, bush, tree, rock), model (below), seed, density (per 100 m^2), min_spacing, slope and height windows, layers (accepted where any listed layer weighs at least its weight) and avoid_layers, road_margin, water_margin, scale, align (lean into the slope), patchiness and patch_scale (clumping), draw_distance, lod_distances, wind, tint colours, blade_height.

model is the look of the instances and is checked, never guessed. A prebuilt look of the rule’s kind: trees conifer (spruce, pine), broadleaf (oak), birch; bushes bush; rocks boulder (rock); grass grass. An empty model, or the field’s default grass on a rule that is not grass, is the kind’s default look. Or, for a tree, a bush or a rock, a glTF model: a model key, a path below the assets folder with an extension (models/ruin.glb; builtin: keys work too). Any other name is E_WORLD_SCATTER at /scatter/<i>/model when the definition loads (a bare word that is not a prebuilt look is a typo, not a file). A model key is read when the world opens (World::open(def, &resolver); a scene’s world uses the scene’s assets folder, glTF and built-in keys): a file that is missing, has no triangles or no size, or has more than 500000 triangles is E_WORLD_MODEL at /scatter/<i>/model, and a world opened with World::new (no resolver) refuses a model key. A glTF model stands with its origin on the ground, +Y up, in metres; it draws with its own materials (no wind, no tint) and is sized from its bounding box (culling sphere) and given three levels of detail from it: the model itself, then vertex-clustered copies with half-size and quarter-size textures (cell = 1/24 and 1/9 of the box diagonal), switched at lod_distances[0] and [1] (a model too light to simplify keeps its model for the level). drawn_by_lod counts them as 0, 1 and 2.

Instance ids and removal. Every tree, bush and rock has a stable id {rule, gx, gz}: the rule’s index in scatter and the cell of the rule’s jitter grid it stands on (grid spacing max(sqrt(100 / density), min_spacing)). The id does not depend on streaming, cells or the order things load (tested). It changes meaning when the rule’s density, min_spacing or seed, or the world seed, change (another grid), and when a rule is inserted before it in scatter (the index); a removal recorded earlier then hides whatever instance stands on that grid cell now. Instances a rule does not place (rejected by slope, height, layers, water, roads, clumping or holes) have no instance, but their grid cell still has an id: removing it is accepted, records the id, and hides the instance if a later edit of the ground makes the rule place one there. Removal is an edit, {"op": "scatter_remove", "ids": [...]} (1 to 100000 ids of non-grass rules, at most 1000000 removed ids in a world), so it is in the edit log, terrain.undo and terrain.redo step over it, the log replays it, scene files and snapshots keep it and world.hash includes it. The terrain holds the removed set, every cell build leaves those ids out, and the loaded cells that hold them are re-scattered in place (no collider rebuild). The per-instance variety (which tree shape) keys on the instance’s place in the rule’s generation order with removed ones counted, so removing an instance changes nothing else in the picture. Grass is generated in patches and has no ids. world.scatter.list gives the ids in a disc (whether or not their cells are loaded; at most limit of them, with total and truncated); world.scatter.remove takes ids or a disc near, removes all of them and echoes at most 1000 ids (ids_total and truncated say when there are more). area, near and each id are strict objects: an unknown key is an error.

Placement is a jittered grid keyed on (seed, grid cell), never on the region asked for, so a streamed world has the same forest as one built in one piece and any cut of the world into regions yields the same instances (tested). The same seeds give the same hash in two processes.

• Grass: clumps merged into 32 m patches (one mesh per patch, positions in world space), two levels of detail (a third of the clumps, larger, beyond grass_lod_distance), built on background threads and uploaded in the budget; the world::grass shader bends blades with a travelling gust field (the shader library’s wind approach: vertex_offset weighted by height) and shrinks them to the ground towards the draw distance so a patch fades out without popping.
• Trees: three shapes per rule, each with LOD0 (about 500 to 900 triangles of leaf cards and bark), LOD1 (about 150) and an impostor: three crossed cards textured with an image of the tree baked from its own geometry by a small software rasteriser (foliage_models::bake_impostor, no GPU needed, deterministic). Instances switch LOD by distance (lod_distances). Wind bends a tree from its base in proportion to the square of the height with a gust field shared by the forest, read from the object matrix so the same mesh sways differently for every instance (world::tree shader). Leaf normals are rounded so a crown shades like a volume.
• Bushes and rocks: single-level instanced models (leaf-card ball, displaced icosphere).
• Culling: hierarchical and on the CPU: grass by patch (distance ring, then frustum), instances by cell, distance and a sphere against the frustum; the renderer batches the survivors of one model into one draw call. There is no GPU-driven culling yet: the renderer has no indirect draw path.

Atmosphere

atmosphere (AtmosphereDef): a physically based sky, single scattering by air (Rayleigh) and aerosols (Mie) through an exponential atmosphere around a spherical planet, integrated numerically along the view ray with the optical depth of the sun path taken per sample, plus a multiple-scattering term (without it the horizon is yellow and the sky too dark). Units are the engine’s lighting: a white surface in full sun reads 1 at sun_intensity = pi. time_of_day (hours), latitude, day_of_year and north place the sun on its arc (Atmosphere::sun): its light is its colour after the long way through the air (red at dusk), faded out over the last degrees; after sunset a cool moon takes over (night_light). The sky is a 64 x 64 cube from the camera’s height; it is the background and the image based lighting (specular cube, SH irradiance), and is rebuilt only when the sun moved about half a degree. The sun becomes the scene’s directional light with cascaded shadows. Exposure: the engine’s light is physically scaled (a sunlit white surface reads 1), which a camera at exposure 0 shows dark and blue-black, so the world sets the renderer’s manual exposure from the sun every frame (exposure_ev, 1.15 stops by day, up to exposure_night_ev more once the sun is well below the horizon; exposure_control: false leaves it to the game).

Fog and aerial perspective are applied by the renderer’s hook_finish_color to every surface: exponential height fog (density, falloff, start, opacity cap) plus the extinction of the atmosphere itself with the same Rayleigh/Mie coefficients as the sky, so distant things turn blue and hazy and the far hills reach the horizon colour; the in-scattered light is the sky cube in the view direction (so the horizon and the hills agree at any time of day) plus a sun glow (Henyey-Greenstein). Settings are RenderSettings::atmosphere (AtmosphereSettings, plain data; off by default, so scenes without a world pay nothing). The racing demo faked this in its shaders; it can switch to these settings.

Volumetric fog (volumetric.enabled, tier medium and up; steps scale down below ultra): the view ray from the camera to the surface is marched (at most max_distance) in the same hook, and the sun reaches each sample through the cascaded shadow map, so light shafts stand between trees. It applies to surfaces, not to the sky pixels; world_volumetric_fog.png shows it (a low sun between conifers).

Streaming

The world is cut into cells (cell_size). A cell owns its collision heightfield, its placed objects and the scatter instances of trees, bushes and rocks (grass has its own patches, mesh nodes their own cache). Each frame World::stream requests the cells within load_radius of the camera, nearest first, builds them on background threads, installs at most cells_per_frame of the finished ones, drops cells beyond unload_radius (hysteresis) and keeps one heightfield collider per cell within collider_radius in the physics world. Mesh and patch uploads have their own per-frame budgets (uploads_per_frame, upload_bytes_per_frame, patch uploads per frame).

Deterministic by design: loads run on background threads, but a cell becomes part of the simulation (the cell table, its collider, world.hash) at a frame fixed when it is requested, never by thread timing. A cell requested at frame F is due at F + install_latency_frames (default 3); each frame installs the due cells in request order, at most cells_per_frame of them, and waits for a build that is not finished yet (the streamer hands results back in request order, stream.rs). The latency is the time the builds get to finish before a frame would ever wait; a slow machine blocks, it never changes the result. threads (0 builds inline at the install point), the machine’s load and the control-map pool size therefore do not change the cells, the frames, the colliders or the hash. The render side is separate: uploads_per_frame, upload_bytes_per_frame and the patch upload budget only decide when meshes reach the GPU, are not read by the simulation, and have no effect on world.hash. Snapshots keep the due frame of every pending cell. deterministic in old files is accepted and ignored. Tested in crates/sparklight-world/tests/stream_determinism.rs: the valley (samples/world/world.json) driven along a path with streaming threads 0, 2, 8 and pool sizes 1, 2, 4, 8 under CPU contention gives one world hash and one event sequence, and a replay (camera path plus edit log) reproduces them. Across processes space3d.determinism.check covers the world because it is part of snapshots and of the state hash.

world.stream.status reports cells loaded and pending, memory, colliders, instances, LOD nodes per level, grass patches and foliage counts.

Rendering integration

Space3D owns the world (space.open_world). Its step streams and keeps colliders; Space3D::world_render(renderer, &mut scene, settled) adds the draw items and the sun (the windowed player calls it with settled = false, space3d.capture with true: a capture shows the final detail), world_apply_settings writes the fog. Space3D::to_json stores {"world": {"definition": ..., "edits": [...]}}; snapshots, restores and the state hash include it.

Performance (this Mac, 1920x1080)

Measured on an Apple M5 Max (Metal), release build, 1920x1080, FXAA, the sample world (4 km, forest, grass, lake), 48 frames of a camera walking at 0.9 m/frame with streaming running (2 threads), the first 6 frames of each view discarded. GPU is the frame’s GPU time from timestamp queries (first pass start to last pass end), CPU the time the renderer spent recording, world the time World::stream + render_items took (selection, culling, requests, uploads), wall the whole render_to_rgba call including the read-back of the 8 MB image (an upper bound that a windowed player does not pay). Reproduce: cargo test -p sparklight-world --release --test world_perf -- --ignored --nocapture (writes .gwdev/out/world/perf.json).

Tier	View	GPU ms	CPU ms	world ms	wall ms	draw calls	items	terrain nodes	grass patches	trees drawn
low	meadow	3.2	1.2	0.32	9.8	69	910	51	0	858
low	forest	8.1	1.8	0.48	30.2	72	1406	54	0	1351
low	overview	6.5	1.3	0.20	16.0	27	412	21	0	390
medium	meadow	5.5	3.6	1.13	19.2	92	2148	64	11	2072
medium	forest	6.3	4.2	0.87	22.4	94	2535	60	11	2463
medium	overview	3.2	3.4	1.46	14.7	35	1653	32	0	1620
high	meadow	7.5	7.8	1.18	32.6	117	3241	77	15	3148
high	forest	9.1	9.5	1.17	34.4	128	4009	84	14	3910
high	overview	12.2	5.0	0.65	34.2	51	2957	44	0	2912
ultra	meadow	49.2	5.6	0.93	63.0	156	3635	112	23	3499
ultra	forest	55.7	5.1	0.99	66.0	164	5294	113	21	5159
ultra	overview	33.0	5.5	0.85	46.6	71	4781	64	0	4716

Reading: the low, medium and high tiers stay near or under 12 ms of GPU time in every view (about 80 fps or better at 1080p on this GPU). ultra is much more expensive on purpose: the volumetric fog marches 24 shadow-mapped steps per pixel on top of the renderer’s heaviest shadow and post settings; it is a 30 fps tier on this machine (33 to 56 ms). Grass is off at low. The forest view costs the most (about 4000 tree items; the renderer batches them into 128 draw calls). Numbers from other GPUs will differ; none are claimed for them.

Known limits (and what is not done)

• GPU-driven culling is not implemented (CPU hierarchical culling instead, see above). A compute culling pass with indirect draws needs a draw-indirect path in the renderer’s frame graph.
• Terrain comes from procedural noise plus edits; there is no importer for heightmap files or world-machine tiles yet. The control map is one texture for the whole world (no virtual texturing), which bounds the material resolution for very large worlds.
• Trees and bushes are procedural card models; imported tree assets can be used through the model cache but have no automatic LODs or impostors. Impostors are three crossed cards (no view-dependent octahedral atlas).
• Volumetric fog marches per surface fragment (no froxel grid, no temporal reprojection) and does not touch the sky pixels; there are no clouds.
• Water is a flat disc per lake with the stylised water shader; there are no rivers or oceans yet.
• Editing a very large brush rebuilds the cells under it (a few frames of streaming); the control map is copied once when an edit arrives while a background thread still holds it.