Scene graph

A tree of nodes, a camera, some lights. Everything on this page needs no device at all, so bounds, culling, framing and picking are testable without one.

Nodes #

SceneNode holds a local transform and caches a world matrix. When children are attached, the world matrix invalidates down the subtree, and a version stamp (not a dirty flag) records that it changed.

final node = SceneNode(name: 'pivot')
  ..setPosition(0.0, 1.2, -3.0)
  ..setRotationYawPitchRoll(math.pi / 4, 0.0, 0.0)
  ..setUniformScale(0.5);

parent.add(node);                 // reparents, keeping the local transform
node.lookAt(Vector3(0, 1, 0));    // orients towards a world point
node.setLocalForward(Vector3(-0.4, -1.0, -0.3));

traverse, findByName, readScenePosition, attach (reparent without moving in scene space) and removeFromParent are the rest of the surface. A cycle guard refuses to make a node its own ancestor.

Scene #

Scene is the root plus registries. Nodes announce themselves through onAttachedToScene, which is why adding a MeshNode makes it drawable and adding a LightNode makes it light without a separate call.

final scene = Scene()
  ..add(meshNode)
  ..add(lightNode)
  ..add(cameraNode);

final bounds = scene.computeBounds(visibleOnly: true);
final sun = scene.firstLightOfType(LightType.directional);

Cameras and projections #

final camera = CameraNode(
  projection: const PerspectiveProjection(fieldOfView: 1.05, near: 0.1, far: 220.0),
  name: 'player',
)
  ..setPositionFrom(eye)
  ..lookAt(target);

A camera's Projection is a PerspectiveProjection or an OrthographicProjection for an ordinary view, both with copyWith, or an OffAxisProjection for a frustum whose four sides are given separately. Three more wrap another projection: TiledProjection and CropProjection for a still drawn in tiles, and JitteredProjection for the sub-pixel shift temporal anti-aliasing needs. A CameraView reads camera.viewProjection(aspect).

A camera calibrated to a photograph is an OffAxisProjection. A calibration gives the focal lengths fx, fy and the principal point cx, cy in pixels of an image width by height, and the principal point is rarely in the middle, which no field of view reproduces. The four sides are tangents of the angles from the view axis, negative to the left and below:

Projection fromIntrinsics({
  required double fx,
  required double fy,
  required double cx,
  required double cy,
  required double width,
  required double height,
}) => OffAxisProjection(
  tanLeft: -cx / fx,
  tanRight: (width - cx) / fx,
  tanUp: cy / fy,             // cy is measured down from the top
  tanDown: -(height - cy) / fy,
  near: 0.05,
  far: 200.0,
);

The frustum states its own shape, so it ignores the aspect the view asks with: show it in a view of the image's aspect (an AspectRatio(aspectRatio: width / height) around the view), or the picture is stretched by the difference. The camera then stands where the calibration's extrinsics put it, looking down its own −Z.

Orbit camera #

final orbit = OrbitController(camera, distance: 6.0)
  ..rotate(dx, dy)                   // pixels, times rotateSensitivity
  ..zoom(1.1)
  ..pan(dx, dy, viewportHeight: size.height)
  ..frameBounds(scene.computeBounds())
  ..apply();
orbit.syncProjectionDepth(camera);   // near/far to fit what is framed

animateTo swings the view to a yaw and a pitch over a quarter of a second, and with target: and distance: it is a fly-to: a part picked in a model, a site on a map, the view leaving and arriving as one movement on the same ease. The controller is driven by the caller's clock (update(dt) each frame), and a drag cancels the travel. minPitch and maxPitch keep a viewer from looking under a car or through a floor, and minYaw and maxYaw hold it to a side; every turn, animateTo and frameBounds stay inside them.

final viewer = OrbitController(camera, distance: 4.0, minPitch: 0.05);
viewer.animateTo(target: Vector3(0.4, 1.1, 0.0), distance: 1.5);

What the widget answers, and what it does not yet. OrbitCamera3D in a Scene3D is this controller with gestures: one finger or a mouse drag turns it, a pinch or the wheel moves it in and out, and a fling spins on and slows. It does not pan, and a two-finger twist does nothing. It also has no controller you can hold in rc.2, so a "reset view" button or a fly-to on a tap cannot reach it. An application that needs either holds an OrbitController over its own camera in a Flutter3dView and writes its own gestures, which is a dozen lines:

var twist = 0.0;
final view = GestureDetector(
  onScaleStart: (ScaleStartDetails d) => twist = 0.0,
  onScaleUpdate: (ScaleUpdateDetails d) {
    if (d.pointerCount < 2) {
      orbit.rotate(d.focalPointDelta.dx, d.focalPointDelta.dy);
    } else {
      // Two fingers: turned about the target by the angle they turned,
      // and slid across the view by the way their middle moved.
      orbit
        ..rotate((twist - d.rotation) / orbit.rotateSensitivity, 0.0)
        ..pan(d.focalPointDelta.dx, d.focalPointDelta.dy,
            viewportHeight: size.height);
      twist = d.rotation;
    }
  },
  child: Flutter3dView(
    scene: scene,
    camera: camera,
    onFrame: (EngineFrame frame) => orbit.update(frame.seconds),
  ),
);
// A button anywhere: orbit.animateTo(yaw: 0.0, pitch: 0.35);

The declarative versions come in rc.3: an OrbitCamera3DController for reset, framing, auto-rotate and fly-to from a button, Scene3D(gestures: ...) to choose which gestures the view takes (a view inside a scrolling page that should leave one-finger drags to the page), onHover, Camera3D(viewport: ...), and Flutter3dView(onTap: ...).

Lights #

Up to eight lights of any type, packed into vec4[8] uniform arrays with the count as a uniform, so switching a light on or off never rebuilds a pipeline.

scene.add(LightNode(
  type: LightType.spot,
  color: const LinearColor(1.0, 0.86, 0.7),
  intensity: 69500.0,          // candela, for a camera exposed for daylight
  range: 14.0,                 // glTF's range window
  innerConeAngle: 0.25,
  outerConeAngle: 0.5,
  castsShadow: true,
)..setLocalForward(Vector3(0.0, -1.0, 0.2)));

A light made with no intensity is the sun or a bulb, as Light3D's are: LightNode.sunIlluminance, 15 000 lux, for a directional light, and LightNode.lampLumens, 800 lumens, for a point, spot or area light, given as candela over the solid angle its cone covers (about 63.7 cd for a point). LightNode.defaultIntensityFor(type) answers the number. Before 1.0 every light without one got the same 5 790.6 whatever its type, which made a default point lamp some 73 000 lumens; that factor is LightNode.legacyUnit now, a conversion and not a default, so a light written as x in the old unit is x * LightNode.legacyUnit.

LightBuffer gathers them each frame and falls back to useDefaultLight() for a scene with none, so an unlit-by-accident scene is visible rather than black.

The BVH, culling and picking #

One BVH over world bounds, shared by frustum culling and by picking, rebuilt only when something actually moves, which the version stamps make cheap to ask.

flowchart TB
  nodes["MeshNode world bounds"] --> spheres["scene_spheres<br>packed Float32List"]
  spheres --> bvh["SceneBvh.refresh(spheres, count, stamp)"]
  bvh --> cull["frustum culling<br>per CameraView"]
  bvh --> pick["Raycaster.intersectScene"]
  cull --> list["RenderList"]
  pick --> hit["HitResult: node, point, normal, distance"]

Picking #

// x and y are widget coordinates from the top left, exactly as Flutter
// reports a pointer. The Y flip and the aspect ratio live in here rather
// than at the call site, because those are the two things that get silently
// reversed.
final hit = Raycaster()
    .setFromScreen(camera, event.localPosition.dx, event.localPosition.dy,
        width: size.width, height: size.height)
    .intersectScene(scene);

if (hit != null) {
  settings = settings.copyWith(highlighted: <SceneNode>[hit.node!]);
}

Three intersection routines back it, all allocation-free: ray/AABB, ray/sphere and Möller–Trumbore ray/triangle. A node whose mesh kept its source MeshData is tested against triangles; one that did not falls back to its bounds.

A reshaped mesh is picked as drawn. DeviceMesh.overwriteVertices writes the new vertices into the mesh's kept source as well as into the device buffer, so a tap after a terrain edit or a sculpt stroke hits the surface on the screen, not the one uploaded. The first overwrite makes the mesh its own copy, leaving the MeshData it was uploaded from alone. The node's bounds follow too: MeshNode keys them on the mesh's version, which every overwrite advances, so nothing needs markBoundsDirty.

What a pick names is a node. HitResult.node is the MeshNode the ray met, and Renderer.pick answers the same kind of thing from the screen, as a PickResult whose node it is (picking by pixel). For an InstancedMeshNode that is the whole batch: the ray is tested against the union of its instances' bounds, and the pixel pass writes one number for the draw. Which of five hundred chairs was clicked is not in either answer yet; when the pixel pass learns it, it arrives as a field of PickResult, with no new method to call. Until then, the application asks the batch itself, which costs a box test per instance and no triangles:

// After intersectScene answered with the batch, or to test it on its own.
int? instanceUnder(Raycaster raycaster, InstancedMeshNode batch) {
  final transform = Matrix4.identity();
  final local = LocalRay.zero();
  var nearest = double.infinity;
  int? hit;
  for (var i = 0; i < batch.count; i++) {
    // An instance is placed relative to its node: compose the two, invert,
    // and carry the ray into the instance's own space.
    batch.readTransform(i, transform);
    final toInstance = batch.sceneMatrix.multiplied(transform)..invert();
    final t = rayAabb(raycaster.ray.transformInto(toInstance, local),
        batch.mesh.bounds);
    if (t >= 0 && t < nearest) (nearest, hit) = (t, i); // a miss is -1
  }
  return hit;
}

The distance comes back in lengths of the transformed direction, so compare it between instances of one batch and not against HitResult.distance. A box answer is good enough for chairs and atoms; a hole in an instance is still a hit.

A pick sees the geometry the engine's vertex stages place, not what a material moved. Both the ray and the pixel pass work from where the engine put the vertices: the ray from the mesh's kept triangles, posed for a skeleton and the morph weights, and the pixel pass with the engine's own vertex stages (plain, skinned, instanced) and a stage of its own for the fragment. A material whose vertex block moves its geometry (a flag in the wind, glyphs turned to face the camera, particles carried by a time uniform) is drawn in one place and picked where its vertices were before the block ran. Pick such a thing by its own arithmetic: the application knows where it sent it.

A pick on a model leads back to the file. A ray or the pixel pass names a MeshNode, and what the file said about the part (its name, its id, its glTF extras) is on the asset node above it. The ModelInstance the model was instantiated as walks up for you:

void describe(ModelInstance model, MeshNode picked) {
  final index = model.indexOf(picked);       // the node's index in the file
  final extras = model.extrasOf(picked);     // that node's glTF extras, or null
  debugPrint('part $index: ${extras?['sku']}');
  // The material it wears, by the name the file gave it: the asset's own,
  // or this instance's copy when it was made with shareMaterials: false.
  model.materialNamed('paint')?.baseColor = LinearColor.fromSrgb(0.8, 0.1, 0.1);
}

The index is the one in the file, so it is also the key into a table the application built from the document. A node of another instance or of the scene answers null.

A hover is a pick on every pointer move. Mesh3D.onTap and Model3D.onTap hand a Tap3D, and only for a tap: a hover arrives through a callback of its own (onHover, rc.3) and never reaches onTap. Until then a view with a camera you hold casts the same ray under a MouseRegion; it costs one BVH walk a move, which a mouse at 120 Hz affords:

import 'package:flutter/gestures.dart' show PointerHoverEvent;

MouseRegion(
  onHover: (PointerHoverEvent event) {
    final hit = Raycaster()
        .setFromScreen(camera, event.localPosition.dx, event.localPosition.dy,
            width: size.width, height: size.height)
        .intersectScene(scene);
    hovered.value = hit?.node; // a ValueNotifier the tooltip listens to
  },
  child: Flutter3dView(scene: scene, camera: camera),
)

Where a tap is in the world. hit.point is in scene space, float32 and relative to the scene's origin, which a large world moves to stay near the camera (a floating origin). scene.toWorld(hit.point) brings it back to a WorldPosition in doubles, the same wherever the origin has moved to. A world tied to the Earth through a GeoOrigin (a site, a twin, a map) turns that into a latitude, a longitude and a height with geoOrigin.geodeticOf(...), on the east-north-up plane the origin stands on. A whole globe (Earth-fixed coordinates, 3D Tiles, elevation models) is 1.x.

A Flutter widget over a point in the scene #

A measurement beside a model, a price tag over a part, a blue dot with a heading over a floor plan: a Flutter widget placed where a point of the scene lands on the glass. projectPoint does the projection, from the camera's own matrix for the view's aspect, and answers null for a point behind the eye:

Widget labelled(CameraNode camera, Size size, Vector3 point, String text) {
  final at = projectPoint(
    camera.viewProjection(size.width / size.height),
    point,
    width: size.width,
    height: size.height,
  );
  return Stack(
    children: <Widget>[
      Flutter3dView(camera: camera),
      if (at != null)
        Positioned(left: at.x, top: at.y, child: Text(text)),
    ],
  );
}

Build it again when the camera moves, from the view's onFrame or a listenable it updates. The label does not know what is in front of the point, so one that should hide behind the model casts a ray from the camera to the point first (Raycaster, above) and hides when something nearer answers. The marker itself is ordinary geometry: a DiscShape for the dot, a ring of lines a given number of pixels wide for the accuracy, a ConeShape turned to the heading. Anchored3D, a widget that follows a node, hides behind geometry and draws its callout line on its own, comes in rc.3, and so does TextNode for text drawn in the scene itself.

The viewer kit comes in rc.3 as well: clip planes and section boxes, measurements, a transform gizmo, a navigation cube, snapping, and saved views. A clip plane's cap fill is 1.x. What 1.0 is lists the rest that comes later.

LOD groups #

scene.add(LodGroup(levels: <LodLevel>[
  LodLevel(node: high, maxScreenFraction: 1.0),
  LodLevel(node: mid,  maxScreenFraction: 0.20),
  LodLevel(node: low,  maxScreenFraction: 0.05),
]));

Selection is by screen coverage, not distance: coverage already folds in the field of view and the object's size, so one threshold works for a pebble and a cathedral.

A level can also carry error:, how far its surface is from the finest one in the group's units. The converter measures it for every level it cuts. With it the renderer switches by pixels of error: the coarsest level wrong by no more than pixelError (default one pixel) at the object's nearest point, for the view's own height and projection. Levels without an error fall back to their screen fraction, and either way a level is held a tenth past its threshold (hysteresis) so a camera sitting on the edge does not make it flicker.

Skeletons #

A glTF skin decodes into a Skeleton of ordinary scene nodes, not a parallel hierarchy, so an animated parent carries its subtree the same way any node does.

skeleton.update(meshNode.sceneMatrix);   // fills the joint matrix palette
final bounds = skeleton.computeBounds(reach: 0.1);

• 64 joint matrices in a uniform array, four weights a vertex.
• A skinned primitive gets the skinned vertex layout, chosen from the data instead of from the caller. The layout and the shader are one decision, so a mesh has both or neither.
• Bounds come from the posed skeleton rather than the bind pose, because a character that reaches out of its bind-pose box gets culled at exactly the moment it becomes interesting.

View-model nodes #

ViewModelNode draws with its own field of view and its own depth range, on top of the scene. That is what a weapon held in the hands needs: a 90° scene camera makes a gun in the corner of the screen look like it is being fired from the elbow.

Next #

• Geometry & materials: what a MeshNode holds
• Assets & animation, where a skeleton comes from
• The frame: what the renderer does with all of it