Testing

15633 tests across 62 packages and 18 applications, counted the same way the the document says how many tests there are rule does: a scan of every test(/testWidgets( call. The rule holds ARCHITECTURE.md §13, the README and this page to the answer. The README went on saying 1242 across thirteen packages for as long as nothing compared it with anything. About thirty need a GPU; the architecture is what keeps the number that low.

The rows sum to 15543 rather than 15633: the remaining 90 live in packages/*/example/test, which the count includes and this table does not, among them the two seeds a new project starts from, packages/flutter3d_app/example and packages/flutter3d_game/example, and the chemistry bench, packages/flutter3d_education/example.

flutter3d_samples is not in the table and has no test/ at all: it is test data with two path constants over it, and other packages' decoder tests are what exercise it. flutter3d_conformance is missing for a different reason: it is invoked as a script harness rather than through flutter test, so it does not surface in a grep of test( calls either. See below for what that cost once.

tool/ci.sh                                   # shaders, analyze, every test
(cd packages/flutter3d_game && flutter test)
(cd packages/flutter3d_physics && dart test) # plain Dart, no Flutter needed

Four independent golden sets, not one #

97 scenes are rendered four times: through Impeller, through the software rasteriser in flutter3d_cpu, through WebGL2, and through WebGPU, the last two in a driven browser. Each backend is held to zero differing pixels against its own set, with a per-channel tolerance of 8.

The browser sets are recorded when a branch lands, not beside it (golden_web.sh holds one fixed port for the whole of its run), so a new scene is in the other sets for as long as that takes. Which scenes, and what they are waiting for, is _provisional in flutter3d_webgl/test/cross_backend_test.dart: the comparison is skipped with the reason printed instead of quietly missing, and the check beside it fails the moment a reference lands and the name is still there.

WebGPU records the same way, from the same stand. flutter3d_webgl/tool/golden_web.sh --backend=webgpu writes into flutter3d_webgpu/test/goldens. One build serves the whole suite for either browser backend, because the scene and the backend are query parameters on the page and not defines on the compile, since a define per backend would have spent the stand's entire saving on a single word. --no-build reuses the build already there, which is what makes a second backend's recording cheap.

impellercpuwebgl

cross_backend_test.dart compares them with per-scene budgets, and any new backend has to pass flutter3d_conformance before it counts as one.

flutter3d_webgpu is the backend that gets the arrangement Impeller cannot. Chrome has a real WebGPU device inside flutter test, so flutter test --platform chrome runs the whole suite against live hardware as an ordinary test file: 41 of 41, two of them passing by declining a capability the device says it has not got: the blend constant and wireframe. Two more used to be there. Rendering into a mip left the list without moving the number, because the two checks that read the capability answered a smaller question rather than skipping when it was false. The block-compressed formats left it when the device started asking its adapter which compression families it carries and requesting exactly those, and the check that had been skipping three candidates now draws a block of each. A decline is reported as a decline and never as a pass, because "the suite is green" and "the suite is green, and here is what it never asked" are different sentences.

The rest of that package's tests are deliberately split by whether they need a browser at all. The translation table, the pipeline signature and every vertex-layout refusal live in files that import neither dart:js_interop nor package:web, so they run on the VM in about a second. A typo in "less-equal" fails there, not as a pipeline a browser rejects at run time on the one machine that has a GPU. The GLSL→WGSL pipeline is checked the same way the WebGL translation is: CI regenerates the table and fails on the diff, which catches a stale table and a different compiler on the machine, since this road runs through glslangValidator and naga rather than one generator.

The same package has a second instrument in the same shape, held by the same rule: tool/surface_probe.sh measures flutter_gpu's GpuImageSurface against the asImage() path presentFrame uses, on a live GPU, and prints what each costs. The probe itself lives in the engine's example beside the entry point that runs it, not in the backend, because it reaches flutter_gpu directly and is no part of what the backend publishes. It is a measurement more than a check. Its exit code says only whether the last frame of each of the five present paths, and of the resized surface, came back holding the colour it was cleared to, which checks the image wrapping the right texture and nothing the compositor did. What it found is on the backends page.

A budget for frames meant to differ #

Zero differing pixels is the right bar for a frame that should not move. It is the wrong one for a frame that is supposed to differ from its reference a little everywhere: a lower render scale, or a stochastic effect against its sorted reference. A lower render scale moves nearly every pixel a little and looks almost the same, so a count of moved pixels cannot say how much worse the frame looks.

flutter3d_testing answers that with FLIP. flip compares two 8-bit sRGB frames and flipLinear two in linear light; both return a FlipResult holding the per-pixel error map and its mean, where 0 is identical and 1 is as different as green is from blue. It is LDR-FLIP (Andersson et al., HPG 2020) ported line for line from the reference implementation, so a value here is the value there, and a test holds it to the error maps that implementation publishes. The viewing condition defaults to defaultFlipPpd, about 67 pixels per degree.

import 'package:flutter3d_testing/flutter3d_testing.dart';

await expectMatchesGolden(frame, 'test/goldens/half_scale.png', flipBudget: 0.02);

flipBudget left null, the default, compares pixels as before. Given, the mean FLIP error replaces the share of differing pixels. measureQualityTable uses the same measure offline, to price each row of the table adaptive quality chooses from: its cost against full quality, and how different it looks.

Every new test is written by breaking what it covers #

Write the test. Break the code it covers. Watch the test fail. Name the mutation in a comment. Then fix the code.

The same habit applies to golden images: swap one reference for another's and confirm the comparison fails. Five of six lighting goldens in this repository were byte-identical, because the scene's lighting model reached the UI field but never the materials, so every one rendered as PBR and every one passed.

Determinism, and the snapshot that proves it #

No step reads the system clock, and no step takes randomness except from an explicitly passed Random.

test('two runs of one seed agree', () {
  final a = play(seed: 7, inputs: recorded);
  final b = play(seed: 7, inputs: recorded);
  expect(a.save().toJson(), b.save().toJson());

  final c = play(seed: 7, inputs: recorded.withOneExtraPress());
  expect(c.save().toJson(), isNot(a.save().toJson()));
});

That third assertion matters as much as the first two. A determinism test where every run agrees might be testing determinism or might be testing that nothing happens.

GameRandom exists because math.Random has no readable state, which makes it the one thing in a simulation that cannot be written down. The determinism test found a real defect on its first run: monster thinking was staggered across steps by Object.hashCode, which is an address, so two runs of the same seed diverged.

The same tape on a different machine #

Two runs agreeing in one process is the easy half. The half a verifying server needs is two runs agreeing on two machines, because a server that recomputes a submitted run only proves something if its arithmetic is the player's arithmetic.

This page used to say that could not be relied on. It had never been measured. parity_test.dart measures it, by digesting a simulation every so many steps and comparing the digests across platforms.

The middle row is why the first two disagreed. Of twelve dart:math functions asked about twenty thousand arguments each, only sqrt and pow give the same bits in both places; every transcendental differs, and so does every combination of them. Whether a run diverged was therefore a question about which arguments it reached. Walking reaches almost no transcendental. Driving is made of them, and a verifying server cannot rest on a guarantee that holds for one genre and not another.

So a step stopped calling them. Portable answers the seven questions a simulation asks (sin, cos, sinCos, tan, atan, atan2, asin, exp) out of +, -, *, /, sqrt and the bytes of a double, every one of which the specification pins. It gives the same bits everywhere by construction rather than by measurement, which is the part that also covers machines nobody has run it on. The polynomials are fdlibm's; portable_math_test.dart holds them to two units in the last place against dart:math, because portable and wrong is a physics bug that no parity test could ever report, because both platforms would compute the same wrong number and every checkpoint would match. A twenty-ninth structure rule, a step asks no machine for an answer, keeps the call sites there; cameras, a sky, a lamp's brightness and the lightmap baker are exempt by name.

The solver row was predicted and measured anyway: flutter3d_physics calls no transcendental at all, but the three ways it could still have diverged (the order the broadphase hands over contact pairs, a long chain of non-associative additions, and a browser's int being a double underneath the spatial grid) have nothing to do with dart:math.

What this settles: a run submitted to a server can be replayed bit for bit on a different platform, and a mismatch is now evidence of a defect rather than of a browser. It is still localised to an interval and quarantined instead of called cheating, because the first thing to suspect is this repository.

A replay test for your game #

The same tape is also the cheapest regression test a game can have. Play a level once with the recorder on, commit the .f3drun, and testReplay in flutter3d_testing plays it back on the software backend in CI:

import 'package:flutter3d_testing/flutter3d_testing.dart';

// `Ascent` is the platformer demo's own game.
void main() {
  testReplay(
    'test/tapes/ascent.f3drun',
    start: Ascent.open,
    goldensAt: <int>[120, 600],
  );
}

start builds the game against the device and the input it is handed, and returns a ReplaySubject: the EngineLoop the run is stepped in (built on that input, the genre installed and its run set), the genre whose run the tape recorded, frame, and the level's hash. The replay goes through the loop's snapshots, the one path a rollback takes: the tape's start is restored as the genre's part (loop.rewindTo(0, state: genre.loopStateOf(start))), each step is loop.runSteps(1) with the tape as the loop's playback, and at every checkpoint the tape holds (or only those named in digestAt) the digest of the genre's part of loop.capture() has to equal the recorded one, and a mismatch names the step and the last one that still agreed. At each step in goldensAt the frame is compared with test/goldens/<tape>-<step>.png, recorded on the first local run. A level edited since the recording fails as exactly that, before a step is taken, rather than as a divergence at step 25 that sends somebody into the physics. That check found its first stale tape the day it was written: the platformer's sample on this site had been recorded before the level was last changed. apps/flutter3d_demo_platformer/test/replay_test.dart is the whole example, about a hundred lines with the comments.

What can be tested without a device #

Everything except the Impeller goldens. In practice that means:

Test doubles ship with the package they double #

Two packages carry a lib/testing.dart. It is a separate library, so nothing a consumer builds pulls it in, and it is importable, which a test/ directory is not:

// FakeBackend:
import 'package:flutter3d_hardware/testing.dart';
// cpuTestDevice:
import 'package:flutter3d_cpu/testing.dart';

The repository's own hygiene checks, soundTableIn (a sound bank against its source) and creditGaps (credits against the shipped models), were in flutter3d_audio and flutter3d_game until 1.0. They check this repository's demo games rather than anything a game of yours needs, so they moved to package:flutter3d_demo_content/repo_checks.dart, which is not published.

FakeBackend is a GraphicsDevice that draws nothing and records everything: which passes were opened, what they were attached to, what was bound, how many times it drew. cpuTestDevice answers a CpuTestKit: a CpuDevice with the builtin shaders, and the two fallback textures a Renderer asks for.

cpuTestDevice stops short of building the Renderer, deliberately: flutter3d_cpu must not depend on flutter3d. A backend that could not be compiled without the engine would not be an implementation of an interface, it would be part of the engine. That is a rule, and one of the 173 checks it.

Play the game in a test #

The most valuable test in either game is the one that plays it.

import 'package:flutter3d_game_shooter/flutter3d_game_shooter.dart';
import 'package:flutter_test/flutter_test.dart';

test('the crypt can be finished', () {
  final game = loadCrypt();
  for (final (action, steps) in route) {
    for (var i = 0; i < steps; i++) {
      game.input.press(action);
      game.loop.runSteps(1);
      game.input.endStep();
    }
    game.input.release(action);
  }
  expect(game.sim.state, GameState.complete);
});

And draw one frame anyway #

test('a frame renders', () {
  // cpuTestDevice, not `CpuDevice(...)`: the device needs a width, a height and
  // a shader library, and the renderer needs the two fallback textures beside
  // it. That is the fifteen lines this helper replaced in eight files.
  final it = cpuTestDevice(width: 320, height: 180);
  final renderer = Renderer.create(
    device: it.device,
    fallbackAlbedo: it.albedo,
    fallbackNormal: it.normal,
  );
  final frame = renderer.render(width: 320, height: 180, scene: scene,
      views: <CameraView>[view], settings: const RenderSettings());
  expect(frame.drawCalls, greaterThan(0));
});

The rules that hold the architecture are not tests #

They ask how the code is arranged: who imports what, what a name says, where a thing may live. A test asks what code does. They live in tool/structure.dart and run first:

dart run tool/structure.dart

173 rules. All but two read source text in under a second and need no pub get, shader bundle or device, so finding out in minute four that a package imports a genre was finding out late what was knowable in second one. The exceptions are the two snapshots, the API and the tools for agents, which parse every published package with the analyzer (and build the plain Dart MCP servers) and run again once CI has resolved the workspace.

160 more check the lists against the workspace and every example outside the repository against both the workspace and CI, a plain Dart package for a dependency (its own, or a sibling's) that would resolve the Flutter SDK, every pubspec's floors and sibling constraints, a package for a dependency on an application, a simulation package for an import of Flutter, the hardware layer for one of its own, the applications for a silenced print and for the flag that turns the GPU on, the Impeller runners (conformance and the surface probe) for rot, every picture this site shows for a golden that is actually recorded, the publishing order for a package it forgot, the README's package table and the package index for one they leave out, every published description for a length pub.dev scores down, every package for a dependency outside the layers below it, for a re-export of another package's API that the list of facades does not name, each facade's show list for a public name of the library under it that it neither lists nor says why it leaves out, the build hooks that fetch wgpu-native for two copies of that fetch that differ, every library above the hardware layer and the backends for a name of where compute runs, for a dependency held only to re-export it, and for a type its code names from a package its pubspec does not (its build hook for an import its run-time dependencies do not list), every demo, viewer and laboratory for a type declared under a name a package publishes, the runtime (the application, the game packages, the elements, their views and the audio) for a run-time dependency on an editor, the build tool, an agent server or the modeller's core, the simulation stack (the simulation, both physics, the elements, the matter and the foundation) for an import of the core, the hardware layer, the shaders or the particles, the plugin API for a type its contract does not name, a type over its budget or a loop phase named after one package, a public member nothing calls for a sentence saying who it is for, the shader table on the backends page for a stage a bundle must answer to, a vertex stage for a texelFetch that aborts impellerc with no diagnostic, the surface buffer for the depth-in-metres channel four passes read and no upstream filing may tempt anybody to drop, every package's agent skills for a directory the skills CLI would install for nobody without saying so, a package that says it runs on the web for a dart:io, dart:isolate or dart:ffi its browser build would still reach — which is how pub.dev decides, every published package for a platforms: block that SUPPORT.md lists the same way, a library outside the simulation stack for a read of a run's live world rather than what it publishes, every library for a gravity, an air or a sea written as a number rather than read from its world or a substance's preset, every light for an intensity literal small enough to be the pre-1.0 unit rather than lux or candela, every library for a getter that hands out a vector its object holds or a member that keeps the vector it was passed, every particle system an application makes beside a world of its own for being handed that world, every property of a world and group of a material for a library that reads it, every package that promises WCAG 2.2 AA for a colour typed in rather than taken from its tokens, a pinned text size, a control a screen reader cannot name, an animation that never asks about reduced motion and an accessibility test that is missing or asks less than the promise, and five numbers that go stale on their own: the test count, the golden scene count, the structure-rule count, the number of checks the conformance suite says it runs, and the number of enums the hardware layer promises not to rename, each compared against the tree; plus one rule that refuses an enum in a published package unless a table says why it is machinery, and three about the public API: every published package's api/<package>.api is what its source makes, a break since the last release tag is labelled **Breaking:** in the CHANGELOG and released as a major, and every @Deprecated says the version it was deprecated in, the one it goes in, and what to use instead; and four about the shader ABI and the languages: flutter3d_shaders' api/flutter3d_shaders.abi keeps every line of the base recorded from the rc.1 tag and holds every kind of fact it promises, so an entry point, a block member, an offset, a vertex input with its format, a sampler's kind or a varying is never dropped or moved in a minor; each language's vocabulary (the material language's, the kernel language's, the Wasm plugin ABI's) keeps every line of its base, and a word new since carries a version above the base's newest; each of those bases is the tagged commit's files, fact for fact; and every frame resource a plugin may read has a row in CONTRACTS saying what it holds; and two about the tools for agents: every published package's api/<package>.mcp (its MCP servers' tools and input schemas) and api/<package>.vm (its VM service extensions and game events) are what its source makes, and a break in either since the last release tag is labelled, released as a major, and moves the schema version the server announces; and one about the command line: packages/flutter3d_build/api/flutter3d_build.cli is what flutter3d help --surface prints, and a subcommand, a flag, an exit code or a --json key gone since the last release tag is labelled **Breaking:**; and one about the files the engine reads: every format a published package declares with a FormatSpec has a fixture minted at each version its reader opens, every format version constant is declared or listed, and no reader refuses an older file by comparing its version for equality; and one about the kernel language: every word its parser reads as its own says the version it arrived in, so a later version's word stays a name in an older source; and four about the people upgrading: every break since the last release tag has an entry in the migration table that dart fix, the flutter3d_lints migrator and flutter3d_build:migrate carry out, or that says what a person does instead; the table leaves no more to be done by hand than its ceiling; and every count of it a person reads is the table's own; and a table is closed once its release is tagged; and three about what the tools promise outside a signature: the plugin author's server keeps no copy of the plugin suite's checks and builds no badge of its own, no published library reads CI as a compile-time define, and a key flutter3d_assets.yaml did not read at the last release tag comes with a newer format; and three about the shape of the 1.0 contract: a hook an outside author implements takes one context object that can gain members, no published type is an interface class or a mixin class without base unless it is a marker or a value shape that can never grow, and every published exception extends Flutter3dException and none is named …Error; and twelve about the names and shapes in it (docs/CONTRACTS.md, "Names"): identifiers spell in American and so do a tool's arguments, no public type takes a name Flutter, dart:core or vector_math already exports to a program that imports them beside it, no type an outside author implements names a record, a reader throws where it fails and answers null only for absent, no constructor takes optional positional parameters, a type has one teardown verb and creation has its own verbs, no public name carries a unit the engine does not use, a boolean reads as a question and none is positional, no constant is named with k, a settings class is final, const, compared by value and copies every field (and its == and hashCode name every field it holds), and every public number says its unit in its doc, held to a per-package count of the ones that do not yet, which only shrinks; and two about registering: a registration returns at once, never a Future of one, and a plugin registry refuses a name already taken with a NameTakenException; and three about what the engine's readers are handed: a reader of a .fmat or a .f3d names the engine's lighting models, a look-up of a model asks the table it was handed and falls back to the built-in ones only when none is, and a loader of an image names the plugins' texture codecs; and three about names that leave the engine: an anchor, a step or a frame resource carries its namespace (f3d. for the engine's own), the shader ABI spells in American, and a package off the engine's number is named with its reason and every pin written on it admits it. And five about compute: a compute pass is opened by the hardware layer, a backend or the site a piece of compute was placed on; what compute keeps belongs to the engine instance and never to a static; every compute encoder checks its storage ranges and its dispatch sizes; a test's knob, the CPU device standing in for a GPU, stays in testing.dart; and the kernel guide names every operation that carries a tolerance, and the sample plugin written outside the engine computes. And four about the reach of the hardware layer: every feature it gained in 1.0 has a conformance check that compares its output rather than calling it, the suite checks the loss, the second dispose and the sRGB every device owes, a widget drawn into a texture asks the device for the platform texture before it pays the round trip through the host, and WebGL2 asks for and uses the extensions its features rest on. And six about the contracts outside the signatures: a container format's fixtures hold every section kind it writes, the net wire only grows within its protocol major, a platform channel is named under dev.flutter3d/, a call to a platform side catches the method it lacks, a recorded run is played only on the simulation it was recorded on, and a snapshot built from another keeps what it requires; and the fixture rule opens what it counts: a fixture says the version of the directory it is in, and a test names it. And four about the API's hygiene: no public field is a late final without an initialiser, a member advances time by step, update or advance and a double that holds a time says seconds in its name, a singleton is process-wide, named with its reason and says so in its doc, and a device names its backend with a BackendId. And five about the user's types and units: a member named for the world answers a WorldPosition, a mask is 32 bits and no public default is ~0, a gain is linear and a level in decibels says Decibels, a warning reaches a caller through an IssueSink, and a member documented as the engine's own is not on a public type. And one about the doc comments: every tree in the workspace, the apps, demos, examples and tools as much as the packages, is held to public_member_api_docs as an error, in its options and in every file they include. A number in prose is a number nobody recounts, so the counting rules read this site's pages too, and the sentence you are reading is one of them: the rule counts the table above and requires the rest to be the rest.

The detectors prove themselves first #

Rule 6.3 applies to these more than to anything else: a scan behind a detector nobody has seen fail is a rule nobody is keeping. The genre scan once lived in two copies and one of them had already lost the check that it fired at all, so it passed whether or not the rule held.

So the detectors run against synthetic input before a single file is read. They must fire on int ammo = 0; and stay quiet on overlaps, elapsed and collapse, all three of which contain lap. A broken detector stops the run and no scan is reported, because a green scan behind one is worse than a red one: somebody believes it.

Coverage that moved with its fixtures #

Four suites covering the game layer's own machinery (the level format, the validator, navigation, and buttons and trigger volumes) now run from the shooter package. Their fixtures are written in that game's vocabulary (monster, key, torch), so they moved rather than being rewritten blind.

The coverage still runs on every CI. What was lost is locality: content-free fixtures for those four have not been written.

Frame capture as a measurement #

flutter run -d macos \
  --dart-define=FLUTTER3D_CAPTURE=shot.png \
  --dart-define=FLUTTER3D_CAPTURE_FRAME=200 \
  --dart-define=FLUTTER3D_SPIN=false \
  --dart-define=FLUTTER3D_ANIM_TIME=0.75

FLUTTER3D_SPIN=false stops the turntable so two captures differ only by what is being tested, and FLUTTER3D_ANIM_TIME freezes a clip at a given second. Without it two captures of an animated model are not comparable, because a format that loads faster starts playing sooner.

Frame pacing on a real device #

A run that averages sixty frames a second can still stop for a tenth of a second every few seconds, and that stop is what a player feels. So the measure is every frame's own time, and the line is 50 ms, three frames at 60 Hz.

tool/pacing.sh                                  # macOS through Impeller, the sample run
tool/pacing.sh -d <device-id> --label a55       # a phone
tool/pacing.sh --run path/to/run.f3drun --repeats 5 --budget 2000
tool/pacing.sh --demo racing -d <device-id> --warm-up 60 --seconds 120 \
  --fail-on p50=16.7,p99=33.3,over50=0.1,max=100,rssGrowth=1.10,rssEnd=400

--demo picks the game: shooter (the dungeon, the default), platformer or racing, each played by its own application's integration_test/pacing_test.dart from its own tape in site/assets/samples/. The report of the platformer and the racing is labelled <device>-<demo> unless --label says otherwise. --seconds and --warm-up size the run in seconds of play: the tape is looped in as many whole passes as cover them, and the warm-up's passes are played and not counted. Without --seconds the run is --repeats passes.

--fail-on holds the run to thresholds in place of "no frame over 50 ms". p50, p99 and max are in milliseconds, over50 is the share of frames over the line in per cent, rssGrowth is the resident memory at the end over the memory 30 s into the run, and rssEnd is the resident memory at the end in MB. A value on the line passes. The memory is read once a second while the run plays, through dumpsys meminfo on Android and ps on macOS; on any other device it is not read, and a memory threshold there fails rather than passing unread.

The script plays a recorded run through the renderer in apps/flutter3d_demo_dungeon with flutter drive --profile, because flutter test on a device builds debug and a JIT frame time is not a frame time. It writes the report to doc/pacing/<label>.md (the device's id when there is no --label; --no-report keeps it out of the tree) and exits non-zero on any frame over the line. --repeats plays the tape that many times, three by default. --budget sets RenderSettings.frameWorkBudget, given in microseconds, and 0, the default, leaves it unlimited. The report keeps the median, the 99th percentile, the worst frame and where on the tape it fell, so a spike can be found again by replaying to it; the macOS report is committed, and the same command runs on the release phone by hand before a release.

Underneath is replayPacing in flutter3d_testing. A frame is one step of the tape through onStep and one drawFrame, back to back and not on a display's clock, and a GPU device's drawFrame finishes with gpuSettled, which closes the frame and waits for the device's onFrameComplete so the GPU's share is counted. The result is a PacingReport: median, 99th percentile, the worst frame and its index, and every frame over limitMillis (50 by default). repeats, rewind and warmUpFrames play a tape several times from its start, or skip a level's first frames.