flutter3d
Showcase Changelog 62 packages API reference

Inspecting a running frame

since 1.0.0-rc.1 Widgets and the rest

When a picture is wrong, the question is usually which pass drew it, which draw put that pixel there, and with what. A game built on flutter3d_app answers those questions about its own frame over the VM service, so a tool attached to it, a person's or an agent's, reads the frame the game really drew on its own GPU rather than a second picture drawn somewhere else.

The extensions are ext.flutter3d.render.passes, draws, draw, passOutput, readPixel, scanNan, stats and pick. Each one is a thin wrapper over a plain function of one captured frame: renderPasses, renderDraws, renderDraw, renderReadPixel, renderStats, renderPicked. This page calls the functions directly, which is also how their tests call them.

Step 1: Ask for the next frame, with its draws and a pick #

captureNextFrame(draws: true) records the next frame pass by pass, with the pixels each pass wrote and a journal of every draw: which node, which material, how many triangles, the state and the uniforms bound for it. Without draws: true the draws are counted but not described. pickPixel asks which mesh the same frame draws at a point. Both are asked before the frame and answered by it, so a draw index and a picked node belong to one frame.

/// Asks the renderer for the next frame twice over: its passes and its
/// draws, journaled, and the node drawn at ([u], [v]), fractions of the
/// frame from the top left. Both are answered by the same frame, so a draw
/// index means the frame the pixel is from.
static Future<_Look> _lookAt(Renderer renderer, double u, double v) async {
  final Future<FrameCapture> captured = renderer.captureNextFrame(
    draws: true,
  );
  final Future<MeshNode?> node = renderer.pickPixel(u, v);
  final FrameCapture capture = await captured;
  return _Look(
    capture,
    await node,
    (u * capture.width).floor(),
    (v * capture.height).floor(),
  );
}

Step 2: Draw a frame to be asked about #

The page draws its scene once through a small camera of its own before it is shown, and keeps what came back. The viewport then asks again about its own frame once a second; the line beside the viewport is that answer. Click it to ask now.

final CameraNode eye =
    CameraNode(projection: const PerspectiveProjection(fieldOfView: 0.8))
      ..setPosition(0.0, 0.8, 4.0)
      ..lookAt(Vector3.zero());
_scene.add(eye);
// The middle of the pixel in the middle, so the pick and the pixel read
// are the same one.
final Future<_Look> look = _lookAt(
  context.renderer,
  (_probeWidth ~/ 2 + 0.5) / _probeWidth,
  (_probeHeight ~/ 2 + 0.5) / _probeHeight,
);
context.renderer.render(
  width: _probeWidth,
  height: _probeHeight,
  scene: _scene,
  views: <CameraView>[CameraView(camera: eye)],
);
_probe = await look;
eye.removeFromParent();

Step 3: Read it the way a tool does #

Each function answers a JSON map, the same one the extension sends. The passes come in the order they ran, with their inputs, outputs, time and draw count. renderPicked names the node under the point and the indices of the draws it made, and renderStats adds the frame up.

/// What an agent reads back: the passes in order with their draw counts,
/// and the node under the middle with the draws it made.
static String _report(_Look look) {
  final List<Object?> passes =
      renderPasses(look.capture)['passes']! as List<Object?>;
  final String ran = <String>[
    for (final Object? pass in passes)
      if (pass case {'name': final String name, 'drawCalls': final int n})
        '$name ($n)',
  ].join(', ');
  final Map<String, Object?> picked = renderPicked(
    look.capture,
    look.node,
    x: look.x,
    y: look.y,
  );
  final Map<String, Object?> stats = renderStats(look.capture);
  return 'passes: $ran. Under the middle: ${picked['node'] ?? 'nothing'}, '
      'draws ${picked['draws']}. ${stats['drawCalls']} draws, '
      '${stats['triangles']} triangles in all';
}

Step 4: What the page checks #

The middle of the probe frame has to be the crate. Its draw, opened by index as a tool opens one, has to name the crate, and the output of the pass that draw ran in, read at the same pixel, has to be orange rather than the barrel's blue. The journal has to describe all three meshes.

final FrameCapture capture = _probe.capture;
final Map<String, Object?> picked = renderPicked(
  capture,
  _probe.node,
  x: _probe.x,
  y: _probe.y,
);
if (picked['node'] != 'crate') {
  throw StateError('the middle of the frame is ${picked['node']}');
}
final List<int> draws = (picked['draws']! as List<Object?>).cast<int>();
if (draws.isEmpty) throw StateError('the crate made no draw that frame');
// The draw it made, opened as a tool opens one, names the crate and the
// pass it ran in; that pass's output, read at the same pixel, is the
// crate's orange and not the barrel's blue.
final Map<String, Object?> draw = renderDraw(capture, <String, String>{
  'index': '${draws.first}',
});
if (isRefusal(draw) || draw['mesh'] != 'crate') {
  throw StateError('draw ${draws.first} is not the crate: $draw');
}
final Map<String, Object?> pixel = renderReadPixel(
  capture,
  <String, String>{
    'pass': '${draw['pass']}',
    'x': '${_probe.x}',
    'y': '${_probe.y}',
  },
);
final List<Object?>? value =
    (pixel['float'] ?? pixel['unorm']) as List<Object?>?;
if (isRefusal(pixel) || value == null) {
  throw StateError('the pixel could not be read: $pixel');
}
final double red = (value[0]! as num).toDouble();
final double blue = (value[2]! as num).toDouble();
if (red <= blue) {
  throw StateError('the pixel the crate drew reads $value');
}
if ((renderStats(capture)['describedDraws']! as int) < 3) {
  throw StateError('the journal missed some of the three meshes');
}

A question these functions cannot answer comes back as {"refused": "..."} with the reason and what would work, rather than as an exception: a draw index past the end, a pixel outside the target, a capture taken without the journal.

Note. scanNan reads a float target's values only on the software backend. On the GPU backends a float target is read back as eight bits, where a NaN has already become an ordinary byte, so it is reported as unread rather than as clean. A full-screen pass's draws are counted per pass as undetailed and are not described one by one.