Assembling an application

The tutorial opens a device and drives a Ticker by hand, because that is what is actually happening underneath. By the second game, the same conditional import, the same forty-line frame widget and the same load-restart-save sequence had been written out three times, close enough to identical that a bug fixed in one copy stayed broken in the other two. Two packages exist because of that: flutter3d_app (the backend choice, the frame surface and level loading, which any application needs) and flutter3d_game (the input, the run, and the settings, rebinding and save screens no game owns). This page is what they do, and how apps/flutter3d_demo_dungeon puts them to use. It is the most complete worked example of both together.

Picking a device #

import 'package:flutter3d_app/flutter3d_app.dart';

final device = await openDevice(width: 1280, height: 720);

openDevice picks between backends two different ways. Web or native is a conditional export, resolved at compile time, because flutter_gpu does not compile for the web and dart:js_interop does not compile for macOS. A file importing both could target neither. On the native half, Impeller or software is a runtime try/catch: flutter3d_impeller's GpuRenderBackend.open(width:, height:) is tried first, and if it throws, openDevice falls back to flutter3d_cpu's CpuDevice, built from CpuShaderLibrary(builtinCpuShaders()). It throws when Flutter GPU refuses to start on Skia, or when the platform never enabled Impeller. It has to be a runtime check: flutter_gpu ships with the SDK and is always importable, so no dart.library.* condition can see whether it will actually start. On the web half it is flutter3d_webgl's WebGlDevice.open(width:, height:), with no software fallback: a software rasteriser would not make a browser without WebGL2 any more able to draw the picture. Every backend opens the same way, await X.open(width:, height:); Impeller only checks the size, since it sizes itself per frame, and WebGL and the software fallback, which cannot, use it.

What it does not decide is the size of a frame. A Flutter3dView draws at the layout's size times the pixel ratio by default, and on every backend the picture follows a resize: Impeller allocates its targets at that size, and on the web the canvas, WebGL2's and WebGPU's alike, takes the size of each frame it is handed, so a 2560×1440 window shows a 2560×1440 frame and not a stretched 720p one. DeviceSettings.canvasSize is where the application says otherwise: CanvasSize.layout(maxPixels: …) caps the frame, and CanvasSize.fixed(960, 540) holds one size whatever the layout, stretched to it by the presenter's BoxFit. fixedResolution is the hint the games read for that choice: true on the web, where fill rate and memory are a tab's, false natively. The crypt and the platformer draw at 720p there, the racing game at 960×540, and each says why where the number is.

A device on the web is one canvas, and a canvas is one element of the page, so it shows in one widget: a second live Flutter3dView borrowing a device another one shows is refused with a DeviceUnavailableException saying so, where it would otherwise take the canvas from the first and leave it blank. Two views on a web page each open a device of their own, and a page holds only as many live WebGL or WebGPU contexts as the browser allows (Chrome keeps sixteen and drops the oldest).

On the web a device shows in one widget. The web presenters blit into the device's own canvas and hand Flutter one HtmlElementView of it, so a page with three views opens three devices, each with its GPU context, and the browser caps how many live contexts a page may hold (Chrome drops the oldest past sixteen). A page that wants several pictures of one scene draws them with one device for now, as RenderTextures shown inside one view. Flutter3dScope, a scope that shares a device, a renderer, assets and one ticker across views (and with them many live views on one web page), comes in rc.3; until then each Flutter3dView on the web is a context of its own.

Natively, two live views may share a renderer and a scene. A floor plan beside the perspective, a front view beside a side view: hand both views the same Scene and a camera each, and they may borrow one device and one Renderer as well. Each view draws through its own CameraView, and the frame the renderer answers belongs to that view until its next frame, so neither shows the other's picture:

// One building, seen from above and from where a person stands.
final plan = CameraNode(
  projection: const OrthographicProjection(height: 30.0),
)
  ..setPosition(0.0, 40.0, 0.0)
  ..setRotationYawPitchRoll(0.0, -math.pi / 2, 0.0);
final eye = CameraNode()
  ..setPosition(6.0, 1.7, 9.0)
  ..lookAt(Vector3(0.0, 1.0, 0.0));

final views = Row(
  children: <Widget>[
    for (final camera in <CameraNode>[plan, eye])
      Expanded(
        child: Flutter3dView(
          scene: building,
          camera: camera,
          device: DeviceSettings(device: device, renderer: renderer),
        ),
      ),
  ],
);

Views of the same size share a renderer well. Views of different sizes make one renderer resize its targets on every frame, so give each its own Renderer.create(device: device) over the shared device. packages/flutter3d_app/test/view_sharing_test.dart holds both cases, and the refusal on the web.

On the web the 3D is a platform view, and it behaves as one. The canvas sits in the page beside Flutter's own, not inside Flutter's picture, which has four consequences an application meets. A Flutter widget drawn above the view (a menu, a tooltip, a dialog's scrim) makes the engine split its picture around the platform view, and each layer above one costs the page a further canvas, so keep overlays few and flat. RepaintBoundary.toImage does not see the 3D: a screenshot of the page has a hole where the view is, and a still of the scene comes from the renderer (capturePhoto for a file, capturePhotoImage for a ui.Image, both described here). A Transform above the view is applied to the canvas as a CSS transform, so rotation and scale work, at the cost of the browser resampling the picture. And a click on the view gives it the keyboard focus, so keys stop reaching Flutter's widgets until something else is clicked, which strands a keyboard-only user; the demo asks for a click on the frame for the same reason.

The frame surface #

SceneSurface is the widget every game's build method ends with:

SceneSurface(
  renderer: renderer,
  scene: loaded.scene,
  view: _view,
  onBeforeDraw: _placeCamera,
  settings: () => RenderSettings(
    reflections: const ReflectionSettings(),
    fog: fogFromLevel,
  ),
  presentFrame: presentFrame,
)

from apps/flutter3d_demo_dungeon/lib/main.dart

renderer, scene and view are unsurprising, and presentFrame is the same backend choice openDevice made, handing the finished frame to Flutter. settings is not a RenderSettings value. It is a function called once per frame, immediately after onBeforeFrame. That ordering is the whole reason it's a function: the racing game's fog colour is the colour of the air along the current view, which only exists once the camera has moved for this frame, and onBeforeFrame is where the camera moves. A stored RenderSettings would describe last frame's air.

SceneSurface clamps the render target to at least one pixel in each dimension, because a panel mid-animation or a window dragged to nothing is a real, zero-sized layout, and a render target of zero pixels is not a state the renderer has to tolerate.

In the background, and on a battery #

SceneSurface leaves the clock to its caller; Flutter3dView and Scene3D keep one, and stop it on their own. While the application is not in the foreground (any AppLifecycleState other than resumed), or the view is not shown (a route pushed over it, an offstage subtree, anything that turns TickerMode off), nothing is drawn and the loop does not step. The time away is handed to nobody, so the simulation does not lurch forward by the minutes the phone spent in a pocket. onPausedChanged is told both ways, which is where an audio engine is paused, since this package does not depend on one.

A frame budget is ViewSettings.frameRateCap. A cap is a whole number of display refreshes, so 30 on a 60 Hz screen is every other refresh and 30 on a 120 Hz screen every fourth, each frame on screen for the same time. A cap that skipped refreshes at random would judder at its own rate. The loop's fixed step does not change with it; a drawn frame brings all the time since the last one.

Flutter3dView(
  // `onBattery` is the application's: Flutter itself does not say.
  view: ViewSettings(frameRateCap: onBattery ? 30.0 : null),
  onPausedChanged: (bool paused) => paused ? music.pause() : music.resume(),
)

A scene that only changes when somebody touches it does better with FrameDemand.onRequest (see the quickstart), which draws nothing at all between changes. The two combine: a cap holds what keepDrawing draws while a model turns.

The run #

RunSession<L> is loading a level, restarting it, moving to the next one, saving, resuming, and reporting how a run ended. Three games wrote that sequence out separately: a 278-line cubit in the dungeon, nine private methods in a 1441-line platformer widget, and nothing at all in the racing game, whose saves never worked. A game answers five questions and gets the sequence for free:

final class DungeonRun extends RunSession<LevelReady> {
  DungeonRun({required super.firstLevel, required super.saves, /* ... */});

  @override
  Future<LevelReady> loadLevel(String asset) async {
    final loaded = await const LevelLoader().load(asset, device: device, registry: registry, rules: sampleRules());
    // ... build the scene and collision visuals, stage the level
    return LevelReady(/* ... */);
  }

  @override
  RunOutcome outcomeOf(LevelReady level) => level.staged.sim.state.outcome;

  @override
  String? nextOf(LevelReady level) => level.staged.sim.nextLevel;

  @override
  Snapshot snapshotOf(LevelReady level) => level.staged.sim.save();

  @override
  void restoreInto(RunRestoreContext<LevelReady> context) =>
      context.level.staged.sim.restore(context.snapshot);

  // Keys do not carry between levels. A level designer's promise, and the
  // plumbing must not break it.
  @override
  void carryFrom(RunCarryContext<LevelReady> context) => inventory.keys.clear();
}

abridged from apps/flutter3d_demo_dungeon/lib/src/run_cubit.dart

RunSession is an ordinary class, not a Cubit. Two of the three games wrap it in one, calling onChanged from emit, and the package neither knows nor requires that. Driving it is five methods: begin() (the saved run, or firstLevel, falling back to a fresh start if the save names a level that no longer exists), load(asset, {resume}), restart(), advance() (moves on once outcomeOf reports the level is over, and does nothing on the second call, since a finished level would otherwise restart a load on every subsequent frame), and save().

Not every game fits the shape. The racing game moves from one circuit to the next and remembers where somebody got to, which looks like a RunSession and isn't one: nobody resumes a race half a lap in, so snapshotOf/restoreInto would be two required overrides returning nothing. It uses SceneSurface without RunSession, and test/one_assembly_test.dart, the check that exactly one function per game turns a level document into a run, reads all three applications instead of assuming the base class applies to all of them.

Settings and saves #

The screens a game has that are not the game are flutter3d_game_ui's settings.dart: SettingsOverlay, a panel of SettingsSections (volumes, gamepad and accessibility sliders, a rebinding list that takes a key or a pad button, and where a licence's attribution goes). What they show and write stays in flutter3d_game: GameSettings, its controller, and SaveFile (where RunSession.saves reads and writes, per platform, through Storage). Wiring the overlay into a HUD is one widget per game:

import 'package:flutter3d_game_ui/settings.dart';

SettingsOverlay(
  settings: gameSettings,          // a GameSettingsController, a ValueListenable
  sections: SettingsSection.standard(
    padConnected: pad.isConnected, // asked every frame, not assumed
    defaultActions: shooterActions, // what "reset" goes back to
  ),
  opening: pauseTheGame,
)

GameSettings is a value: a volume per AudioBus, typed SettingKeys under namespaced ids, and the player's action map, each change a copy the controller saves in the format envelope. The accessibility sliders are its keys (GameSettingKeys.cameraMotion and its kin), and Accommodations supplies their defaults from the platform's own reduce-motion flag, so a slider the player moved wins over the flag. The widgets read the flag themselves: while MediaQuery.disableAnimations is set, a Model3D's clip and a Particles3D hold still, and respectReduceMotion: false on either plays motion that carries meaning, a gauge's needle or a pump's flow.

The engine's words #

The few sentences the engine shows itself (Loading…, the screen when the renderer did not start or a level would not load, a panel's Close) come from Flutter3dAppLocalizations, in English and Russian. Another language is a subclass that overrides what it translates; every member has an English body, so a member added in a minor reads in English until the subclass catches up. delegateFor lists the application's tables beside the engine's own, the app's winning for a language both have:

import 'package:flutter/material.dart';
import 'package:flutter_localizations/flutter_localizations.dart';

final class GermanFlutter3d extends Flutter3dAppLocalizations {
  const GermanFlutter3d();

  @override
  String get languageCode => 'de';
  @override
  String get loading => 'Wird geladen…';
}

final app = MaterialApp(
  localizationsDelegates: <LocalizationsDelegate<Object>>[
    Flutter3dAppLocalizations.delegateFor(
      const <String, Flutter3dAppLocalizations>{'de': GermanFlutter3d()},
    ),
    ...GlobalMaterialLocalizations.delegates,
  ],
  supportedLocales: const <Locale>[Locale('en'), Locale('ru'), Locale('de')],
  home: const SizedBox(),
);

Without a delegate the widgets speak English, so a test or an app that never thought about languages still reads; a widget that takes a string parameter still says the caller's words.

It was extracted when the second game wanted rebinding, which mattered for accessibility more than for convenience: the alternative was four hundred lines of panel copied into the second game, and the racing game's copy had already drifted: it told its own panel there was no controller connected while the other two asked the pad.

Accessibility #

To a screen reader a 3D view is one rectangle of pixels. Flutter3dView and Scene3D publish no semantics of their own in rc.2, so VoiceOver or TalkBack arriving at one finds nothing to read, and a person exploring by touch finds no target. Describing the scene from its own nodes, as Scene3D(semantics: ...) will, comes in rc.3. What works today is the following, and an application that uses it is not waiting on anything.

A label for the view. Wrap the view in Flutter's own Semantics, saying what it shows and what a gesture does there, the way a chart or an image is labelled. That is the alt text, and it is most of what a product picture or a decorative scene needs.

The objects that matter, where they are on the glass. SceneSemantics from flutter3d_app lays one semantics node over each object you name, at the rectangle the object projects to, so a screen reader can list the parts, read their names, put its focus ring around each and activate it. semanticObjectsFor does the projection from a camera and the view's size; the label is yours, a name such as "left wheel", not "3D object":

import 'package:flutter/semantics.dart' show SemanticsService;

Widget described(BuildContext context, Scene scene, CameraNode camera,
        SceneNode crank, SceneNode pump) =>
    Semantics(
      container: true,
      label: 'Engine block. Drag to turn it.',
      child: LayoutBuilder(
        builder: (BuildContext context, BoxConstraints box) => SceneSemantics(
          objects: semanticObjectsFor(
            <SceneAnnouncement>[
              SceneAnnouncement(
                id: 'crank',
                label: 'Crankshaft',
                node: crank,
                // What a tap on the part does, said back to the reader.
                onTap: () => SemanticsService.sendAnnouncement(
                  View.of(context),
                  'Crankshaft selected',
                  TextDirection.ltr,
                ),
              ),
              SceneAnnouncement(id: 'pump', label: 'Oil pump', node: pump),
            ],
            camera: camera,
            size: box.biggest,
          ),
          child: Flutter3dView(scene: scene, camera: camera),
        ),
      ),
    );

The rectangles are worked out when this builds, so build it again when the camera comes to rest after a drag; a focus ring that chases a turning model helps nobody. It needs a camera you hold, which is why the example is a Flutter3dView. Hand it the few objects a person would ask about, not every mesh: a reader handed a thousand nodes is a reader nobody can use. The modeler's viewport (apps/flutter3d_modeler/lib/src/modeler_viewport.dart) announces its objects this way.

Say what happened. A tap that selects a part, a measurement that finished, a simulation that reached its end: SemanticsService.sendAnnouncement reads a sentence in the reader's own voice, and goes nowhere when no reader is on.

Motion and flashes. While the platform asks for less motion (MediaQuery.disableAnimations), a Model3D's clip and a Particles3D hold still, as described above; respectReduceMotion: false is for motion that carries meaning. A camera you move yourself, a fly-to or an auto-rotate, should check the same flag. A full-screen change of brightness (a fade to white, a hit, lightning) can go through FlashLimit from flutter3d_game_ui/access.dart, which holds it to three flashes in any second (WCAG 2.3.1). It is a plain class with one method, so an application that is not a game uses it the same way.

A path without the pointer. OrbitCamera3D answers a drag, a pinch and the wheel, not the keyboard, and a click on a view on the web takes the keyboard focus from Flutter's widgets (see above). Give every camera move that matters (reset, the next part, zoom) a Flutter button beside the view; a button already has semantics and a keyboard path.

Games. flutter3d_game_ui's access.dart goes further for a game: SpokenEvents says the game's events aloud through the same announcements, RoleRings and RolesApart keep colours apart for the player's colour vision, and the settings screens carry the accessibility sliders. Its README has the details.

Input #

A game rarely calls pad_input or flutter3d_pointer_lock directly. flutter3d_game's DesktopInput and pad-action layer already wrap both, and that is what _devices.isCaptured and _devices.captureMouse() in an app's main.dart reach. The two packages underneath matter because they're where a fifth platform gets implemented:

import 'package:pad_input/pad_input.dart';

final pad = Gamepad.instance;
final snapshot = PadSnapshot();
pad.read(snapshot);                                  // once per frame
if (snapshot.down(PadButton.faceSouth)) jump();

Gamepad is pull, not push. A fixed-step simulation asks what is the pad doing now exactly once per step, and a stream would push edge detection into every caller differently. Button identifiers are physical positions (face.south, not a), because the string is what lands in a player's config file, read back on a possibly different pad. Implemented on macOS, iOS, Web and Android; Windows and Linux are not yet.

import 'package:flutter3d_pointer_lock/flutter3d_pointer_lock.dart';

final lock = PointerLock.instance;
if (lock.isSupported) await lock.capture();
final delta = lock.drainDelta();                       // synchronous, drained once per step

PointerLock fills a gap Flutter leaves on every desktop platform: no pointer lock, which an FPS-style camera needs, since without it the cursor reaches the window edge and the view stops turning. drainDelta() is synchronous on purpose. It is called from inside the simulation step, where awaiting anything would mean the step no longer sees a consistent snapshot of its inputs. Two kinds of backend: a method channel to native code on macOS, Windows, Linux, Android and iPadOS, and the browser's own requestPointerLock through static interop. The choice is a conditional export, so nothing registers a plugin on the web and flutter test --platform chrome can exercise it. A phone or a tablet captures only while a mouse is connected and says so on supportChanges; without one isSupported is false, so a game falls back to touch controls instead of to a camera that does not turn. Losing focus drops the capture automatically, which a game should treat as a reason to pause rather than as an error; in a browser the capture also has to be asked for inside the handler of the press that prompted it, because requestPointerLock is refused without a user gesture behind it.

A view that runs for weeks #

A kiosk, a control-room wall, a twin on a screen in a plant: the view opens once and is expected to stay up. The longest run we measure is tool/soak.sh, two hours of the platformer's tape on a device with its resident memory read once a second, and nothing has been run for weeks yet. What the engine gives such an application is the following.

Give back what you took. Every add, register and on… that returns a Registration is undone with cancel(), and a listener left behind keeps its object alive for the life of the process. A model the screen no longer shows is released through its ResourceCache handle, and evictUnused() frees what nobody holds (the resource cache). A ResourceCache given a MemoryBudget does that on its own: past a limit it evicts what nobody holds, least wanted first, and onCrossed tells you once when a limit is crossed and once when it is back under, which is the moment to lower texture quality or stop streaming.

Know what the view holds. Renderer.memoryReport is an account of what the renderer holds on the device, by category: textures, buffers, render targets, meshes, and shaders, counted but not weighed. Pass the scene to add what it draws. It counts the bytes the data needs, the floor under every driver's padding, so it is the number to set a budget against rather than a measurement of the process. It reads nothing back from the device, and is cheap enough to call every second:

Flutter3dView(
  onFrame: (EngineFrame frame) {
    final report = frame.engine.renderer.memoryReport(scene: frame.engine.scene);
    if (report.totalBytes > 600 * 1024 * 1024) dropToLowerQuality();
    debugPrint('textures ${report.bytesOf(MemoryCategory.textures)} B');
  },
)

The report is per renderer. Views that share one renderer report together, and a view with its own renderer reports alone, so a page of several views that needs a figure per view gives each one a renderer of its own. The render-target pool keeps one texture of every shape a frame has ever needed; MemoryPressureRelease(renderer: renderer, child: ...) gives the spare ones back when the platform warns about memory, and Renderer.releaseTransientTargets() does it on your own signal.

A lost device comes back. A WebGL context lost, a WebGPU device lost, a GPU reset: DeviceSettings.onDeviceLost is told first, nothing is drawn while the loop goes on stepping, and onDeviceRestored is told when the view draws again on the restored device or a new one. What the application uploaded went with the old device, so onDeviceRestored is where meshes and textures are uploaded again, on engine.device. A loss the view cannot recover from shows its errorBuilder.

XR #

A view can be driven by an XR session instead of the display. Flutter3dView.xr (and Scene3D(xr:)) takes XrSettings: the XrSession, a runtime's side of the contract, and an XrSessionRequest for the mode, the reference space and the capabilities required and wanted. Once a session starts, the runtime paces the frames, its poses enter the loop's input as external state (so a recorded session replays), the eyes are placed under the camera by the runtime's poses and angles, and the screen shows them side by side.

rc.2 ships the contract and one runtime. HolderXrSession in flutter3d_stereo is a phone in a holder, or held up in the hands, as an inline session. The head comes from a HeadTracker (Android's rotation sensor through SensorHeadTracker), and each eye's frustum from the holder's lenses:

import 'package:flutter3d_stereo/flutter3d_stereo.dart';

final room = Scene3D(
  xr: XrSettings(
    session: HolderXrSession(
      tracker: SensorHeadTracker(),
      viewer: StereoViewer.cardboardV2,
    ),
    request: const XrSessionRequest(mode: XrCapability.inline),
  ),
  children: const <Widget>[Light3D.sun()],
);

A holder has no immersive mode, no anchors and no swapchain, and refuses each with an UnsupportedCapabilityException that names it, as every runtime refuses what it lacks. package:flutter3d_app/xr_testing.dart has FakeXrSession for tests.

What the contract names for runtimes that come later. XrSession.placeAnchor fixes a point to the real world. An XrAnchor keeps its id across sessions and is kept in a level as an entity, and an outdoor one carries a GeoOrigin, so a bench placed on a real table, or a marker placed at a real address, is there when the level opens again. XrCapability.lightEstimation brings the room's light in XrFrame.light, and planes, hands, depthOcclusion and passthrough are named too. None of them has a runtime behind it in rc.2:

• WebXR (immersive sessions in a browser, hit tests against the floor) comes in rc.3, and so does launching a model into the platform's own viewer (Quick Look on iOS, Scene Viewer on Android).
• AR on a phone through ARKit and ARCore (planes, face and body tracking, the camera's intrinsics in the frame) is 1.x.
• Showing a configured variant of a model in AR needs ModelInstance.toDocument, which comes in rc.3.

What 1.0 is lists the rest of what comes after rc.2.

One import for all of it #

The packages this page walks are re-exported by flutter3d_app, so an application names the layer once rather than naming its parts:

import 'package:flutter3d_app/flutter3d_app.dart';

Thirty-five lines of export, plus the backend choice itself, which lives in this package directly rather than behind one more export line. It changes nothing about how the four sibling packages relate: none of them knows about the others, and the barrel does not make them. What it saves is import lines being copied into every new project, which is how this page's subject started.

What is deliberately not behind it: flutter3d, flutter3d_sim, flutter3d_game and a genre package. Those say what a scene looks like and what kind of game this is, and a facade cannot choose a genre for an application.

Where one application has not caught up #

The four demo games import flutter3d_app, and so does packages/flutter3d_game/example, the game a new project starts as: the scaffold names flutter3d_app in its pubspec and opens its device through openDevice, which picks Impeller or WebGL for the build and falls back to the software rasteriser at run time when flutter_gpu will not start. That is the pattern this page teaches, fallback included. It also names flutter3d_game, for the input and the run.

apps/flutter3d_editor is the one that does not: it names flutter3d_impeller and opens its device with GpuRenderBackend.open(width:, height:), the way the tutorial does. That is defensible where it is, since the editor is desktop-only and there is no backend to choose between, but it is the reason the editor is not the thing to copy. Copy the scaffold, which is what it is for.

Next #

• Core: what core is: the packages underneath all of this
• Tutorial: first scene: the raw version of device-opening this page builds on
• The level editor: the fourth application, and the one with no genre package at all