Assets & animation

Four decoders, one abstraction, one upload path. Decoding runs on a background isolate; nothing in the decoding layer names flutter_gpu, dart:io or dart:ui.

One abstraction, four formats #

Every one of them (the two text formats, the binary container, and STL) emits the same ModelDocument: a list of ModelSurface, a list of SurfaceMaterial, a list of EncodedImage, and warnings.

flowchart LR
  gltf[".gltf / .glb"] --> doc
  obj[".obj + .mtl"] --> doc
  stl[".stl"] --> doc
  f3d[".f3d"] --> doc
  doc["ModelDocument<br><i>surfaces · materials · images · warnings</i>"]
  doc --> asset["ModelAsset.fromDocument<br><i>mesh + image dedup, material conversion</i>"]
  asset --> instance["asset.instantiate(scene)"]
  instance --> nodes["SceneNode tree · MeshNodes<br>Skeletons · AnimationPlayer"]

That is what makes ModelAsset.fromDocument the single upload path, and it means adding a fifth format is a matter of writing a decoder rather than touching the loader.

A format of your own #

A decoder extends ModelDecoder: handles says whether a file is its own, from a ModelProbe of the file's name and bytes, and decode turns a ModelDecodeContext into a ModelDocument. A load's ModelLoadRequest(decoders: ...) tries the application's decoders before the built-in ones, so one can add a format or replace the engine's reader of one. They travel to the decode isolate with the request, so a decoder must be sendable: no open sockets or closures over the UI's state in its fields.

A big file can be read in ranges. A pager through splats, a point cloud's octree or a tiled terrain wants the header first and the rest as it is drawn, not the whole file in memory before it starts. A decoder whose readsRanges is true is chosen by the file's name alone and handed its context before anything is read: context.read(range) is a ranged read of the source, through the load's own cancel signal, and context.progress(fraction) is what the load's progress listener hears.

import 'package:flutter3d_core/formats.dart';

final class PointPages extends ModelDecoder {
  const PointPages();

  @override
  bool get readsRanges => true;

  @override
  bool handles(ModelProbe probe) => probe.fileName.endsWith('.pages');

  @override
  Future<ModelDocument> decode(ModelDecodeContext context) async {
    final header = await context.read(const AssetByteRange(0, end: 16));
    context.progress(0.1);
    // ... read the node table and the root's pages from what the header says
    return PlainModelDocument(warnings: <String>['header ${header.length} bytes']);
  }
}

A ranged decoder runs where the load was called, not on a background isolate, because its reads and the listener live there; moving its own heavy work off that isolate is the decoder's to do. Any other decoder is handed the whole file as before, and may call read too, which then cuts the bytes it already has, so a decoder written for ranges reads the same either way.

Loading a model #

final document = await decodeModelInIsolate(
  ModelLoadRequest(source: const BundleAssetSource('assets/models/hero.glb')),
);

final asset = await ModelAsset.fromDocument(
  document,
  device: device,
  name: 'hero',
);

final instance = asset.instantiate(scene, name: 'runner');
final player = instance.player;          // null when the file has no clips

Bytes you already hold #

A model the app downloaded with its own HTTP client, a file the user picked, a GLB a server generated for this order: in each case the bytes are already in a Uint8List, and the engine only needs to be told where to find them. AssetSource is open for this. A subclass owes a key, a read and a resolveUri for the files the model refers to, and a GLB carries its buffers and images inside, so a source over one file is ten lines:

final class BytesSource extends AssetSource {
  const BytesSource(this.name, this.bytes);

  final String name; // 'sticker-4711.glb': the extension helps pick the decoder
  final Uint8List bytes;

  @override
  String get key => 'bytes:$name';

  @override
  Future<Uint8List> read(AssetRead request) async => bytes;

  @override
  AssetUriResolver get resolveUri => (AssetRequest request) async {
    if (request.uri.startsWith('data:')) return decodeDataUri(request.uri);
    throw StateError('$name refers to ${request.uri}, which is not in memory');
  };
}

Model3D takes such a model through load, which hands you the device and wants the asset back; skinning, morph targets and clips come along as they do from the bundle:

Model3D(
  load: (device) async => ModelAsset.fromDocument(
    await decodeModelInIsolate(
      ModelLoadRequest(source: BytesSource('sticker-4711.glb', downloaded)),
    ),
    device: device,
  ),
  animation: 'wave',
)

The key is the cache key, so two different files must not share a name: put the download's id or hash into it. A .gltf with separate .bin and image files needs a resolveUri that answers those names too, from a map of the files you hold. The source travels with the request to the decoding isolate, so it holds plain data and nothing that closes over widgets or the device.

A plain https: path is not fetched by the engine in rc.2. Model3D(source: 'https://…') and loadModelAsset treat a path whose scheme nobody registered as a file of the project, so a URL fails as a file that is not there. Download the file yourself and use a source like the one above, or register your own source for the scheme with addSource on the application's ModelDecoders; Model3D reads through the engine's registered sources, so source: 'https://…' then works as written. The application may take any scheme, https included, and plugins may not. A network source with an offline cache, and the client behind HubAssetSource, come in rc.3; until then HubAssetSource refuses with HubRefusal.offline. The flutter3d_testing package has a MemoryAssetSource built the same way, with sibling files by name and a log of every read, but it is meant for tests.

Do not await a model before the first frame #

// A box now, the model when it arrives.
final placeholder = _boxRunner(device, scene, runner);
unawaited(_dressRunner(device, scene, runner));   // swaps it in later

glTF 2.0 / GLB #

Compressed geometry is decoded, both KHR_draco_mesh_compression and EXT_meshopt_compression. Not supported: TEXCOORD_1 and up. That, and a Draco payload that does not decode, are reported in warnings rather than failing the file, and the demo surfaces those, a skipped primitive explains a model that looks odd but still loaded.

KTX2 and compressed textures #

A texture can arrive as a KTX2 container, sniffed before dart:ui ever sees the bytes, and two kinds of file come out of it. A Basis Universal ETC1S file is transcoded to RGBA8 in Dart, on an isolate where there is one, with its mip chain and its alpha slice; it costs what a PNG of the same size costs on the device and downloads at a fraction of it. A file carrying its own BC, ETC2 or ASTC blocks is uploaded as those blocks, levels and all, which is the upload that shrinks device memory: a 2048² BC7 texture is 5.6 MB where RGBA8 with a chain is 22.

The device is asked first. GraphicsDevice.supportsTextureFormat answers per format, from flutter_gpu's own per-family capability on Impeller, from the extensions the context handed back on WebGL2, and with a constant no for anything compressed on the software rasteriser, which samples raw texels. A no is a texture left out with a sentence in warnings naming the format, never a guess at a decoder the engine does not have. In glTF, KHR_texture_basisu is read: a core source wins while it exists, since a PNG always decodes, and the extension's KTX2 is what a file that ships only that falls back to.

UASTC LDR 4×4 files open too, unpacked to RGBA8, and Zstandard and ZLIB supercompression are unwrapped by a decompressor written in Dart, so a web build reads them as well. Still refused by name: UASTC HDR, texture arrays, cube maps, 3D textures, and a file that asks for its mip chain to be generated at load. The key/value section is read for the same reason and refuses three more (a bottom-up KTXorientation, a KTXswizzle that is not rgba, and KTXpremultipliedAlpha), because those three describe the pixels themselves and not where they came from, and honouring none of them silently draws a texture upside down, channel-shuffled or twice darkened, which reads to an artist as their own mistake. The converter writes KTX2 as well: a build's texture family decides which blocks.

Wavefront OBJ #

OBJ predates PBR, so its parameters are explicitly approximated: Kd becomes base colour, the Phong exponent Ns becomes roughness, and a bright neutral Ks is the only hint of metalness available. The approximation is documented in MtlMaterial rather than hidden.

STL #

Nothing about STL is approximated the way OBJ's materials are, because there is nothing to approximate: a triangle soup with an optional normal a facet either does or does not carry.

Writing goes the other way: StlWriter bakes every surface's own transform into its positions the same way ObjWriter does, and a document of more than one surface merges them into the single flat list STL has room for, with a warning naming the merge, since STL has no boundary between surfaces to keep them apart on the way back in.

The .f3d container #

The format matters far more than the language. The same geometry is about 360× slower to load as OBJ text than as a binary buffer, and native code does not close that gap (measured in ARCHITECTURE.md §14). So .f3d moves the parse off the device entirely.

dart run flutter3d_build:convert ../flutter3d_samples/assets/teapot.obj \
  -o ../flutter3d_samples/assets/f3d/teapot.f3d

Vertex and index arrays are stored exactly as MeshData holds them, so loading builds Float32List.views over the file rather than copies, every blob entry is 4-byte aligned precisely so those views are legal. A section directory rather than fixed header fields, so a reader skips a kind it does not know and the version only changes when an existing record does.

F3dDocument is a ModelDocument, so ModelAsset.fromDocument, the cache and instancing are all unchanged. The converter re-reads what it wrote and compares it against the source before reporting success.

Everything 0.8.0 added went in as new sections, each written only when a file has something to put in it: material layers (22), variants (23), animation pointers (24), a baked impostor (25), mesh clusters (26) and each level of detail's measured error (27). A reader that predates one skips it. For clusters that means drawing the whole mesh, for an impostor the chain ends at its coarsest mesh, and without the errors the levels switch by screen fraction. What the converter can add, and the per-device-class files loadModelAsset reads first, are on the asset pipeline page.

The resource cache #

ResourceCache is reference-counted, so a mesh shared by forty torches is uploaded once and freed when the last one lets go.

final handle = await cache.acquire('assets/models/torch.glb');
final asset = handle.value;
// ...
handle.release();

cache.evictUnused();

A cache belongs to whoever built it, normally one level load. A GPU resource that outlives the level owning it is a leak nobody notices.

Model3D does not go through a shared cache in rc.2: two widgets showing the same file decode and upload it twice. To share one upload between widgets today, load the asset once yourself and hand it to each through load. A model cache shared by widgets, with Model3D(asset:), comes in rc.3.

Animation #

final player = instance.player;
if (player != null) {
  player.playNamed('run');
  player.crossFadeToNamed('jump', fade: const Duration(milliseconds: 60));
  player
    ..speed = 1.4
    ..advance(dt);
}

Animation is presentation, not simulation #

Advance the player once a frame, not once a step. The simulation runs at a fixed 60 Hz and the animation should run at whatever the display does. Which clip to play is a pure function of the simulation's state, which makes it testable on its own:

// The platformer demo's own `RunnerClips`: a pure function, tested as one.
final wanted = RunnerClips.forRunner(runner);
if (wanted != current) {
  player.crossFadeToNamed(
    wanted,
    fade: wanted == RunnerClips.jump
        ? const Duration(milliseconds: 60)
        : const Duration(milliseconds: 140),
  );
  current = wanted;
}
player
  ..speed = RunnerClips.rateFor(wanted, groundSpeed)
  ..advance(dt);

Skinning #

A glTF skin decodes into a Skeleton of ordinary scene nodes. 64 joint matrices in a uniform array, four weights a vertex, a separate skinned vertex stage, because a second layout is a second shader, and bounds taken from the posed skeleton rather than the bind pose.

A garment is moved onto the body by name. Clothing is exported rigged to its own copy of the body's joints, which no clip moves. skeleton.rebound(body.root) keeps the garment's inverse binds and takes the body's nodes that carry the joints' names (nameOf: maps them where the garment's joints carry a prefix the body's do not), and throws an ArgumentError naming every joint it does not find; assign the result to the garment's meshes and they bend with the body. The cookbook's clothing_on_a_body.dart dresses the walking robot in gloves that way, and its test holds them to the body's joints, node for node.

Morph targets #

A face is a mesh blended towards the shapes its file carries. The deltas go into an r32g32b32a32Float texture, one column a vertex and three rows a target (positions, normals, tangents), uploaded once with the mesh and sampled in the vertex stage. A mesh wider than the device's widest texture (8192 texels on many) wraps its vertices into further bands of rows, which MorphTexture.bands counts, so a head of ten thousand vertices morphs where it used to load expressionless. The weights live on the node, so two copies of a model wear different expressions from one upload, exactly as two copies of a rigged model pose from their own skeletons.

// From a clip, or by hand.
node.morphWeights[0] = 0.6;              // one mesh
batch.setMorphWeights(7, [0.6, 0.0]);    // one copy in a batch

The targets are known by name too, where the file names them (glTF keeps the names in mesh.extras.targetNames, a convention every exporter follows). MorphState.targetNames lists them, setWeight('jawOpen', 0.4) answers false and changes nothing for a target the mesh lacks, which is what a face-tracking rig driving fifty-two names at a head that carries some of them wants, and copyWeightsFrom(body.morph!) puts a garment's shapes where the body's of the same names are.

Material variants #

A glTF file with KHR_materials_variants carries several looks for one model, such as a chair in three fabrics, and names them. ModelAsset.variants lists the names, and an instance changes into one by name:

final chair = asset.instantiate(scene);
chair.selectVariant('leather');   // false, and nothing changes, for a name the file lacks
chair.selectVariant(null);        // back to the default look
chair.variant;                    // which one it wears

A part the variant does not mention wears its default material, as the extension says. Switching changes only which material each mesh node points at, and the materials themselves stay as they are, so two instances sharing the asset's materials can wear two variants at once. ModelAsset uploads every variant's material with the model, so the first switch uploads nothing. In a .f3d the variants are ModelDocument.variants and a per-surface variantMaterials map.

Animated materials and lights #

KHR_animation_pointer lets a clip move something other than a node. A track on a material's base colour, emissive strength, roughness, metallic or texture offset, or on a light's colour or intensity, plays through the same AnimationPlayer as the rest of the clip and lands in the instance's pointerTargets.

final lamp = asset.instantiate(scene, shareMaterials: false);
lamp.bindLight(0, bulb);             // a LightNode you placed stands in for the file's light 0
lamp.player?.playNamed('flicker');

Two things are the caller's decision. Materials are shared by default, so a material track moves every instance wearing that material, the same way tinting one does; instantiate with shareMaterials: false for copies that animate on their own. Instantiating creates no lights, so a track aimed at the file's light reaches only a light you placed and handed to bindLight by the file's index. ModelAsset.materials keeps each bound material by its index in the file, which is how a pointer names it.

Splat captures by the page #

A Gaussian splat capture loads directly, with parseSplatPly, parseSplatSpz (SPZ versions 1 to 4, in Dart) or as KHR_gaussian_splatting primitives in a glTF, where each lands on its node in ModelDocument.splats. A large capture is better converted first into a .f3dsplat tree (how) and drawn at a budget, a page at a time:

final pages = await PagedSplatOctree.open(
  (offset, length) => fetchRange(url, offset, length),   // an HTTP Range request
);
final lod = SplatLod.paged(pages, budget: 400000);
renderer.renderSteps.addContributor(SplatContributor.lod(lod));

open reads the header, the node table and the root's page. After that a page is fetched only when the cut wants to draw finer than what has arrived, most visible first and at most maxInFlight (4) requests at a time, so a distant cloud on a slow connection costs the pages it is seen at. The fetching is the caller's: a SplatRangeReader is one function from a byte range to its bytes, and this package depends on no network library. splatBytesReader wraps bytes already in memory. SplatLod.budget is the most splats a cut may hold, and a budget below the root's own count draws nothing.

Next #

• Simulation layer: the fixed step the animation is not tied to
• Scene graph, where an instantiated model lands
• Tutorial: first scene: loading a model end to end
• The asset pipeline: converting a project's own sources into what this page's decoders read, on every build, with levels of detail, impostors and a file per device class