---
name: life2film-engine
description: Analyse video and audio on-device with life2film-engine (WebAssembly) — detect tempo and beat positions, find shot boundaries and cuts, score frames for picture quality, and build OpenTimelineIO. Use when the task involves beat detection, BPM, scene/shot detection, cutting video to music, ranking footage by quality, or building editor markers, and the files must not be uploaded to a server.
---

# life2film-engine

Rust video-analysis engine compiled to WebAssembly. Runs in a browser tab or in Node with no server,
no API key and no upload. It **analyses**; it does not decode or encode.

```bash
npm i life2film-engine
```

## The division of labour

This is the mistake to avoid: the engine never touches containers or codecs.

| Job | Use |
|---|---|
| Decode audio to PCM | `AudioContext.decodeAudioData` (browser) |
| Decode video to frames | `<video>` + canvas, or WebCodecs |
| Analyse | **life2film-engine** |
| Encode / trim / convert | [mediabunny](https://mediabunny.dev) (MIT) |

## Loading

Browser — bundlers resolve the `.wasm` automatically:

```js
import init, { detect_beats } from 'life2film-engine';
await init();
```

Node — `fetch` cannot read `file://`, so pass the bytes:

```js
import init from 'life2film-engine';
import { readFileSync } from 'node:fs';
import { createRequire } from 'node:module';

const require = createRequire(import.meta.url);
await init({ module_or_path: readFileSync(require.resolve('life2film-engine/va_wasm_bg.wasm')) });
```

Every export returns a **JSON string** — parse it. The module is 1.2 MB (~410 KB compressed), so
load it on user action, not on page load.

## Beat detection

```js
const ctx = new AudioContext();
const buffer = await ctx.decodeAudioData(await file.arrayBuffer());

// Down-mix to mono at 22050 Hz. The onset envelope lives far below that, so the source rate
// costs several times the memory and time for an identical answer.
const offline = new OfflineAudioContext(1, Math.ceil(buffer.duration * 22050), 22050);
const source = offline.createBufferSource();
source.buffer = buffer;
source.connect(offline.destination);
source.start();
const mono = (await offline.startRendering()).getChannelData(0);

const { bpm, beats } = JSON.parse(detect_beats(mono, 22050, null));
```

Tempo is genuinely ambiguous: 140 BPM is also 70 in half time. If a result is out by exactly 2×,
that is the other valid reading — surface both rather than treating it as an error.

Config (optional third argument, JSON string): `min_bpm`, `max_bpm`, `tightness`, `trim`.

## Shot boundaries

**Sampling rate decides accuracy, not the algorithm.** Measured against a video with cuts at known
times: at 2 samples/second the boundaries land 1–3 seconds out and all sixteen algorithms are wrong
the same way; at 10/second they land on the frame. A boundary cannot be located more precisely than
the gap between the samples either side of it.

Sample frames at ~64 px wide, take the mean RGB of the centre column, then:

```js
const { scenes } = JSON.parse(detect_scenes_slick(JSON.stringify({
  pixels,       // [{ r, g, b }, ...] one per sampled frame
  timestamps,   // [0, 0.1, 0.2, ...] seconds
  duration,
  algo: 'adjacent_diffs',   // scene_algorithms() lists all 16
})));
// scenes: [[0, 3.0], [3.0, 6.0], ...]
```

For long videos, use a coarse pass to find candidates, then re-sample tightly around each boundary
(±one coarse step, at a tenth of it) and take the largest frame-to-frame change. Twenty extra seeks
per cut beats ten times the seeks everywhere.

Also available: `detect_scenes_content` (HSV deltas) and `detect_scenes_features` (N-dim vectors).

## Frame quality

31 measurements per frame — sharpness, exposure, contrast, colourfulness, entropy — as one score
plus a garbage flag.

```js
// RGB24, alpha dropped. 64 px wide is plenty.
const { score, features, is_garbage } = JSON.parse(
  score_frame(width, height, rgb24, timestamp, null),
);
```

Take the **median** score across a shot, not the mean: one bad frame should not sink a good shot.
Weight by duration when combining shots into a clip.

## Full surface

`detect_beats` · `analyze_audio` · `beat_sync_timeline` · `detect_scenes_slick` ·
`detect_scenes_content` · `detect_scenes_features` · `scene_algorithms` · `score_frame` ·
`score_segments` · `select_segments` · `compose_montage` · `build_otio` · `parse_otio` ·
`feature_names` · `get_weights` · `cache_fingerprint` · `cache_prepare` · `cache_validate`

Types in `va_wasm.d.ts`.

## Getting frames in a browser

No WebCodecs needed — `<video>` plus a canvas works everywhere:

```js
const video = document.createElement('video');
video.src = URL.createObjectURL(file);
video.muted = true;
await new Promise(r => video.addEventListener('loadedmetadata', r, { once: true }));

async function frameAt(seconds, width = 64) {
  await new Promise(r => {
    video.addEventListener('seeked', r, { once: true });
    video.currentTime = Math.min(seconds, video.duration - 0.02);
  });
  const canvas = document.createElement('canvas');
  canvas.width = width;
  canvas.height = Math.round((video.videoHeight / video.videoWidth) * width);
  const ctx = canvas.getContext('2d', { willReadFrequently: true });
  ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

  const { data } = ctx.getImageData(0, 0, canvas.width, canvas.height);
  const rgb = new Uint8Array((data.length / 4) * 3);
  for (let i = 0, j = 0; i < data.length; i += 4, j += 3) {
    rgb[j] = data[i]; rgb[j + 1] = data[i + 1]; rgb[j + 2] = data[i + 2];
  }
  return { rgb24: rgb, width: canvas.width, height: canvas.height };
}
```

Each seek costs tens of milliseconds — budget accordingly and show progress.

## Exporting to editors

Beat and cut positions usually need to reach a timeline. Markers land on whole frames, so the
project frame rate changes where they sit: a grid exported at 25 fps sits up to 20 ms off in a
30 fps sequence. Ask for the rate rather than assuming.

- **EDL** (CMX 3600) — DaVinci Resolve, Premiere. One event per marker, with `|C:ResolveColorCyan |M:name |D:1` beneath it.
- **OTIO** — Resolve, Avid, Premiere. `build_otio` handles timelines; for plain markers, hang `Marker.2` objects off a `Gap.1` spanning the duration.
- **FCPXML** — Final Cut. `<marker start="…" duration="…" value="…"/>` inside a gap.
- **Audacity labels** — `start\tend\tname`, tab separated.

## Licence

PolyForm Noncommercial. Free for personal projects, study, research, charities and public
institutions. Anything commercial — including internal use inside a company — needs a licence:
info@life2film.com.

## Working examples

Seven tools built on this engine, with source-level explanations, at
<https://life2film.com/tools/>. Each page is available as markdown by appending `.md`.