How your video becomes subtitles, without leaving your device

The model: Whisper, in two sizes

The words come from Whisper, a speech recognition model that OpenAI published in 2022 with its weights under the MIT License. It runs through whisper.cpp, an open-source C++ version of it, which we compile to WebAssembly so that it can run in a web page. You choose between two sizes:

• The smaller model (the default) is Whisper base, stored with its weights rounded to 5 bits: a 59.7 MB download.
• The larger model is Whisper small, stored with 8-bit weights: a 264.5 MB download that needs more memory.

In OpenAI's paper, the larger Whisper models did better in most of its tests, with the smallest gains for English, but we have not yet measured how these two compare on this engine, so the page calls them "smaller" and "larger" rather than promising that one is more accurate. Before the model, a small voice detector, Silero VAD, marks where people speak, so the model spends its time on speech. When you leave the language on automatic, the model identifies it in the first window that has speech, and the page keeps that language for the rest of the file.

Each window is written with the last words before it, about 200 characters, as a prompt: the model reads them as the text that came just before the cut. As soon as a window is done, its cues appear on the page, and you can edit them while the next window is written. Make subtitles from your video to see it.

How words become cues

The model gives segments of text, each with a start and an end, and its estimate of when the words in it start and end. A segment can be a long sentence, so the page cuts it into cues a viewer can read in time:

• At most two lines of 42 columns per cue; a wide character, as in Chinese or Japanese, counts as two columns. Scripts written without spaces are split into words with your browser's word segmenter.
• At most 7 seconds on screen, and at least 1 second when the gap to the next cue allows.
• Cuts fall between words: at the end of a sentence first, then at a comma or a pause of half a second or more, then where the two parts come out even. A closing mark stays with the word before it.
• The cut is timed from the word times. Where the model gave none that match the text, the time is shared in proportion to the text.
• Cues never overlap, and none ends after the file does.

The times come from the model's own estimate of where its words start and end, or from that share of the text, so a cue can appear a little early or late. We have not measured by how much on this engine yet. If a cue is off, you can fix a cue by hand: move its start or end by 0.1 second, split it, or merge it with the next.

Which engine build runs

The engine exists in three builds, and the page picks one when you add your first file:

• WebGPU, tried first when your browser offers a graphics adapter that passes the page's check. If it fails while loading or while writing a window, the page switches to a processor build and writes that window again.
• WebAssembly on several threads, one thread per logical processor your browser reports. Threads need a cross-origin isolated page, which is why the site sends the COOP and COEP headers.
• WebAssembly on one thread, the last resort for a browser that cannot run threads, and the build used when your browser reports one logical processor.

The line under the preview names the build that ran, for example "WebAssembly on several threads". We have not measured the WebGPU build on a real graphics card yet, so the site makes no claim about its speed. Whatever the build, your device does the work: the status line shows an estimate of the time left, measured on your device as it goes, instead of a promised speed.

What your browser downloads, and what it keeps

Everything comes from this site. The models are split into parts of at most 20 MiB, because the host serves static files of up to 25 MiB, and each part is checked against its SHA-256 fingerprint before use, so a damaged download is refused instead of producing wrong subtitles. Sizes are those of the files as published, in megabytes of a million bytes:

The engine, the media reader, the model, and the detector download only once you add a file, not when you open the page. Your file, its sound, and your subtitles are never stored by the page: they live in the open tab and go when you close it. To remove the kept model, clear this site's data in your browser. The network test step by step shows how to watch all of this in your browser's developer tools.

How we test it

An automated browser test runs on every change to the site. It loads a built copy of the site in Chromium, the engine of Chrome and Edge, with the site's own security headers, and checks, among other things, that:

• a 3.5-second English sentence from the LibriSpeech corpus, "Concorde returned to its place amidst the tents.", comes out as cues with at most two words wrong, with the smaller model, from a WAV file, an MP4 video, and a WebM file;
• after an edit, a 0.1-second move, and a split, the SRT and VTT files read back to exactly the cues on the page, and Chromium's own subtitle reader loads the VTT;
• on a 60-second file with the same sentence every 10 seconds, run with test windows of 20 seconds instead of the tool's 3 minutes, cues appear in at least three steps, and at each of the six marks a cue holding the whole sentence starts between half a second before the mark and 2 seconds after it;
• Cancel stops a run, and the next file reuses the model the browser kept, with no new download;
• a silent clip gives no subtitles, a damaged part of a file keeps the cues before it, and a file that is not media is refused;
• every request goes to this site, as a GET or HEAD with no body.

That test uses one short sentence from one speaker, in one language. It shows that the pieces work together; it is not a measure of how accurate the subtitles are.

Limits worth knowing

• Accuracy and timing are not measured yet on this engine, so no page gives a figure. In OpenAI's paper, Whisper's error rate differed from language to language, closely following how much training speech each language had, and it rose as added background noise got louder.
• No speaker names, no sound descriptions, no translation. The subtitles are the words spoken, in the language spoken.
• Long files take long. Where your browser can, the page reads a slice at a time and never holds the whole file's sound at once, but every minute of audio is work for your processor. The longest file we have run so far is 12 minutes.
• Some files are read whole. Where your browser cannot decode a file's sound in slices, the page decodes it at once and stops at 500 MB or 1 hour.
• Phones are untested. The larger model may need more memory than a phone has free; the page says so when the browser reports running out.

The open-source components and their licenses are listed on the open-source credits page.