Whisperai
Updated November 2026 Whisper guide

Meet Whisper, speech recognition built for real voices

A clear look at Whisper, the open neural network that transcribes and translates speech with near-human accuracy, even through accents, noise and jargon.

This guide covers Whisper as first released in 2022.

Need a ready-made tool for audio and transcripts today?

Open the transcriber
Listen to this page 2:21

Whisper examples:

“Okay, the shipment lands Tuesday, the invoice goes out Wednesday, and if anyone asks, the demo moves to Friday at nine sharp.”

“We grew up near the harbour, and every summer my grandfather took us out on his little fishing boat before sunrise.”

“Sorry, it is loud in here. Can you hear me? I said the Whisper notes are in the shared folder now.”

[French to English] “I would like to book two tickets for the Saturday night concert, please.”

Whisper is a speech recognition model that learned from 680,000 hours of labelled audio covering many languages and tasks, gathered from across the open web. That breadth pays off: the model copes with accents, background chatter and technical vocabulary far better than narrowly trained systems. It transcribes dozens of languages and can render them into English. The weights and inference code are openly released, so teams can build real products on Whisper and researchers can keep pushing sturdy speech processing forward.

Whisper architecture diagram: a log-mel spectrogram passes through convolution layers and stacked encoder blocks, linked by cross attention to decoder blocks that predict multitask text tokens

Under the hood, Whisper is a plain encoder-decoder Transformer trained end to end. Incoming audio is cut into 30-second windows and turned into a log-Mel spectrogram before it reaches the encoder. The decoder then predicts the matching text, mixed with special tokens that tell one shared model which job to do: detect the language, stamp each phrase with timings, transcribe in many languages or translate speech into English.

Flowchart of the multitask token format: previous text, start of transcript, language tag or no speech, transcribe or translate, then timestamped or plain text tokens ending in EOT

Many earlier systems lean on smaller datasets where audio and text are tightly matched,1 2, 3 or on huge unlabelled audio corpora used for pretraining.4, 5, 6 Because Whisper saw a broad, varied dataset and was never tuned for one benchmark, it does not top the specialist models on LibriSpeech, the classic yardstick of the field. Tested zero-shot across many other datasets, though, Whisper proves far sturdier and makes roughly half as many errors.

Around one third of the Whisper training audio is not in English. For those clips the model is asked either to write down the source language or to translate it into English. This mix turns out to teach speech-to-text translation remarkably well, beating the supervised state of the art on CoVoST2 into English with no task-specific training at all.

English transcription

Please book a table for four…
Please book a table for four…

Any language to English

El tren sale a las ocho…
The train leaves at eight…

Non-English transcription

오늘 날씨가 정말 좋네요…
오늘 날씨가 정말 좋네요…

No speech

(rain on a window)
∅

Whisper training pairs: audio in, text out

We expect the accuracy and simplicity of Whisper to put voice input within reach of many more apps. Revisit the sources, the model diagram and the data samples for the full picture, then turn your own recordings into text.

Sources

  1. 1

    Chan et al. SpeechStew: pooling every available speech dataset to train one large network. arXiv 2104.02133, 2021.

  2. 2

    Galvez et al. The People’s Speech: a large, varied English corpus cleared for commercial use. arXiv 2111.09344, 2021.

  3. 3

    Chen et al. GigaSpeech: a multi-domain corpus holding 10,000 hours of transcribed audio. arXiv 2106.06909, 2021.

  4. 4

    Baevski, Zhou, Mohamed and Auli. wav2vec 2.0: self-supervised learning of speech features. arXiv 2006.11477, 2020.

  5. 5

    Baevski, Hsu, Conneau and Auli. Speech recognition without labels. NeurIPS 34, 2021.

  6. 6

    Zhang et al. BigSSL: scaling semi-supervised learning for speech recognition. arXiv 2109.13226, 2021.

Get started
Get started