Whisper accuracy by language: measured with FLEURS

Why accuracy differs by language

Speech-recognition models learn from the audio and text pairs they are trained on, and that training data is not evenly split across the world's languages. English, Spanish, and the other languages near the top of this page's table have decades of transcribed broadcast, audiobook, and web audio behind them. Languages spoken by fewer people online, or written in scripts with less digitized text, contribute a smaller share of the training mix, and a model's accuracy on them tracks that imbalance closely. This is not a FastScribe-specific effect; it shows up in every large speech model built the same way, ours included.

Script also matters on its own terms, apart from training volume. Languages written without spaces between words, Mandarin and Japanese among them, force a transcription model to make word-boundary decisions that alphabetic languages never ask for, which is one reason those two languages get scored in characters rather than words later on this page. Languages with rich diacritics and vowel marks, Hindi and Arabic among them, add another failure mode: a model that gets the consonants right but drops a mark changes the word, and a text scorer that strips those marks, a common shortcut, hides the very errors that matter most. Our own scoring method, covered below, keeps the marks in.

None of this shows up on a marketing page that lists "99 languages supported." Supported means the model produces some output in that language, not that the output is any good, and a vendor with no measured numbers to publish has no reason to tell you which of its 99 languages are excellent and which are barely usable. The only way to know is to measure each language on its own, print the number, and let a bad score stand next to a good one. That is what the rest of this page does.

How we measured

The benchmark runs on Google's FLEURS dataset, a read-speech corpus built specifically to compare speech models across languages, using its test split. We drew 200 utterances per language across twelve languages: English, Spanish, French, German, Italian, Portuguese, Russian, Arabic, Mandarin, Japanese, Korean, and Hindi. Two of Hindi's 200 clips had a corrupted transcript file on our end and were excluded after a repeat run reproduced the identical corruption, so Hindi is scored over 198 clips rather than 200; every other language kept its full 200.

Every clip ran through the exact engine that processes a customer's upload: large-v3-turbo, quantized to 8-bit, with voice activity detection and automatic language detection both switched on, nothing pinned or tuned for the test. That is the same model and the same settings covered in our Whisper speed benchmark, which measures how fast this engine runs; this page measures how accurate it is. Running the production path, rather than a cleaner research configuration, means the numbers here are the ones your upload actually gets, not a best case.

Scoring compared the model's output against FLEURS' reference text after light normalization: English uses an English-specific normalizer, and every other language uses a marks-preserving basic normalizer, the method the original Whisper research itself uses, so the numbers below are comparable to published benchmarks rather than inflated or deflated by a home-grown scoring rule.

Two limits are worth holding in mind for every number on this page. FLEURS is read speech: one person, a quiet room, a written sentence read aloud, which flatters every speech model against the meetings, calls, and multi-speaker recordings people actually upload. And 200 clips is enough to be confident, not enough to be exact: the 95% confidence interval runs about plus or minus one to one and a half points for the strongest languages, widening to about plus or minus two and a fifth points for Arabic and almost five points for Hindi. Treat every point estimate below as sitting inside that range, not as a fixed number.

Want to try it now? Upload a file to FastScribe. One file a week is free, no signup.

Results

The table below ranks all twelve languages from best to worst on their primary metric, word error rate for alphabetic languages, character error rate for Mandarin and Japanese, with the 95% confidence interval and automatic-language-detection accuracy alongside. Lower is better on the error-rate column; 100% is the ceiling on the detection column, and only one language misses it.

The Hindi number, explained

Hindi's 48.5% is the number that ships, and it deserves the explanation rather than a footnote. Whisper's automatic language detector misdetects 44 of Hindi's 198 clips: 40 as Urdu, the same spoken language written in a different script, Perso-Arabic instead of Devanagari, and the rest split across English, Indonesian, and Spanish. Scoring a Perso-Arabic transcript against a Devanagari reference produces close to 100% error on every one of those Urdu-labeled clips even when the words were heard correctly. Restricting the comparison to the 154 clips Whisper correctly detected as Hindi, 77.8% of the total, drops the error rate to 32.5%, a large improvement, though it still sits well above the 17% that OpenAI's own published FLEURS numbers report for the larger, unquantized version of this model on Hindi. We do not fully understand that remaining gap. The likeliest contributors are that 8-bit quantization degrades a lower-resource language like Hindi more than a well-resourced one, and that a general-purpose normalizer may not capture Hindi-specific spelling variation as well as a language-tuned one would, but that is a plausible explanation, not a diagnosis, and we would rather print the number than wait until we have one.

WER vs CER, and what the numbers mean

Word error rate counts the edits, substitutions, deletions, and insertions, needed to turn the model's output into the reference text, divided by the number of words in the reference. A 5% word error rate means, on average, one word in twenty is wrong; at Spanish's measured 3.7%, it is closer to one word in twenty-seven. These are not exotic units: they are the same metric OpenAI itself publishes for Whisper, which is why this page's numbers sit next to theirs without conversion.

Character error rate does the same arithmetic at the level of characters instead of words, and it is the right metric for Mandarin and Japanese specifically, because neither language marks word boundaries with spaces the way English or Spanish does. A word-error scorer needs to know where one word ends and the next begins; asking it to guess that boundary in unsegmented Chinese or Japanese text introduces noise that has nothing to do with transcription quality. Korean is also written without spaces between syllable blocks by strict linguistic convention, but in practice Korean text is whitespace-segmented, roughly 14.5 tokens per sentence in this test set, so word error rate is a valid metric for it, and that is what this page reports.

The last thing worth holding onto is what FLEURS is not. Every number on this page comes from one speaker at a time, reading a written sentence, in a quiet studio recording. Real audio, a meeting with three people talking over each other, a phone call through a bad connection, a lecture hall's echo, is harder than this test in every direction at once. Read-speech benchmarks like this one are a floor on real-world error, not a ceiling: expect your own results to run somewhat worse than the numbers above, more so as your audio moves further from a single clear voice reading prepared text.

What this means for your uploads

If you are transcribing English, Spanish, French, German, Italian, Portuguese, Russian, Japanese, or Mandarin, this benchmark's news is good: every one of them lands under eleven percent error on clean read speech, with the top four, Spanish, Italian, Portuguese, and German, spanning 3.7% to 6.1%. That is squarely in the range where an automated transcript is a strong first draft you proofread, not a rough guess you have to rebuild from scratch.

Arabic and Hindi are where you should budget more review time. Arabic measures 17.0% word error, roughly one word in six, and while that is not a language-detection artifact the way Hindi's is, it is meaningfully worse than every language above it and worth a careful proofread on anything you plan to publish or quote. Hindi is the language where you should expect to do the most correcting: 48.5% as shipped, 32.5% even on the clips the detector got right, both numbers well above what this page's other eleven languages return.

Beyond the per-language numbers, three situations degrade any language's accuracy below what this page measures. A speaker with an accent or dialect the training data under-represents will transcribe worse than the reference speakers in this test, since FLEURS reads clips from a limited pool of readers per language. Crosstalk, more than one person talking at once, breaks a transcription model the same way it breaks a human listener, and no per-language number here accounts for it. And code-switching, a speaker moving between two languages mid-recording, is the hardest case of all: FastScribe's automatic detector picks one language for the whole file, so a recording that mixes English and Hindi, for instance, transcribes well in the stretches matching whichever language got detected and badly in the stretches that do not.

Where FastScribe fits

These are the same twelve-language numbers FastScribe actually runs in production: large-v3-turbo, quantized, with voice activity detection and automatic language detection on, the identical engine behind every upload regardless of which of these languages you send it. Your first file needs no account, up to 10 minutes and 50 MB, one per rolling week; a free account gives 5 files a day at up to 30 minutes and 100 MB each; Pro is $12 a month for files up to 5 hours and 2 GB, with speaker labels on every file, batch uploads and DOCX export. FastScribe transcribes in the language your audio was spoken in; it does not translate, so a Spanish recording returns a Spanish transcript, not an English one. If your material is in one of the nine languages that measured under 11% error here, treat the automated draft as a strong starting point. If it is Arabic or, especially, Hindi, budget real proofreading time; we would rather tell you that plainly than let a bad first impression happen on your material.