Whisper Fails Outside Standard Norwegian: The Real Numbers

Headline Norwegian WER hides the Nynorsk gap

Whisper large-v3 scores 6.8% Word Error Rate (WER) on standard Norwegian Bokmål read speech on the NST test set, a benchmark result that looks production-ready on paper. Give the same model Nynorsk speech from the Common Voice test set and WER climbs to 30% (Kummervold et al., Interspeech 2024). That is not a rounding error. That is nearly one in three words wrong, in the same language, from the same model.

The gap is not a Whisper-specific flaw. It is a structural consequence of how general-purpose Automatic Speech Recognition (ASR) models are trained.

The Training Data Problem Behind the Benchmark

OpenAI trained the original Whisper on 680,000 hours of web-scraped audio; large-v3 raised that to roughly 1 million hours of weakly labeled audio plus 4 million hours pseudo-labeled by large-v2. That scale sounds exhaustive until you examine the distribution. Web-scraped speech data skews heavily toward English, and within non-English languages, it skews toward broadcast-quality, standard-dialect recordings, the kind of Norwegian spoken on NRK national radio, not in a Trøndersk fishing cooperative or a Northern Norwegian municipal office.

The result is a model that has learned Norwegian as it appears on the internet, not as it is spoken by the 5.5 million people who actually use it in daily life. Regional dialects, code-switching patterns, and spontaneous conversational speech are systematically underrepresented. Scandinavian languages are a textbook case of this failure mode, but the same dynamic affects Finnish, Danish regional varieties, and Swedish dialects outside the Stockholm standard.

Why This Is a Production Problem Right Now

This matters beyond academic benchmarks. Automotive OEMs shipping voice interfaces into Nordic markets are encountering in-cabin ASR failures that trace directly to dialect coverage gaps in their ASR training data, not to model architecture decisions. Nordic fintech platforms and telehealth providers face the same exposure: voice interfaces that perform adequately in controlled demos and degrade in the field once real users, speaking real dialects, in real acoustic environments, start using them.

The regulatory clock makes this concrete. EU AI Act (Regulation 2024/1689) enforcement for high-risk systems begins August 2, 2026, weeks away as of this update, and Article 10 requires documented data governance for the training data behind in-scope voice interfaces. A dialect gap you cannot explain is exactly the kind of finding an audit surfaces.

The following sections walk through the published evidence, examine what the data distribution underneath it actually looks like, and provide a practical framework for building speech corpora that close the WER gap at the source.

What the Published Benchmarks Cover, and What They Cannot

The most rigorous public evaluation of Whisper on Norwegian is the National Library of Norway’s NB-Whisper work (Kummervold et al., Interspeech 2024). It measures OpenAI’s Whisper variants against three public test sets: NST (studio-quality Bokmål read speech), Fleurs (Bokmål), and Common Voice (Nynorsk). As of mid-2026, large-v3 remains the strongest open-weights Whisper release; the distilled large-v3-turbo trades a small amount of accuracy for roughly 8x inference speed and is trained on the same distribution, so faster inference changes nothing about coverage.

Two things stand out in that evaluation landscape. First, the standard-versus-non-standard gap is measured at the written-standard level (Bokmål versus Nynorsk), because that is what public test sets support. Second, and more damning: no public Norwegian test set is dialect-tagged. The paper states plainly that test data lack dialect metadata to give a realistic picture of ASR performance. The Nordic Dialect Corpus documents 38 distinct pronunciations of the interrogative “who” alone; none of the public benchmarks can tell you how a model handles any of them. The published numbers are therefore a floor on the problem: they show the standard-variety gap and cannot yet see the spoken-dialect gap underneath it.

The spoken dialect groups a production Norwegian corpus must cover:

1. Standard Bokmål (Oslo region), the closest match to Whisper’s training distribution and the de facto baseline
2. Trøndersk (Trondheim region), characterized by distinctive pitch accent inversion and retroflex consonant clusters absent from standard Bokmål
3. Northern Norwegian (Tromsø/Bodø), flat tonal contour, significant phonological distance from Oslo speech norms
4. Western Norwegian / Nynorsk-dominant (Bergen, Sogn og Fjordane), includes speakers who code-switch between Nynorsk lexical forms and Bokmål in the same utterance

The same structure repeats across Scandinavia: Skånska Swedish carries Danish-proximate vowel reduction, and Jutlandic Danish adds stød patterns that sit far from any standard-variety training distribution. The corpus framework later in this article generalizes accordingly.

Why Spontaneous Speech Matters More Than Read Speech

Read speech and spontaneous conversational speech are not the same task. This is well-established in ASR research and consistently underweighted in vendor benchmarks: models lose a double-digit relative margin moving from read to spontaneous speech from the same speakers, before any dialect or acoustic environment factors are introduced.

For in-cabin voice, the compounding is worse: active road noise, HVAC fan noise, multi-speaker overlap, natural hesitations, self-corrections, and mid-command dialect switches. A driver beginning a navigation command in standard Norwegian and completing it in Trøndersk is not an edge case. It is normal speech behavior.

The public Norwegian corpora underneath the benchmarks sit at the easy end of this spectrum. NST (Nordisk Språkteknologi) is studio-condition read speech, and the NB-Whisper team had to assemble parliamentary proceedings and NRK broadcast subtitles on top of it to get anywhere near real speech diversity. Neither distribution reflects what in-cabin ASR systems encounter at 110 km/h on the E6.

If your ASR training data corpus is 80% read speech from capital-city speakers, your benchmark results will not predict production performance. They will predict performance on a task your production system never actually faces.

Audio Annotation Protocol for Dialectal Speech

Dialectal speech annotation introduces problems that generic transcription pipelines are not designed to handle. The first is orthographic ambiguity: Trøndersk and Northern Norwegian have no standardized written form. An annotator transcribing a Trøndersk speaker saying what sounds like “kæm ær du” faces a genuine decision, transcribe in normalized Bokmål (“hvem er du”), attempt a phonetic approximation, or use a dialect-aware orthographic convention. Each choice has downstream consequences for ASR training data quality.

The convention YPAI applies in dialectal transcription work uses normalized Bokmål as the reference transcription for Norwegian dialect groups, with a secondary phonetic tier for dialectal forms that have no Bokmål equivalent. This is consistent with the NST corpus convention and allows WER calculation against a stable reference. The trade-off is that it understates the model’s phonological confusion, a Bokmål-normalized reference will not capture whether the model failed on a phoneme or a lexical form.

Annotator agreement drops sharply on dialectal audio relative to standard speech, which is why disagreements need adjudication by a dialect-specialist annotator and why agreement must be measured per dialect group, never as a blended average. Using general-purpose Norwegian or Danish speakers as annotators without dialect screening produces reference transcriptions with systematic errors, errors that propagate directly into WER calculations and, if the corpus is used for fine-tuning, into the model itself.

Results: Where Whisper Breaks Down and Why

The published results are not ambiguous. Whisper large-v3 is production-grade on standard Bokmål read speech and falls off a cliff the moment the input stops matching its training distribution. The collapse accelerates as models shrink.

WER, lower is better; above 100% is possible when a model inserts more words than the reference contains. Source: Kummervold et al., Interspeech 2024.

Three failure modes account for the majority of errors.

Vocabulary gaps. Dialectal lexical forms that have no Bokmål equivalent, and no representation in Whisper’s training corpus, are either substituted with phonetically similar standard-dialect words or deleted entirely. Trøndersk high-frequency function words with no Bokmål cognate are the canonical example.

Phonological mapping errors. When Whisper encounters a phoneme outside its learned distribution for a given language, it maps it to the nearest standard-dialect equivalent. Northern Norwegian retroflex consonant clusters and the Jutlandic “stød” (a laryngealization feature with no equivalent in Standard Danish) are classic triggers for systematic substitution patterns. The model does not fail randomly, it fails predictably, in ways that reflect the phonological distance between the dialect and the standard variety it was trained on.

Language confusion. This is the most operationally damaging failure mode, and it is addressed in detail below.

Language Confusion: When Whisper Thinks Norwegian Is Swedish

Whisper’s language identification operates on the first 30 seconds of audio using a classification head trained on language-level features. For closely related languages, Norwegian, Swedish, Danish, the acoustic and lexical overlap is substantial, and misidentification on short dialectal utterances is a well-documented failure pattern in the Whisper ecosystem.

The consequence is not a modest accuracy penalty. When language ID is wrong, Whisper applies the wrong language model during beam search decoding, and error rates escalate far beyond the acoustic gap alone. NB-Whisper, the fine-tuned Norwegian model released by the National Library of Norway (Nasjonalbiblioteket), substantially reduces this confusion by retraining on Norwegian-specific data, but no fine-tune can add phoneme coverage the evaluation sets cannot even measure yet.

Forcing the language tag via Whisper’s --language no flag eliminates the language-ID failure but does not close the acoustic model gap. The decoder is now operating in the correct language space, but the underlying encoder still lacks the phoneme coverage to represent dialectal speech accurately. Language forcing is a workaround, not a solution.

The Automotive Edge Case: Dialect + Noise + Short Utterances

The hardest real-world combination is utterances of a few words, ambient road and HVAC noise, and dialectal phonology, all simultaneously.

A driver saying slå på varmen (turn on the heat) in Trøndersk dialect, with HVAC fan noise at highway speed, is a fundamentally different acoustic signal than the same phrase spoken in Standard Bokmål in a quiet recording studio. The phonological form is different. The signal-to-noise ratio is different. The utterance duration, often under 1.5 seconds for short commands, falls below the window where Whisper’s language-ID mechanism has sufficient signal to operate reliably.

No published benchmark measures this combination yet, which is exactly the problem: the conditions your product ships into are the conditions no public test set covers. Given that large-v3 already loses 4.4x from Bokmål read speech to Nynorsk in clean conditions, and that medium and small collapse to 60% and beyond on the same test, shipping an in-cabin dialect deployment without your own evaluation corpus means shipping blind.

The path forward is not prompt engineering or language tag forcing. It requires ASR training data that reflects the actual acoustic conditions and dialectal distribution of the deployment environment. Combining audio with vehicle telemetry, speed, HVAC state, window position, cabin occupancy, as multimodal training data provides contextual signals that partially compensate for acoustic degradation. A model that knows the HVAC is running at high speed can apply a more aggressive noise prior. That kind of domain-specific context does not exist in general-purpose speech corpora, and it cannot be retrofitted through fine-tuning on read speech.

Closing the Gap: Building Dialect-Aware Speech Corpora

The benchmark results above are not an argument against Whisper. They are an argument for building the right training data before deploying it. A structured approach to dialect-aware corpus construction predictably closes the WER gap, but only if the process is designed around the actual deployment conditions, not general-purpose speech collection norms.

Here is a five-step framework for building ASR training data that reflects dialectal reality.

Step 1: Dialect mapping. Before recruiting a single speaker, inventory the specific dialect groups your product must support. Weight them by user population and commercial priority, not by linguistic convenience. A Norwegian automotive voice interface deployed nationally must treat Northern Norwegian dialects as first-class targets, not edge cases. Document which dialects are in scope, which are out of scope, and why. This decision determines your collection budget and annotation requirements downstream.

Step 2: Speaker recruitment. Recruit native dialect speakers, not standard-dialect speakers asked to “speak naturally.” The phonological differences between Standard Bokmål and Trøndersk are not stylistic; they are structural. Standard-dialect speakers cannot produce them reliably on demand. Within each dialect group, recruit across age cohorts, gender, and sociolect. A corpus built exclusively from 25–40 year-old urban speakers will underperform on elderly rural speakers, and that failure will surface in production.

Step 3: Recording environment realism. For automotive AI data, record in actual vehicles under real road conditions, not anechoic chambers or quiet offices. Capture HVAC noise at multiple fan speeds, road noise at highway and urban speeds, and window configurations. For telehealth applications, record with consumer-grade microphones in home environments with representative background noise profiles. The acoustic conditions in your corpus must match the acoustic conditions in your deployment environment. Any gap between the two is a gap in model performance.

Step 4: Annotation with dialect expertise. Assign annotators who are native to each dialect region. Establish transcription conventions before annotation begins, decisions about how to represent dialect-specific phonology, code-switching, and non-standard orthography must be made once and applied consistently. Measure inter-annotator agreement per dialect group separately. A corpus where annotators disagree on 15% of tokens in Northern Norwegian speech is not a 15% quality problem; it is a systematic bias that will propagate through fine-tuning.

Step 5: Iterative fine-tuning and evaluation. Fine-tune your target ASR model on the new corpus, then evaluate per-dialect WER separately, not as a blended headline number. An acceptable blended score can conceal severe failure on a dialect group that represents a material share of users. Identify remaining high-error dialect groups and feed them into the next collection cycle. This is not a one-time project; it is a pipeline.

How Much Dialect Data Do You Actually Need?

The NB-Whisper model, released by the National Library of Norway (Nasjonalbiblioteket), demonstrates what targeted corpus investment produces. Trained on roughly 66,000 hours of Norwegian speech assembled from NST, parliamentary proceedings, and NRK broadcast subtitles, it cuts Whisper large-v3’s WER from 30% to 12.6% on Nynorsk and from 6.8% to 2.2% on NST read speech, relative reductions of roughly 40 to 70 percent depending on test set (Interspeech 2024).

You do not need 66,000 hours to move your metrics meaningfully. Fine-tuning literature consistently shows targeted corpora in the tens to low hundreds of hours producing consequential WER reductions, when the data matches the deployment distribution. That match, not raw volume, is the variable that decides whether the investment pays off.

What does not work: adding 500 hours of standard-dialect read speech. This approach may improve headline WER on clean benchmark sets while leaving dialect-specific error rates unchanged. The model learns more of what it already knows. Annotation quality compounds this dynamic: 50 hours with consistent, dialect-aware transcription outperforms 200 hours with inconsistent annotation. Volume does not compensate for systematic transcription errors; it amplifies them.

The practical target for a production-grade dialect-aware corpus is 50–200 hours per dialect group, sourced from spontaneous speech in realistic acoustic conditions, with annotation handled by dialect-native contributors working from documented transcription conventions.

Compliance Requirements for Nordic Speech Data Collection

Speech data collected in EU and EEA jurisdictions is not generic data. Under GDPR Article 9, voice recordings are biometric data, a special category requiring explicit safeguards beyond standard GDPR Article 6 lawful basis requirements. GDPR Article 7 mandates that consent be freely given, specific, informed, and unambiguous. For a speech corpus, this means each speaker must understand the purpose of the recording, how long it will be retained, whether it will be used to train commercial AI systems, and how they can withdraw consent after the session.

EU AI Act Article 10 adds a second layer for automotive deployments specifically. Voice interfaces in vehicles qualify as high-risk AI systems under Annex III of Regulation 2024/1689. Article 10 requires documented data governance for training data used in high-risk systems, covering data sourcing methodology, annotation processes, known limitations, and quality assurance procedures. This documentation must be maintained throughout the system lifecycle, not assembled retroactively before an audit.

The practical implication: every speaker in your speech corpus needs a documented consent framework covering purpose, retention period, and withdrawal rights. Data provenance, the chain of custody from recording session through annotation through model training, must be auditable. These are not procedural formalities. A corpus built without documented consent and provenance cannot legally serve as training data for a high-risk AI system under the EU AI Act, regardless of its acoustic quality.

Building compliance into corpus design from the first recording session is materially less expensive than retrofitting it after the fact. It is also a prerequisite for any enterprise deployment in European markets.

Build a Scandinavian Speech Corpus That Actually Works

Closing the WER gap on Norwegian dialects, Swedish regional speech, or Danish spontaneous conversation requires training data that was collected with intent, dialect-stratified speaker recruitment, GDPR Article 9-compliant consent frameworks, and annotation by dialect-native linguists who can distinguish Trøndersk from Eastern Norwegian at the phoneme level.

YPAI builds production-grade speech corpora across 100+ languages, including deep Scandinavian dialect coverage, with annotation pipelines designed to meet EU AI Act Article 10 data governance requirements from day one.

For the full corpus build process, see the guide to speech corpus collection for enterprise ASR; for how the engine choice interacts with corpus strategy, the ASR software comparison. And for every published dialect WER result across European languages, not just Scandinavia, see our quarterly European Dialect ASR Benchmark.

Explore YPAI’s speech data collection services or contact us to scope a custom Nordic speech corpus for your ASR system.