ASR & Speech Model Testing
Speech recognition testing across WER, CER, semantic and entity error rates, diarization and noisy-condition performance in 30+ languages and regional accents.
What We Test
Transcription accuracy; accent, dialect and demographic coverage; acoustic robustness; speaker attribution and structure; and downstream usability.
Why Word Error Rate Isn't Enough
Every error weighs the same, better capture can be penalised, the human reference is not neutral and valid formatting variants can score as errors.
| Metric | What it measures | When it matters |
|---|---|---|
| WER | Word insertions, deletions and substitutions | Baseline comparability |
| CER | Character-level error | Indic scripts and rich morphology |
| Semantic error rate | Whether meaning survived | Meaning-sensitive products |
| Missed entity rate | Whether critical terms survived | Names, drugs, amounts, IDs and phone numbers |
| Keyword / intent accuracy | Whether downstream words survived | Assistants, IVR and commands |
| Diarization error rate | Speaker attribution and overlap | Calls, meetings and consultations |
| Formatting accuracy | Casing, punctuation and numerals | Human or parser consumption |
| Latency / real-time factor | Speed under load | Streaming and agents |
Accent, Dialect and Regional Testing
The external Voice of India benchmark evaluated 15 major Indian languages and 139 regional clusters—306,230 utterances, 536 hours and 36,691 speakers.
| Finding | Published number |
|---|---|
| District-level WER range | ~4% Nainital to ~44% Mannarakkat |
| Best overall model, Hindi | ~5% WER |
| Same model, Bhojpuri | 20.9% WER |
| Same model, Maithili | 24.8% WER |
| Hindi-belt districts | Below 10% |
| Kerala, interior Karnataka | Substantially higher |
| Female vs male speakers | 3.1–4.3 points better for female speakers |
| Speakers aged 18–22 | Higher error rates |
| Audio-quality quartiles | 15.31% → 25.20% WER |
Source: Voice of India benchmark, 2026. External evidence, not eQOURSE measurement.
Testing Under Real Conditions
| Condition | What we test | Why it bites |
|---|---|---|
| Telephone bandwidth | 8 kHz, codecs and packet loss | Wideband-trained models degrade |
| Background noise | Traffic, crowd, home, office and machinery | Performance has a cliff |
| Reverberation & distance | Echo, far-field mic and speakerphone | Common in real deployment |
| Overlapping speech | Interruptions and multiple speakers | Diarization and transcription fail together |
| Compression & pipeline | Codec chains and resampling | Production differs from test audio |
| Disfluency & hesitation | False starts, fillers and correction | Absent from read prompts |
Voice Agents and Conversational Systems
Endpointing, barge-in, latency under load, error propagation, correction and recovery, plus ASR hallucination on silence, noise and non-speech.
The Reference Transcript Is Part of the Test
Native-variety transcribers, double-pass agreement, documented normalisation and a stated measurement ceiling.
How an Engagement Runs
| Step | Activity | Window | Output |
|---|---|---|---|
| 01 | Scope & metric selection | Week 1 | Metrics, languages and conditions |
| 02 | Test set design | Weeks 1–3 | Matched, stratified test set |
| 03 | Reference transcription | Weeks 2–4 | Native-variety references and agreement |
| 04 | Measurement | Weeks 4–5 | Metrics per stratum |
| 05 | Failure analysis | Weeks 5–6 | Typed error backlog |
| 06 | Report & walkthrough | Week 6 | Findings and reusable assets |
What You Get
| Deliverable | Contents |
|---|---|
| Accuracy report | Every agreed metric by language, region, demographic and condition |
| Missed entity analysis | Critical terms by category |
| Error taxonomy | Failures ranked by frequency and impact |
| Audio examples | Clip for every error class |
| Reference transcripts and test set | Versioned and reusable |
| Agreement statistics | Reference quality before findings |
| Condition curves | Where performance falls off |
| Live walkthrough | Engineering review |
Where ASR Testing Goes Wrong
| Failure | What happens | How we avoid it |
|---|---|---|
| WER only | Critical errors averaged away | Use entity and semantic metrics |
| Single national number | 4%–44% spread hidden | Report every stratum |
| Unmatched group difficulty | Condition gap mistaken for accent gap | Control content and condition |
| Studio audio only | Production path omitted | Reproduce the actual path |
| Standard-variety transcribers | Reference errors attributed to model | Use native-variety transcribers |
| No normalisation policy | Valid variants score as errors | Document rules first |
| Read prompts only | Natural speech absent | Include spontaneous conversation |
| Diarization ignored | Right words, wrong speaker | Measure DER |
| Hallucination untested | Fabricated transcript passes | Use silence and non-speech probes |
| Ceiling ignored | Reference limits look like model error | State measurement ceiling |
What We Do Not Do
We do not build or tune ASR models, report one blended number in isolation, accept unaudited sampling as representative or process voice data outside agreed controls.
How to Engage
| Programme | Best fit | Window |
|---|---|---|
| Baseline accuracy programme | Full matched baseline | 5–6 weeks |
| Accent & dialect audit | Regional coverage | 3–4 weeks |
| Release-cycle testing | Repeat regression test | 1–2 weeks |
| Vendor comparison | Matched provider comparison | Scoped |
Related Services
AI Model Testing Human Evaluation & A/B Testing Audio & Speech Annotation AI Bias & Fairness Audit LLM Evaluation Data Collection Dataset QA & Label Audit
ASR & Speech Model Testing FAQs
What is ASR testing?
Structured measurement of a speech recognition system's accuracy and robustness — word and character error rates, plus semantic and entity-level error rates, diarization accuracy, formatting quality and latency — across the languages, accents, demographics and acoustic conditions your users actually present. The output is a per-stratum report and a typed error backlog, not a single accuracy figure.
Isn't WER enough?
No. WER weights every error equally, can penalise a more accurate model when references miss valid speech, and scores formatting variants as errors. We report WER for comparability alongside metrics that reflect what the product depends on.
What is missed entity rate?
The proportion of important terms — names, drug names, amounts, account IDs, phone numbers and addresses — that failed to survive transcription. Published comparisons reported 0% versus 8.3% on drug names and 19.6% versus 30.0% on phone numbers.
Why does regional accent matter so much in India?
The Voice of India benchmark covered 15 languages and 139 regional clusters across 306,230 utterances. District WER ranged roughly 4%–44%; one model scored about 5% on Hindi, 20.9% on Bhojpuri and 24.8% on Maithili.
Do you test code-mixed speech?
Yes. Hinglish, Tanglish, Benglish and mid-sentence switching are included as spontaneous speech, not only read prompts.
Can you test in noisy and telephone conditions?
Yes. We test telephone bandwidth, codec chains, graded noise, reverberation, far-field capture and overlapping speech, preferably from the real production path. Published Voice of India results moved from 15.31% to 25.20% WER across audio-quality quartiles.
How do you make sure reference transcripts are accurate?
We agree verbatim or clean standards, use native-variety transcribers, double-pass a sample, report inter-transcriber agreement, document normalisation and identify the human-reference ceiling.
What is ASR hallucination and do you test for it?
Some end-to-end speech models produce fluent fabricated text for silence, noise or non-speech. We probe it explicitly because fabricated speech can become fabricated intent downstream.
How is this different from audio annotation?
Audio and speech annotation produces transcripts and labelled audio used to train or fine-tune a model. ASR testing produces error rates and failure analysis used to decide whether a model is fit to ship.
How long does it take, and can we reuse the test set?
A first programme covering two or three languages with regional strata typically runs 5–6 weeks. The versioned test set is delivered to you, so later model versions can run in 1–2 weeks.