Addis Scribe now transcribes Afaan Oromo, live and from uploads. We scored it against every Oromo system we could run on the same held-out natural speech, for accuracy and for how fast text reaches the screen. The clips, every system's transcripts and the scorer are public at addisai/addis-scribe-benchmark, so you can run it again with your own keys.
Addis Scribe has the lowest error rate of every system we tested on natural Oromo speech, live or offline, for both male and female speakers.
On 150 WAXAL test clips, streaming in 1.12 s steps, Addis Scribe records 21.1% WER and 6.8% CER. Google Chirp 2 records 42.4% WER streaming and 34.8% given the whole file, and Google Gemini 29.8% at best. The best open model, Ethio-ASR Oromo, records 24.8% offline.
Read the full accuracy tablesThe transcript is final 37 to 43 ms after speech ends, and first words appear after 1.57 s with 0.32 s chunks. Google Chirp 2 streaming takes 1,051 ms and 6.37 s on the same clips.
Read how latency was measured150 clips from the WAXAL Afaan Oromo test split, 23 speakers, 45 minutes, human transcripts. Every system transcribed the same clips.
150 test clips, word error rate, lower is better
WAXAL, whole-file transcription, lower is better
| System | Mode | WER | CER | Final text after speech ends | First words | Processing per minute of audio |
|---|---|---|---|---|---|---|
| Addis Scribe Oromo | Live, 1.12 s chunks | 21.1% | 6.8% | 43 ms | 2.21 s | n/a |
| Addis Scribe Oromo | Live, 0.32 s chunks | 21.5% | 6.9% | 37 ms | 1.57 s | n/a |
| Addis Scribe Oromo | Upload | 22.3% | 7.6% | n/a | n/a | 0.78 s |
| Ethio-ASR Oromo (w2v-BERT) | Offline | 24.8% | 7.1% | n/a | n/a | 0.68 s |
| Ethio-ASR multilingual 600M | Offline | 25.7% | 7.2% | n/a | n/a | 0.69 s |
| WAXAL benchmark Whisper-small Oromo | Offline | 28.2% | 8.2% | n/a | n/a | 2.70 s |
| WAXAL benchmark MMS-300M Oromo | Offline | 29.5% | 7.9% | n/a | n/a | 0.21 s |
| Google Gemini Flash (latest) | Whole file, default settings | 29.8% | 9.7% | n/a | n/a | 39.90 s |
| Google Gemini 4 Argon | Whole file, default settings | 29.9% | 7.8% | n/a | n/a | 38.13 s |
| Google Chirp 2 (om-ET) | Batch | 34.8% | 9.9% | n/a | n/a | 9.77 s |
| Google Chirp 2 (om-ET) | Streaming | 42.4% | 14.2% | 1,051 ms | 6.37 s | n/a |
Bold marks the best value in each column. Processing per minute is whole-file transcription, one clip at a time; for Google it is the full request time, network included. The processing chart cuts the Gemini bars at 10 s; their values are in the table.
Audio is fed at real-time speed, as from a live microphone. We report how long until the first words appear, and how long after the speaker stops until the transcript is final.
WAXAL, 150 clips, lower is better
WAXAL, 150 clips, lower is better
Addis Scribe was measured on one NVIDIA A100-class GPU. Google Chirp 2 was measured from its europe-west4 endpoint, the nearest region that serves Oromo, so its figures include the network round trip.
Uploads scored on the whole WAXAL Oromo test split, for the systems run on all of it. The ranking matches the 150-clip set.
| System | Mode | WER | CER |
|---|---|---|---|
| Addis Scribe Oromo | Upload | 20.8% | 5.8% |
| Ethio-ASR Oromo (w2v-BERT) | Offline | 24.2% | 6.2% |
| Ethio-ASR multilingual 600M | Offline | 24.5% | 6.1% |
WAXAL word error rate split by speaker gender, taken from the dataset's own speaker metadata.
| System | Mode | Male WER | Female WER | All WER |
|---|---|---|---|---|
| Addis Scribe Oromo | Live, 1.12 s chunks | 22.6% | 19.3% | 21.1% |
| Addis Scribe Oromo | Live, 0.32 s chunks | 22.5% | 20.3% | 21.5% |
| Addis Scribe Oromo | Upload | 23.2% | 21.3% | 22.3% |
| Ethio-ASR Oromo (w2v-BERT) | Offline | 26.3% | 23.0% | 24.8% |
| Ethio-ASR multilingual 600M | Offline | 26.8% | 24.4% | 25.7% |
| WAXAL benchmark Whisper-small Oromo | Offline | 30.2% | 25.9% | 28.2% |
| WAXAL benchmark MMS-300M Oromo | Offline | 30.5% | 28.4% | 29.5% |
| Google Gemini Flash (latest) | Whole file, default settings | 31.4% | 28.0% | 29.8% |
| Google Gemini 4 Argon | Whole file, default settings | 32.1% | 27.4% | 29.9% |
| Google Chirp 2 (om-ET) | Batch | 34.2% | 35.6% | 34.8% |
| Google Chirp 2 (om-ET) | Streaming | 45.9% | 38.4% | 42.4% |
| System | Mode | Male WER | Female WER | All WER |
|---|---|---|---|---|
| Addis Scribe Oromo | Upload | 22.9% | 18.7% | 20.8% |
| Ethio-ASR Oromo (w2v-BERT) | Offline | 25.0% | 23.5% | 24.2% |
| Ethio-ASR multilingual 600M | Offline | 25.1% | 23.9% | 24.5% |
Addis Scribe is the most accurate system for both male and female speakers. Every system except Google Chirp 2's batch mode errs more on the male speakers of this test set; with only 11 to 12 speakers per group, that gap may reflect these speakers and recordings rather than gender itself.
Addis Scribe and Google Chirp 2 run on live audio. Google Gemini and the open models need the whole recording, so they are scored for accuracy and processing time only.
| System | Live audio | How it was run |
|---|---|---|
| Addis Scribe Oromo | Native streaming | Streams in 0.32 s or 1.12 s chunks, so text appears while people speak. Also tested as an upload of the whole file. |
| Ethio-ASR Oromo (w2v-BERT) | Offline only | Needs the whole recording before it can answer. Scored for accuracy and processing time. |
| Ethio-ASR multilingual 600M | Offline only | Needs the whole recording before it can answer. Scored for accuracy and processing time. |
| WAXAL benchmark Whisper-small Oromo | Offline only | Transcribes complete audio. Scored for accuracy and processing time. |
| WAXAL benchmark MMS-300M Oromo | Offline only | Needs the whole recording before it can answer. Scored for accuracy and processing time. |
| Google Gemini Flash (latest) | Offline only | Called through the Gemini API (gemini-flash-latest) with the whole clip and one transcription prompt, default settings. No live mode, so scored for accuracy and processing time. |
| Google Gemini 4 Argon | Offline only | Called through the Gemini API (gemini-4-argon) with the whole clip and one transcription prompt, default settings. No live mode, so scored for accuracy and processing time. |
| Google Chirp 2 (om-ET) | Native (cloud API) | Server-side streaming over the network from europe-west4, the nearest region that serves Oromo. Also tested as a batch request on the whole file. |
The test set is public. None of the WAXAL test clips were used to train our models.
| Test set | Repository | Split | What it measures | Selection |
|---|---|---|---|---|
| WAXAL | google/WaxalNLP | orm test | Natural speech with human transcripts | 150 clips, 23 speakers, 45 minutes, the same clips for every system. Upload accuracy is also scored on the full split of 3,782 clips. |
Every system is scored with the same text normaliser: Unicode NFKC, lowercase, apostrophes kept because they mark the Oromo glottal stop, other punctuation removed. WER is word error rate and CER is character error rate, both lower is better.
Both Gemini models were called through the Gemini API on 27 October 2026 (model ids gemini-4-argon and gemini-flash-latest) with each model's default settings, no thinking or temperature overrides: the whole clip plus one transcription prompt. Processing time is the request time, network included. They were not run on the full test split.
The clips, human references, every system's transcripts from our run, the scorer and a runner for each system are published at addisai/addis-scribe-benchmark. Run any system again with your own keys and see your numbers next to ours on the same 150 clips.
The 150 WAXAL test clips: ids, speakers, speaker gender from WAXAL's metadata, durations and human reference transcripts.
Every system's transcript for every clip from our run, with live latencies where they were measured.
The scorer: the shared text normaliser, WER, CER and the split by speaker gender.
One runner per system with the exact settings we used, a command-line runner and a local web app.
Published at addisai/addis-scribe-benchmark.
pip install -r requirements.txt # add requirements-open-models.txt for the open modelspython fetch_audio.py # downloads the WAXAL clips into data/audio/python app.py # open http://127.0.0.1:8787, paste your keys, press Submit
The app opens at http://127.0.0.1:8787 and runs on your computer only. It shows your WER and CER next to ours on the same clips, marks every error word by word, splits by speaker gender, and measures speed and estimated cost from your run. Your keys are sent only to each service and are never written to disk. You need Python 3.10 or later and ffmpeg.
Add language=om to a Scribe upload, live session or long-file job. Uploads, live streaming, word timestamps, captions and speaker labels all work in Oromo, at the same per-character price as Amharic.