Measured 27 August 2026 · harness v1.0.0
Benchmark: what the pipeline costs in time
This is a real run of the call-analysis pipeline over a fixed corpus of eleven scripted calls, timed stage by stage, with the results published as a data file you can download and check. It measures capture decoding, transcription latency, transcription error against the exact scripts the audio was synthesized from, analysis latency across three tiers, and how often those tiers return the same recommendation.
What this benchmark does not establish
It says nothing about how accurately the platform detects fraud. There is no accuracy, precision, recall or false-positive rate anywhere on this page, because computing one honestly requires a labelled corpus of real calls, and this corpus is synthesized speech generated from scripts written inside this project. Scoring our own scripts would measure our script writing, not the product. What is measured here is time, transcription error against a known reference, and how far independent analysis tiers agree — see what we have not measured yet.
Method
How the run was made
The harness ships in the repository and drives the same code paths a real call takes: the capture decoder, the speech-to-text engine and the analysis engine. Each is timed with a monotonic clock around the real call into it. Nothing is simulated, and no stand-in is substituted for a configured engine: a configuration that cannot run is recorded as an error row in the data file.
Each corpus item is processed exactly as a call would be: RTP captures decoded to a stereo 8 kHz recording, then transcribed. Every recording goes through every transcription tier, and one transcript per call goes to every analysis tier, so the tier comparison is like for like. Model loading sits outside the timed region — a worker loads once at start-up, not once per call — and is reported separately.
the harness, as invoked for this page
python -m app.scripts.benchmark \
--pcap-dir <the two RTP captures and their script> \
--wav-dir <the nine scripted scenario recordings> \
--stt-models <four transcription tiers> \
--llm <three analysis tiers> \
--repeats 2 \
--out benchmark-final.jsonOne fixture needed fixing before it could be decoded
The two capture items ship as playback fixtures for the test call generator, so both legs carry the same stream identity — same synchronization source, same address pair. Fed to the decoder as they are, the legs collapse into one channel and the callee is lost. The harness therefore gives the second leg a distinct stream identity before decoding, as the legs arrive off the media relay in a real call; the rewrite is recorded per call in the data file. Earlier passes, before that step existed, produced single-channel transcripts and were discarded rather than published.
| Property | Value |
|---|---|
| Measured at (UTC) | 2026-08-27T05:41:22+00:00 |
| Harness version | 1.0.0 |
| Container architecture | x86_64 |
| Host | x86-64 container image on an Apple-silicon host |
| CPUs visible to the container | 11 |
| Passes per measurement | 2 |
| Corpus size | 11 calls |
| Total run time | 1267 s |
These are developer-machine numbers
The run was made in an x86-64 container image on an Apple-silicon laptop, so the image runs under emulation, and the analysis tiers were reached over the public internet from a home connection. Production runs on native x86-64 server hardware. Treat the latencies below as this machine's numbers, not a production SLA. The shape of the comparison between tiers carries over; the absolute milliseconds do not.
Corpus
The eleven calls, and where they come from
Two items are RTP captures of the same scripted dialogue, about 73 seconds each, one G.711 A-law and one µ-law; they exercise the decoder as well as transcription. The other nine are stereo recordings of scripted scenarios — scam calls and ordinary business calls — synthesized turn by turn, one voice per party, caller left and callee right, exactly as the decoder lays out a real call.
No real customer or scammer recording is used anywhere in this benchmark.
| Call | Seconds | Media | Scenario | SHA-256 |
|---|---|---|---|---|
| speech-pcma | 73.4 | PCMA | 75 s synthetic card-fraud dialogue, G.711 PCMA RTP capture (two legs) | 666666948fea… |
| speech-pcmu | 73.4 | PCMU | 75 s synthetic card-fraud dialogue, G.711 PCMU RTP capture (two legs) | dccb362086f5… |
| appointment-reminder | 40.4 | WAV | synthetic scenario recording 'appointment-reminder' (stereo, caller left / callee right) | 245d6b855da1… |
| cc-fraud-dept | 57.3 | WAV | synthetic scenario recording 'cc-fraud-dept' (stereo, caller left / callee right) | feb23f3701ee… |
| electric-bill | 59.8 | WAV | synthetic scenario recording 'electric-bill' (stereo, caller left / callee right) | 8230ef6e2f8c… |
| irs-tax-debt | 46.9 | WAV | synthetic scenario recording 'irs-tax-debt' (stereo, caller left / callee right) | 5dbc825bfae1… |
| lottery-fee | 44.5 | WAV | synthetic scenario recording 'lottery-fee' (stereo, caller left / callee right) | 0c56dc8c3a47… |
| medicare-card | 55.3 | WAV | synthetic scenario recording 'medicare-card' (stereo, caller left / callee right) | 11181a446aec… |
| order-status | 47.1 | WAV | synthetic scenario recording 'order-status' (stereo, caller left / callee right) | 1403e1b9d7f2… |
| sample-call | 73.2 | WAV | synthetic scenario recording 'sample-call' (stereo, caller left / callee right) | a90644a1d513… |
| tech-support | 60.3 | WAV | synthetic scenario recording 'tech-support' (stereo, caller left / callee right) | f06efb78296f… |
Stage 1
Decoding the capture
RTP reassembly and G.711 decoding into a stereo recording, timing rebuilt from RTP timestamps so loss and silence suppression become silence rather than drift.
| Call | Media | Audio (s) | Capture (KiB) | Decode (ms) | × real time |
|---|---|---|---|---|---|
| speech-pcma | PCMA | 73.4 | 1643 | 159 | 0.002 |
| speech-pcmu | PCMU | 73.4 | 1643 | 143 | 0.002 |
Stage 2
Transcription: four tiers on the same audio
Three tiers run on the worker itself and one is a hosted service. All four ran over the whole corpus on the same machine, with the same per-channel handling.
| Tier | Calls | Median (ms) | Range (ms) | × real time | Median WER (%) | WER range (%) |
|---|---|---|---|---|---|---|
| On-premise, compact | 11 | 4,868 | 4,301–6,656 | 0.091 | 26.9 | 16.6–43.7 |
| On-premise, standard | 11 | 8,389 | 7,404–12,363 | 0.151 | 19.8 | 2.2–42.9 |
| On-premise, extended | 11 | 24,974 | 23,208–32,651 | 0.445 | 21.8 | 5.4–32.7 |
| Hosted, standard | 11 | 4,895 | 4,127–6,339 | 0.043 | 7.3 | 0.0–12.4 |
Reading the real-time factor
0.20 means a 60-second call took 12 seconds to transcribe on one worker thread. Both channels are transcribed, so the factor already includes the doubled work of per-channel speaker labelling.
Word error rate, and exactly how it was computed
A word error rate is a function of its text normalizer, so here is the normalizer in full. Reference and transcript are both lower-cased, then:
- speaker prefixes are removed;
- & becomes and, % becomes percent;
- a currency symbol moves behind its number ($499 → 499 dollars);
- thousands separators are dropped, and a decimal point becomes point;
- integers are spelled out the way a voice reads them, so a transcript that writes 499 is not punished against a script that spells it out;
- remaining punctuation is dropped, and apostrophes inside words are kept.
The rate is the Levenshtein edit distance over those word tokens, divided by the number of reference words. Substitution, deletion and insertion counts are in the data file, per call.
The reference is the text the speech was synthesized from. This is a transcription quality figure on clean synthetic 8 kHz speech. It is not a claim about transcription quality on real telephone audio with loss, jitter, accents and hold music, and it is not a fraud-detection accuracy figure.
| Call | Audio (s) | On-premise, compact | On-premise, standard | On-premise, extended | Hosted, standard |
|---|---|---|---|---|---|
| speech-pcma | 73.4 | 4,412 / 16.6 | 7,721 / 12.9 | 30,198 / 27.7 | 5,552 / 8.3 |
| speech-pcmu | 73.4 | 6,656 / 30.9 | 12,363 / 31.8 | 32,651 / 27.7 | 5,975 / 12.4 |
| appointment-reminder | 40.4 | 4,868 / 43.7 | 9,014 / 3.9 | 23,334 / 18.4 | 5,198 / 2.9 |
| cc-fraud-dept | 57.3 | 5,769 / 20.3 | 8,643 / 42.9 | 25,742 / 15.8 | 4,378 / 10.7 |
| electric-bill | 59.8 | 4,900 / 30.8 | 8,389 / 29.6 | 24,974 / 32.7 | 5,140 / 6.3 |
| irs-tax-debt | 46.9 | 4,439 / 20.8 | 7,502 / 16.9 | 23,788 / 24.6 | 4,522 / 2.3 |
| lottery-fee | 44.5 | 4,301 / 27.8 | 7,459 / 27.8 | 23,778 / 19.1 | 4,661 / 11.1 |
| medicare-card | 55.3 | 4,592 / 26.1 | 7,969 / 2.2 | 24,116 / 18.8 | 4,895 / 0.0 |
| order-status | 47.1 | 4,326 / 20.7 | 7,404 / 16.2 | 23,208 / 5.4 | 4,127 / 4.5 |
| sample-call | 73.2 | 5,520 / — | 10,773 / — | 28,184 / — | 6,339 / — |
| tech-support | 60.3 | 4,870 / 31.6 | 8,601 / 22.6 | 26,034 / 30.4 | 4,464 / 8.3 |
Stage 3
Analysis: three tiers, same transcripts
Every tier scored the same eleven transcripts, from the on-premise standard transcription tier, so differences below are the analysis engine, not the transcript.
| Tier | Calls | Median (s) | Range (s) | Recommendations returned |
|---|---|---|---|---|
| High-accuracy tier | 11 | 13.8 | 7.6–16.3 | 9 block, 2 allow |
| Balanced tier | 11 | 6.5 | 3.5–8.3 | 8 block, 2 allow, 1 review |
| Economy tier | 11 | 5.2 | 2.9–6.3 | 8 block, 2 allow, 1 review |
Do the tiers agree?
This is the closest thing to a quality signal a corpus this small honestly supports. Agreement is not accuracy — three tiers can agree and all be wrong — but disagreement shows where a call is ambiguous, and how much a cheaper tier would change your queue.
| Pair | Calls | Same recommendation | Same block decision | Median probability gap | Max gap | Disagreed on |
|---|---|---|---|---|---|---|
| High-accuracy tier vs Balanced tier | 11 | 10/11 (90.9%) | 90.9% | 3 | 33 | electric-bill |
| High-accuracy tier vs Economy tier | 11 | 10/11 (90.9%) | 90.9% | 4 | 50 | electric-bill |
| Balanced tier vs Economy tier | 11 | 11/11 (100.0%) | 100.0% | 0 | 17 | none |
Probability gap is the absolute difference between the two tiers' fraud probabilities on the same call, in points of 0–100.
Reproduction
Check it yourself
The published file carries the environment and everything in the table below. Every number on this page is generated from it. Where page and file disagree, the file is right.
The corpus, the harness and the script that generates the scenario recordings are all in the repository, so you can repeat the run on your own hardware. That is the only way to get numbers that mean anything for your deployment.
| Property | Value |
|---|---|
| File | /assets/site/data/benchmark-final.json |
| Size | 81 KiB |
| Schema | ovs.benchmark.public/1 |
| Contains | corpus manifest with SHA-256 per file, every timing, every transcript produced, the word-error breakdown per call, per-tier aggregates and the agreement matrix |
Limits