We'll publish our numbers. Here's how.
Spark Bench tests Pentecost Spark against single AI engines on the same church recordings. Every result is shown per language, the worst language comes first, and the method and raw outputs are published.
Why the worst language comes first
A single engine rarely fails on average. It fails in one language, or on names, and an average hides that. A congregation that reads Cantonese doesn't care that the English captions were fine. So every Spark Bench result is reported per language, and the headline is the lowest one.
Single engines against Spark's route
The cleaner stage on 282 lines in eight languages. A line counts as dropped when no caption came back within 6 seconds.
| Setup | Lines dropped | Worst language | Cost per 1,000 lines |
|---|---|---|---|
| GPT-6 Luna alone | 12 of 282 | Cantonese: 35% dropped | $0.49 |
| Qwen 3.8 Flash alone | 12 of 282 | English: 20% dropped | $0.16 |
| DeepSeek V4.1 Flash alone | 11 of 282 | Thai: 18% dropped | $0.19 |
| GPT-6 Sol alone | 1 of 282 | English: 3% dropped | $2.78 |
| Pentecost Spark route | 0 of 282 | None | $0.51 |
One run, text input, costs at October 2026 list prices. On line-by-line translation quality the engines scored close to each other in a blind review. In the same test, a pack of Bible names lifted names in the right form from 91% to 98% on the lowest-cost engine.
Spark Bench v1: the method
What we test against
| Group | What it is | What it shows |
|---|---|---|
| Single engines, as sold | One vendor's speech to text plus its machine translation, with its own custom-vocabulary feature: Azure AI Speech translation, Google Cloud Speech-to-Text with Cloud Translation, OpenAI speech to text with one translation model | What you get from a single-engine setup |
| Single engines inside Pentecost | Our cleaner with one model for every language | What per-language routing adds |
| Spark with one feature off | No cleaner, no glossary and Bible texts, no manuscript, or no confidence routing, one at a time | What each feature is worth |
The test set
- Mandarin and Cantonese speakers; English services captioned into Traditional and Simplified Chinese, Japanese, Korean and Thai; then Tagalog and Indonesian.
- Two or three speakers per language, 10-minute clips from the soundboard feed, recorded with each speaker's consent.
- Each clip comes with its manuscript and glossary, as a real service would, and includes quoted scripture and Bible names.
- Reference texts from each speaker's manuscript and the published Bible translations they quote.
What we measure
| Measure | How |
|---|---|
| Dropped lines | Lines that never got a caption |
| Name and term accuracy | Share of glossary terms and Bible names in their canonical form |
| Translation errors per 100 lines | Blind review against the reference texts, counting wrong meaning, wrong name, omission and made-up content |
| Speech accuracy | Word error rate (character error rate for Chinese, Japanese and Thai), before and after cleaning |
| Caption delay | End of sentence to caption on screen, median and 95th percentile, measured from Singapore |
| Cost per service hour | One hour with five caption languages, at list prices |
Fairness rules
- Every single engine is set up the way its own guide recommends, with the same glossary loaded.
- Scorers never see which setup produced an output.
- Spark's results come from a pinned Spark version, tested on the same day as everything else.
- We publish the method, the clips we have rights to share, the raw outputs and our scripts, and re-run every quarter.