Skip to main content
SPARK BENCH · FULL RESULTS IN PROGRESS

We'll publish our numbers. Here's how.

Spark Bench tests Pentecost Spark against single AI engines on the same church recordings. Every result is shown per language, the worst language comes first, and the method and raw outputs are published.

Why the worst language comes first

A single engine rarely fails on average. It fails in one language, or on names, and an average hides that. A congregation that reads Cantonese doesn't care that the English captions were fine. So every Spark Bench result is reported per language, and the headline is the lowest one.

Early internal test, October 2026

Single engines against Spark's route

The cleaner stage on 282 lines in eight languages. A line counts as dropped when no caption came back within 6 seconds.

Setup Lines dropped Worst language Cost per 1,000 lines
GPT-6 Luna alone12 of 282Cantonese: 35% dropped$0.49
Qwen 3.8 Flash alone12 of 282English: 20% dropped$0.16
DeepSeek V4.1 Flash alone11 of 282Thai: 18% dropped$0.19
GPT-6 Sol alone1 of 282English: 3% dropped$2.78
Pentecost Spark route0 of 282None$0.51

One run, text input, costs at October 2026 list prices. On line-by-line translation quality the engines scored close to each other in a blind review. In the same test, a pack of Bible names lifted names in the right form from 91% to 98% on the lowest-cost engine.

Spark Bench v1: the method

What we test against

GroupWhat it isWhat it shows
Single engines, as soldOne vendor's speech to text plus its machine translation, with its own custom-vocabulary feature: Azure AI Speech translation, Google Cloud Speech-to-Text with Cloud Translation, OpenAI speech to text with one translation modelWhat you get from a single-engine setup
Single engines inside PentecostOur cleaner with one model for every languageWhat per-language routing adds
Spark with one feature offNo cleaner, no glossary and Bible texts, no manuscript, or no confidence routing, one at a timeWhat each feature is worth

The test set

  • Mandarin and Cantonese speakers; English services captioned into Traditional and Simplified Chinese, Japanese, Korean and Thai; then Tagalog and Indonesian.
  • Two or three speakers per language, 10-minute clips from the soundboard feed, recorded with each speaker's consent.
  • Each clip comes with its manuscript and glossary, as a real service would, and includes quoted scripture and Bible names.
  • Reference texts from each speaker's manuscript and the published Bible translations they quote.

What we measure

MeasureHow
Dropped linesLines that never got a caption
Name and term accuracyShare of glossary terms and Bible names in their canonical form
Translation errors per 100 linesBlind review against the reference texts, counting wrong meaning, wrong name, omission and made-up content
Speech accuracyWord error rate (character error rate for Chinese, Japanese and Thai), before and after cleaning
Caption delayEnd of sentence to caption on screen, median and 95th percentile, measured from Singapore
Cost per service hourOne hour with five caption languages, at list prices

Fairness rules

  • Every single engine is set up the way its own guide recommends, with the same glossary loaded.
  • Scorers never see which setup produced an output.
  • Spark's results come from a pinned Spark version, tested on the same day as everything else.
  • We publish the method, the clips we have rights to share, the raw outputs and our scripts, and re-run every quarter.

What Spark does to every line →