Multivocab teaches 15 languages side by side, and AI wrote all of it. Here is how I validate 36,000 language entries without reading every one of them by hand.

I built Multivocab because I wanted to study languages next to each other instead of one after another. Every card holds the same concept in every language: English, German, Spanish, French, Italian, Portuguese, Russian, Persian, Urdu, Hebrew, Turkish, Dutch, Swedish, Japanese, and Korean. Open a card in two languages and the two versions sit right next to each other.

That is the whole product, and it is also the hard part. If the German version of a card means something slightly different from the English version, the side-by-side comparison teaches the wrong thing instead of the right thing.

In this post I will write about why AI translation fails quietly, how I turned one card into twelve small typed questions, what my first run got wrong, and how the output turned into a fix list.

Why AI Translation Fails Quietly

The content was generated, so I expected some noise. What I did not expect was how ordinary the mistakes looked.

• Word-by-word translation. "Good morning" becomes bon matin in French instead of bonjour. Grammatical, understandable, and not what anyone says.
• The wrong sense. English words are often ambiguous, and a translation can be a real word for a completely different meaning of the English term.
• Drift across languages. When one language says "my brother" and another says "my sister", a learner comparing them learns the wrong thing.
• Misleading quiz options. A distractor that is also a correct answer punishes the learner for being right.
• Wrong notes and grammar. An incorrect article, gender, or explanation gets memorized, because that is exactly what a note is for.

None of these are schema errors. A JSON schema cannot see them, and reading 36,000 entries by hand is not realistic.

Why Not Just Ask a Chat Model

The obvious approach is to ask a chat model whether a translation is good. That returns prose. The wording changes between runs, there is no stable value, and there is nothing to sort or filter.

I wanted every check to end in a number, so I could sort by it, threshold it, and diff it between runs.

I used Jev, the decision model from TypeSafe. It does not generate text. You send it state plus typed questions, and it returns typed answers. Two of the three question types were enough for this work:

• Noul is a yes or no question. The answer is the probability that the answer is yes, from 0 to 1.
• Choice picks one option from a defined list. The answer is the winning option plus the probability for every option.

So one card review becomes twelve small judgments instead of one long opinion.

One Card Becomes Twelve Questions

For every card, in every language, I send one request. The state holds the English card as the reference meaning, the card I am checking, and the same term in the other 14 languages. Every question is evaluated in parallel against that same state.

{
  "model": "jev-latest",
  "state": "Reference (English): term 'good morning', example 'Good morning!', note 'A greeting before the lesson starts.' | Check (de): term 'Guten Morgen', example 'Guten Morgen!', note 'Eine Begrüßung vor dem Unterricht.' | Other languages for 'good morning': ...",
  "questions": {
    "term_relation": {
      "type": "choice",
      "instructions": "How does the German term relate to the English term in this sense?",
      "criteria": {
        "equivalent": "The same meaning as the English term in this sense.",
        "near": "A common alternative a learner would also accept.",
        "literal": "A word-by-word rendering that nobody would say.",
        "other_sense": "A real word, but a different sense of the English term.",
        "wrong": "Not related to the English term at all."
      }
    },
    "example_relation": {
      "type": "choice",
      "instructions": "Does the German example mean the same as the English example?",
      "criteria": {
        "same": "A learner sees the same meaning in both languages.",
        "shifted": "The meaning moved, for example a different time, person, or object.",
        "different": "The example says something else."
      }
    },
    "example_natural": {
      "type": "noul",
      "instructions": "Would a native German speaker find this example sentence unnatural?"
    },
    "term_used_correctly": {
      "type": "noul",
      "instructions": "Does the example use the term in the intended sense?"
    },
    "note_error": {
      "type": "noul",
      "instructions": "Does the note for this card contain an error?"
    },
    "article_gender": {
      "type": "noul",
      "instructions": "Is the article or the gender wrong for this term?"
    }
  }
}

Related words are checked the same way, but with one Noul per word, because each word needs its own answer. A card has up to six of them.

Test questions get the same treatment. Four questions each: does the answer fulfill the prompt, is it natural, does the prompt match the English prompt, and could any of the distractors also produce a correct answer.

The English card is the anchor. Every language is judged against the same English sense and the same English example. That is exactly what matters when someone studies two languages at the same time.

The Scale

• 2,170 concepts across 60 decks, plus 240 test questions across 12 test decks.
• 32,550 card requests and 3,600 test question requests.
• About 405,000 individual judgments, twelve per card and four per test question.
• A few dollars in total, because Jev only charges for input tokens.
• About half an hour of wall clock time, which is roughly the rate limit: 1,200 requests per minute.

Results are cached by content hash, so a re-run only checks the content that actually changed.

Output Built for Fixing

Every flagged problem is written as one self-contained JSON line. The line has the source file, deck, concept, language, field, what is wrong, the probability, the current card, and the English reference.

{"file":"decks/first-communication/concepts.json","deck":"First Communication","concept":"good morning","language":"ja","field":"relatedWords","issue":"Literal rendering of the English phrase. A Japanese learner would not meet this in real conversation.","probability":0.94,"card":{"term":"よい朝を","example":"先生、よい朝を！"},"reference":{"term":"good morning","example":"Good morning!"}}

Sorted by probability, that file goes straight into another LLM that proposes a fix and applies it. One issue at a time, with everything it needs in the same line. I do not have to reconstruct the context of a flag by hand, and the model does not have to guess which card it is looking at.

The First Run Raised 101 Flags Because of One Bad Question

This is the part I would keep from the whole project.

My first sample was 50 records. It produced 101 flags. That is more than two flags per record, and I did not write two errors into every card on purpose.

Ninety-four of those flags came from a single check. I had asked whether each related word was also a correct answer to the English term. But the app shows those words as related words, where a synonym is exactly what I want. The check was precise about the wrong thing, and it was confident about it too.

After I rewrote the question to ask whether each related word is really related, natural, and not a repeat of the term itself, the same 50 records dropped to 7 flags. All 7 were real.

A check has to match how the product actually uses the content. That is the lesson, and it cost me a full sample run to learn.

Results for the First Deck

Deck 1 is called First Communication. It has 45 concepts in 15 languages, which is 675 records.

• The full deck raised 50 flags. 36 were real and I fixed them. The examples had drifted from the English meaning, with "choose German" instead of "choose the first lesson". One Spanish sentence was ungrammatical. Some related words were unnatural, like the literal Japanese よい朝を for "have a good morning".
• 14 flags I reviewed and kept. Some were intentional localization, like the English example "How do you say this in Spanish?". Some were natural colloquialisms that the model simply did not like. One was a real gap in my content: Korean has no everyday phrase for "good afternoon".
• The re-run after the fixes was cheap. Only 33 of the 675 records had changed, so only those needed a new request. Everything else came from the cache.

Each run appends its numbers to stats.jsonl, and summary.json tracks coverage and lifetime totals. Between them I can always see how many of the 36,000 records are actually tested, how many checks came back, and what the run cost.

Caveats

• Jev is strongest in English. Other languages are supported, but not equally well, so I validate one deck at a time and read every flag before I change content.
• A probability is a judgment, not proof. I sort by confidence and re-check each flag after a fix, because a fix can introduce a new problem somewhere else in the card.

Final Thoughts

AI-generated content needs AI-scale validation, but asking a chatbot is not a test.

Breaking the work into small typed questions gave me something that behaves like a test suite. Every entry gets checked, every result is a number, and the output is a clean list of what to fix.

What I did not expect was the first sample. 101 flags on 50 records sounded like a broken model, but the model was fine and my question was wrong. That is the failure mode I will watch for in any AI check I build next: a confident, precise answer about something the product never asked for.