Auto-optimizing Jev: half the errors, 1/7 the cost

What's autoresearch

Autoresearch is a research loop that runs itself: a coding agent pointed at a benchmark and a budget. It proposes changes, tests them cheaply, measures the survivors, keeps what holds up, and carries what it learned into the next round - the experiment cycle a researcher would run, executed end to end by software.

Two things about it matter for what follows. It trains nothing: the loop optimizes the world around a model - the code that decides what the model sees before it is asked. And it is built to measure: a change counts only when it survives scoring, not when it sounds convincing. This post is one run of that loop on one real task.

What's Jev

TypeSafe recently released Jev, the first of what they call System One models. We handed it to our autoresearch loop with a benchmark and a budget and let it work.

Jev is a classifier, and it will classify almost anything. You give it three things: state, the material to judge; instructions, the question you want answered; and criteria, the options it is allowed to answer with.

It returns one of those options, a probability for each of them, and a confidence.

Jev is a general-purpose classifier: it can answer a closed-set task without task-specific examples or training.

It generates no text. So there is no chain of thought to read, nothing to fine-tune, and no way to ask it to think step by step. What you can change is the request, and what you put in front of it.

The task

Converting Chinese characters to pinyin is a "grapheme-to-phoneme" task: turning written symbols into their pronunciation. Most characters have exactly one pronunciation and can be handled with a simple table lookup. A few hundred - the polyphonic characters - have several, and the correct reading depends on the word containing the character or, when it stands alone, on its grammatical role. For example: 还 is hái when it means still, huán when it means to give back.

Some popular published models for this, g2pM and g2pW, have to be hosted: g2pW is a 606 MB model that takes 6.7 seconds a sentence on CPU.

LLM baseline

A frontier model can instead work sequence-to-sequence over a whole sentence, or classify each ambiguous character separately. Neither approach needs task-specific training or a model you host yourself.

On our benchmark - the 61,750 positions that need a decision, drawn from 1,085 human-edited texts (details below) - DeepSeek V4.1 Flash with reasoning on, used over the whole sentence, makes 285 mistakes, at $11 a pass over the corpus and eight seconds a sentence. It only aligned its output consistently in reasoning mode; without reasoning, the predicted readings were frequently assigned to the wrong characters. That is the baseline for the comparisons below.

Jev baseline

Our first Jev request was the obvious one: the sentence with every polyphonic character tagged, each reading described by its dictionary senses, one line of instruction. It made 316 mistakes against 1,434 for a plain phrase-table lookup (pypinyin), for $1.22 a pass - a ninth of the reasoning model's price, a thirtieth of its latency. But 316 against 285 is decent, not a win: a replacement has to beat the thing it replaces. The gap - 31 mistakes - was the loop's job.

Every number in this post is a count of mistakes on the 61,750 positions that need a decision, not an accuracy - accuracy is a poor scoreboard here.

Handcrafting experiments is too slow

Unlike a traditional classifier, Jev needs no training. We could therefore use a separate configuration for each of the 341 character types, rather than train a separate model for each one.

To start, we wrote better requests by hand - ordered procedures, part-of-speech rules, worked examples. Occasionally something helped the sentences it was written for; usually it did nothing for the character's other samples on the split, or made them worse by distracting the model. Unlike a reasoning model, Jev did not reliably follow a sequence of instruction steps or reason through indirect rules. Saying what the distinction is moves the number; saying what to think about and what order to think in, like a recipe, does not.

Manual work was too slow for exploring many configuration combinations and testing them well. Jev was newly released, and each idea required a written variant, a paid pass over the corpus, and a read-through.

Auto-optimizing Jev

We pointed Kiln's auto-optimizer at the problem. Importantly, we targeted it at the harness; not the model, and not only the wording of the request, but all the code that decides what reaches the model. It had a small set of pre-approved libraries to build its harness from. They included jieba for word segmentation and access to CC-CEDICT (an English-Chinese dictionary) lookups for words covering the marked character and the readings those words provide.

We asked it to work one character at a time, twelve in all. Nothing couples the requests to each other, so each character is a small problem on its own terms: a closed set of answers, its own sentences, and a score that moves independently of every other character. Several can be poked at in parallel, and whatever survives on one gets tried on the next.

The probe step is the cheap one, and it does most of the filtering. A probe is one yes-or-no question about a mechanism - does the segmenter actually keep this word together, does the dictionary have anything to offer here at all - answered from a handful of sentences, with no scoring run behind it. This ruled out many ideas before we built and measured them across the full split.

What survives gets built. The loop then clusters the remaining failures and infers their shared cause before choosing the next change. Take 得, a very common character that can be read either de or dé. Its leftover mistakes looked like unrelated sentences, but the loop found a single cause: the correct reading depends on whether the character stands alone between words or sits buried inside a longer word, and the harness never checked which.

The harness needed to identify whether the character was part of a word or stood alone before asking Jev to choose a reading.

Every proposed change is evaluated on that character's entire development half - we split each worked character's sentences in two, and keep the other half back for scoring. We reject it if it fixes its target cases but creates mistakes elsewhere, or if the improvement is smaller than about fifteen positions, the variation we saw across repeated calls. A surviving change is then scored once on the held-out half, which the loop does not access while working on that character.

Experiments

The loop's first instinct was the same as ours: write the semantics down. The rule for each reading, the constructions each one turns up in, when to prefer which. With a reasoning model this approach works, but Jev doesn't have reasoning. Then additional prompt engineering ideas: grammar labels for adjacent characters, English translations, and more context.

The autoresearch loop ran these experiments competently. None of it worked. The translations changed zero answers at roughly triple the tokens. The extra context landed inside the noise floor, and the whole article added to context cost 39% more input tokens for the same result. The grammar labels each helped on their own and cost sixteen to twenty hard cases the moment they were stacked with anything else.

A change that fixes its target character often breaks a different one, so no result can be trusted on its target cases alone. That is why every change was re-scored on the character's whole development half before it could be kept.

For Jev: show the evidence, not the steps

Multi-step instructions weren't helping, since Jev doesn't reason. Autoresearch eventually found a pattern that worked. It pivoted to case-specific hints, computed per sentence: segment the sentence, look up every dictionary word covering the marked character, note the reading each word gives and whether the segmenter keeps the word in one piece, then ask exactly the same question as before. The loop called that block word evidence.

The loop developed it on twelve characters, then we tested it unchanged on twenty other, unseen characters: it took 97 mistakes down to 11 and made none of them worse. Common combinations become lookups instead of grammar judgment calls.

This is not a dictionary lookup. pypinyin already uses a phrase table and still makes 1,434 mistakes on these positions - the harness supplies the candidate readings, and Jev makes the call the dictionary cannot.

Chaining Jev

Autoresearch then found a second optimization that worked: a Jev call is almost free and returns in under 300 ms, so the harness can afford more than one per decision. Instead of asking for the reading directly, it asks an easier question first - which word is this character part of? - and uses the answer to settle the reading. For one character (要), it went from 28 mistakes to 0 on its held-out sentences. It didn't help for most characters, so the loop updated the harness to make the second call only for characters where it had a measurable impact.

state        sentence: 业内人士认为，…不仅【为】行业提供了新思路…
instructions `sentence` is a Chinese sentence with one character
             wrapped in 【 】: 为. Which pinyin reading does that
             marked occurrence of 为 have?
criteria     wèi: wèi
             wéi: wéi

state        sentence: 业内人士认为，…不仅【为】行业提供了新思路…
instructions `sentence` is a Chinese sentence with one character
             wrapped in 【 】: 为. Which pinyin reading does that
             marked occurrence of 为 have?
criteria     wèi: preposition. It introduces who or what something
                  is done for, or the reason it happens. What
                  follows is a noun phrase, and the main verb comes
                  after that.
             wéi: verb. To be, to act as, to become, to count as,
                  linking one thing to what it is or turns into.

state        sentence: …交警知道了这件事，认【为】他闯红灯是为了救人…
             word: 认为 (为 is syllable 2 of 2)
             dictionary: 认为 = rèn wéi (为 is syllable 2 of 2;
                         the segmenter keeps this word together):
                         to believe; to think; to consider.
instructions … If the character sits inside a word the segmenter
             and the dictionary agree on, that word's reading is
             the answer, whatever the surrounding grammar
             suggests. Only when it stands on its own does the
             question become which part of speech it is.

def word_evidence(char, sentence, only_other_than="",
                   candidates=None):
    lo, hi, token = token_span(sentence)     # the segmenter's word
    index = sentence.index("【")
    lines = []
    for match in words_around(sentence):     # every dictionary word
        if only_other_than and match.reading == only_other_than:
            continue                  # 的 shows only exceptions
        begins = index - match.offset
        together = lo <= begins and begins + len(match.word) <= hi
        note = (
            "the segmenter keeps this word together"
            if together
            else "the segmenter splits this across two words, so it "
            "is probably a coincidence of characters rather than "
            "the word here"
        )
        if candidates and _unstressed_variant(match.reading, candidates):
            note += ("; the dictionary writes this syllable "
                     "unstressed, and the same word is often said "
                     "with the full tone instead")
        lines.append(
            f"{match.word} = {' '.join(match.syllables)} "
            f"({char} is syllable {match.offset + 1} of {len(match.word)}; "
            f"{note}): {match.gloss()}"
        )
    return {
        "word": (f"{token} ({char} is syllable "
                 f"{index - lo + 1} of {hi - lo})"),
        "dictionary": ("\n".join(lines[:4])
                       or "no dictionary word covers it"),
    }

Results

The optimized request makes half the mistakes of the reasoning model it replaces - 147 against 285 - for a seventh of its cost and a thirtieth of its latency. The unoptimized request was already a ninth of the price; the loop closed then surpassed the quality gap.

The whole campaign - every experiment, probe, ablation and corpus pass - cost about $18, less than two passes of the reasoning model it was measured against.

What solved this case

The final configuration is hard to find, hard to measure, and not intuitive. Hard to find: most reasonable ideas change nothing, and a few quietly make things worse somewhere else. Hard to measure: effects are local to one character, regressions turn up on others, and the gap between a real gain and noise is about fifteen positions. Not intuitive: the strategy that fixed almost everything was not a better sentence about the readings - it was a segmenter and a dictionary.

The prompt is only one part of the system, and not the part that mattered. What solved the task was the work around the whole harness: wiring up jieba for segmentation and CC-CEDICT for candidate words, checking whether the segmenter agrees a word is really there, and shaping all of it into the one form Jev judges best - a single closed choice among named options. The model does the one thing it is good at; the harness does everything else. That division of labor is what the loop found, and finding it is what the loop is for.

Harness optimization beats model optimization

Increasingly, the biggest wins come from optimizing the harness, not the model. Kiln offers many ways to optimize agents - fine-tuning, GEPA prompt optimization, model selection, hyperparameter tuning - but none of that matters if your harness is unoptimized.

Instead of tuning a prompt or skill to handle 50 parallel cases, branch into smaller, more focused decisions. Instead of optimizing prompts to help a model navigate 200 tools, build a smaller/better set of tools or add expert sub-agents. The two wins for this project - word evidence and chained model calls - both lived in the harness, not the model.

Autoresearch loops make harness optimization practical. They can drive hundreds of experiments autonomously. Jev's case was particularly interesting: the more typical methods used for LLMs failed, so the loop had to try a range of new ideas. The research model had no knowledge of Jev's quirks - Jev had only just been released - but it still found a solution.

Oh, and if you want to optimize your agent - please reach out (discord/founders)! We can apply these techniques and more to your use case.