The build-or-buy question
jev-curate vs custom judge scripts and manual labeling
Most teams filter training data with a script they wrote: a prompt template, a loop, a JSON blob of grades. This page compares that approach to the preset pipeline, and says plainly when the hand-rolled script is the right answer.
custom: for row in rows:
prompt = template.format(row)
text = llm(prompt)
grade = parse(text) # fragile
curate: jev-curate -p reasoning-math -i set.jsonl --dry-run
Side by side
| Axis | Hand-rolled judge script | jev-curate |
|---|---|---|
| Rubric | Prompt template you maintain | Checked-in preset, same questions every row |
| Verdict parsing | You write the parser | Typed Choice, Score, Noul answers |
| Check before spend | Only by reading code | --dry-run reports keep and reject counts |
| Row streaming | You handle batches and resume | Built-in streaming reader, Parquet and JSONL |
| Throttling | Your sleep loop or library | Adaptive token bucket matching provider limits |
| Output shape | Whatever you designed | clean.jsonl and rejected.jsonl with verdicts |
| Cost of trying | Days | Minutes to a dry run |
The honest difference is where the complexity lives. A judge script concentrates all the risk in code you own: prompt drift, parse failures, missing resume logic, throttling bugs. jev-curate moves that risk into a binary with tests, typed outputs, and a dry run that tells you the keep count before a single token is spent.
When the hand-rolled script still wins
Three cases. First, exotic input formats: if your data lives behind an API, in a private store, or as custom objects, the streaming reader does not know it. Wrap the rows to JSONL first, then sift. Second, rubrics that change per row: presets assume a stable judgment across a batch. Third, non-Jev judges: if your team is committed to a specific frontier model for judging, jev-curate is not the wrapper for that model; it is the wrapper for Jev.
The middle ground is the common one: route with jev-curate, escalate the edge cases to a heavier judge. The Choice answers exist for exactly that, a fast typed decision on which rows deserve the expensive pass.
Manual labeling, scored honestly
Against human annotation, the tool does not compete on quality of judgment. It competes on coverage and consistency. A labeler reads a row, judges with one model of quality, and drifts with fatigue. The preset asks the same questions to every row with the same thresholds, and the dry run shows the drop rate before the batch starts. The workflow that pays is presets for the wide pass, humans for the narrow slice the presets cannot agree on.