Source: README claims neither side can earn
jev-curate vs judging by chat prompt
Both workflows call a language model to grade data. The difference is what comes back: a prose paragraph you read and act on, or a typed verdict the pipeline acts on. This page compares them on the axes that matter at dataset scale.
chat: "This step is mostly sound but could
be clearer about the chain rule
application..." (0 keep/file)
curate: {"has_circular_reasoning": noul 0.05,
"reasoning_depth": score 4.8} -> kept
Side by side
| Axis | Prompt in a chat window | jev-curate |
|---|---|---|
| Verdict form | Free text you re-parse | Choice, Score or Noul typed answer |
| Row coverage | What you have patience for | Every row, streaming |
| Verbatim data | Copied, pasted, often rewritten | Passed as state, never rewritten |
| Output | Chat log | clean.jsonl and rejected.jsonl |
| Cost shape | Prose billed per token | 0.042 USD per million input, output unmetered |
| Repeats | Same prompt retyped per row | Preset rubrics, same questions each row |
| Check before spending | None | --dry-run reports keep and reject counts |
The chat flow is the incumbent workflow for most dataset work, and it is the right tool for the first dozen rows you inspect on screen. The break point is anything you plan to train on: at that size the cost is in the reshaping, not in the model call.
The shorter path is the pipeline
A chat judgment is an event. jev-curate makes the judgment a table: same rubric per row, same questions, typed answers written next to a keep or reject decision. When a reviewer asks why a row was dropped, the answer is an object in rejected.jsonl, not a scrollback.
That also makes the tool auditable. The three presets map to three evaluation templates, and each template asks Jev only what that job needs: math presets check entailment, anti-sycophancy checks rated honesty, code presets check correctness. No generic grade, no rewriter, no taxonomy drift between rows.
When to keep using chat
- Exploring an unfamiliar dataset for the first time. Look at twenty rows by eye before you set a rubric.
- Ad hoc questions that change every row. Presets assume a stable rubric across a batch.
- Writing prose feedback as the deliverable. jev-curate returns verdicts, not paragraphs.
When the pipeline pays for itself
The crossover is not about model quality. It is about repetition. A chat grader re-reads the prompt on every row and re-emits the justification on every row, and the grader's attention is the same scarce resource each time. jev-curate evaluates the rubric once as questions, then replays it across the batch with zero per-row prompt authoring and zero prose output to eyeball.
Teams that keep chat judging past that crossover are usually not paying in tokens, they are paying in drift: the prompt changes slightly between rows, the keep threshold shifts mid-batch, and nobody can rerun yesterday's filter. Presets and --dry-run make the filter a checked-in artifact. The same command that filtered the training set filters next month's batch.
None of this argues that chat judging is useless. It argues that it is a different tool, for inspection before you commit to a rubric. The rows you save as a rubric example in a chat thread become the rows you route with a preset.
The second comparison
Hand-rolled judge scripts and manual labeling are the other incumbent. The versioned comparison walks through what you write, what you maintain, and where presets and typed verdicts change the math.