Rust + TypeSafe Jev judgment
Sift synthetic and pretraining datasets with Jev judgment
jev-curate streams Parquet and JSONL through a Jev evaluation pipeline, keeps rows that pass your rubric, and writes the rejects to a second file. Measured numbers in the benchmarks, no invented speeds.
Output unmetered. About $4.20 per 100M input tokens.
$ jev-curate --input corpus.jsonl --preset reasoning-math --concurrency 32 --dry-run
[info] preset: reasoning-math
[info] concurrency: 32
[info] dry-run: no rows sent, no credits spent
[info] would keep 812,044 of 1,000,000 rows
[info] would reject 187,956 rows (18.8%)LLM judges are an API call with a prompt bolted on
You paste a csv into a chat window and get grades back in prose. jev-curate treats judging as a pipeline: predefined rubrics, batched questions, typed answers, two output files ready for training.
- Judgment by TypeSafe Jev. Machine-native Choice, Score and Noul questions. One state ingest per micro-batch, questions fanned into one request.
- No token waste. Blank and padding lines drop in Rust before the API. 8k token ceiling per row.
- Parquet in, Parquet out. Streaming Arrow/Parquet reader and writer. JSONL too. No pandas step in the middle.
- Safety rails built in. Token bucket rate limiter with 429 backoff. Dry run verifies the pipeline without spending a credit.
- Zero runtime dependencies. Tests run against an in-process mock server. Curate on a laptop with a Rust binary.
Rubrics you can point at a file
Three presets ship with the binary. Each is a Jev evaluation template, not a regex or a rewriter.
reasoning-math
Grades chain-of-thought correctness and arithmetic soundness. Keeps steps that actually entail the answer.
keepanti-sycophancy
Flags flattery that tracks the user instead of evidence. Common in RLHF and chat fine-tuning sets.
flagcode-correctness
Detects syntax breakage, hallucinated APIs and logic slips in code generation outputs.
keepOne command in, two files out
$ cargo install jev-curate
$ jev-curate --input corpus.parquet --preset anti-sycophancy --out ./curated/The binary writes clean.jsonl with accepted rows and rejected.jsonl with the rest, plus structural stats. Run --dry-run first to see keep and reject counts before any row touches the API. Requires only a TYPESAFE_API_KEY.
Run it in six steps
- Install the binary with
cargo install jev-curateor grab a release build. - Export a
TYPESAFE_API_KEYor run without one in dry-run mode. - Point the CLI at a
.jsonlor.parquetfile with-i. - Pick a preset with
-p: reasoning-math, anti-sycophancy or code-correctness. - Check what would change with
--dry-runbefore spending a token. - Run for real and read
clean.jsonlandrejected.jsonlfrom the output folder.
| Flag | Default | What it does |
|---|---|---|
-i, --input | required | .jsonl or .parquet input path |
-p, --preset | reasoning-math | rubric to evaluate against |
-o, --out | ./curated/ | output directory |
-c, --concurrency | 32 | parallel worker count |
--dry-run | off | count before scoring; no key needed |
Judgment at pipeline cost
1500+
rows per second, per worker target
$0.042
per million input tokens
free
output tokens, unmetered
444.6x
cheaper, TypeSafe System One workflow benchmark
Rates from the TypeSafe AI model docs; 444.6x figure is TypeSafe's own System One benchmark. 1500+ is the project target, benchmark runs live in the repo.