Every open source project has a queue of issues nobody has sorted. Someone has to read each one and decide what it is: a bug, a feature request, a question, a docs problem. Then they decide whether an engineer could start on it as written. It is PM work, and nobody has time for it.
In September 2026 TypeSafe launched Jev, a small model built for exactly this kind of decision. The launch claimed up to 193x the speed of the big general models. I wanted to know whether that held on real work, not on the vendor's own benchmark.
I pulled 1,895 open issues from five projects: n8n, Supabase, Cal.com, Appwrite and Home Assistant. Each issue got the same nine questions: what kind of issue it is, how severe it is, and seven yes or no checks such as whether it has steps to reproduce, which version or environment it happened on, what the author expected and whether there is a workaround. That is 17,055 decisions per model.
Jev and Claude Sonnet 5 answered the same questions on the same issues. To check who was right, I used the 767 issues where the maintainers had already labelled the issue type themselves. Their label is the ground truth, not either model's opinion.
Both models got exactly the same text. The nine questions and their one line criteria are public, so anyone can rerun the test: the questions on GitHub
| Measure | Jev | Sonnet 5 |
|---|---|---|
| Issue type matches the maintainers' label (767 issues) | 96.1% | 94.0% |
| Median time per call | 0.82 s | 1.49 s |
| All 1,895 issues, 128 at a time | 12.9 s | 28.9 s |
| Cost for all 1,895 issues | $0.11 | $18.36 |
The speed claim did not hold. Per call, Jev was about 2x faster, not 193x. The real story is cost: 160x cheaper, at an accuracy gap I would call a tie.
The two models agree on issue type 97% of the time. They agree on severity only 59% of the time, and on "can an engineer start on this" 65% of the time. The more a question bundles judgment, the less two good models agree with each other. That is where a human still belongs.
This is the part I care about most, because a confident number on a question nobody checked is worse than no number.
- Issue type: validated. 96.1% against labels the maintainers set themselves.
- Version check: validated. Against a pattern match for a real version string, it caught 93% of issues that had one and correctly cleared 82% of those that did not.
- "Can be started as written": not validated. Against the projects' own triage labels its score was barely better than a coin flip. Against what actually happened next, where the model judged the first human reply, it was only slightly better. Only 10% of issues ever got a reply asking for more information, so incomplete reports are not the queue's real problem.
One label that looked like ground truth was not. n8n's "Needs Feedback" label sits on bug reports that already carry exact versions and commit hashes. It does not mean the reporter left something out. It means the team wants input from another team. Had I scored the models against it, I would have been measuring the wrong thing.
After the post went out, new models kept launching with the same promise, most of them with a benchmark they won. I ran them on the same 767 labelled issues.
| Model | Issue type | Speed | Cost |
|---|---|---|---|
| Jev | 96.1% | 0.82 s per call | $0.11 for 1,895 |
| GLiDE (Fastino) | 95.8% | 7.1 s per call | about 50x Jev |
| Sonnet 5 | 94.0% | 1.49 s per call | $18.36 for 1,895 |
| Laya, on my laptop | 94.0% | 32 minutes for 1,895 | free |
GLiDE launched with a benchmark lead over Jev. On real issues it tied Jev and was about 9x slower per call. Laya matched Sonnet on issue type for free, but fell well behind on the yes or no questions. A third, Needle 3, scored 54.8% before any tuning, below the 62.6% you get by calling every issue a bug.
None of the benchmark leads showed up on real issues.
I posted the first result on LinkedIn with a video of the run. It reached 151,237 people and passed 218,000 impressions, and drew 162 comments. Engineers argued about the method in the comments, which was the best feedback I got. Testing Laya was a promise I made in that thread.
Limits, stated up front: Sonnet's 28.9 seconds is its real per call times scheduled 128 at a time, not a live race, and its cost is list price. Ground truth exists for issue type only, on 767 of the 1,895 issues. Apart from the version check, the yes or no answers have no ground truth, only agreement between models. Each model ran once.
A launch benchmark tells you what the vendor optimised for. Your own queue tells you what you need. They do not have to overlap, and the only way to find out is to run the test on work you can check.
The harder lesson was about my own measurements. The label I almost used as ground truth meant something else, and the question I most wanted to answer, whether an issue can be started as written, is the one I could not validate. Deciding what counts as a right answer was more work than running the models.