Finding 2,442 Needles: Jev 1.14 on a Slab Shoring Plan

A small benchmark: can a vision model find every “150” joist-spacing label on one construction sheet? We tested Jev 1.14 on a dense slab shoring plan, measured accuracy against exact PDF text-layer ground truth, and tracked the time and cost. The result: the model processed the full sheet in 31 seconds for $0.0927, but its individual label locations were not accurate enough for takeoff.

Benchmark headline metrics from the report.

At a glance

• 2,442 “150” labels in the ground truth.
• 588 image tiles and 18,816 questions processed.
• 31 seconds wall-clock time with 12 requests running concurrently.
• $0.0927 total cost.
• At the 0.5 threshold, 799 labels located correctly (32.7% recall), alongside 2,308 false boxes.

Jev counted the labels in a tile exactly right on 69% of tiles, but it placed only about a third of the individual labels. Speed and cost are promising; location accuracy is not yet suitable for takeoff.

Why this is a hard test

The test sheet is BT-S01, Level 01 Zone X Slab Shoring Plan, a 48 × 36 inch vector PDF containing 9,106 drawn objects. Joist labels give spacing in millimetres. Across the sheet are 4,752 similar-looking labels: 2,442 reading “150” (the target), 1,122 reading “180,” 1,100 reading “120,” and 88 reading “90.”

Each target label is only about 0.10 × 0.13 inches on paper. At Jev’s 1,568-pixel image limit, a full-sheet view renders a label at roughly 3 × 4 pixels, so the plan had to be split into hundreds of zoomed tiles. The labels are rotated 90 degrees, overlap linework, and sit beside near-identical decoys. Box returns one box per question and has no “not found” answer, making it necessary to ask thousands of questions and determine which responses to trust.

The whole sheet. At this size, every target label is effectively invisible.

Method

Ground truth

The PDF contains a text layer. We used its exact text spans reading “150” to establish the position of all 2,442 targets. Jev did not receive this information; it was used only to score the results.

Tiling and questions

The plan was divided into 588 overlapping tiles, each rendered at 1,568 pixels wide (about 540 dpi). The grid was not aligned to the labels, but the overlap ensured each label appeared whole in at least one tile. Each tile received 32 questions: 16 yes/no count questions (“Are there at least k separate text labels reading exactly 150?”) and 16 box requests for the kth label, counted left to right along the top row and then down the sheet.

The 588-tile grid over the plan.

At working resolution, one tile shows the tiny repeated targets clearly enough to test whether the model can count and locate them:

One tile at working resolution: 11 “150” labels along one joist row. The full sheet has about 220 rows like this.

A box was retained when its corresponding count answer met the selected confidence threshold. A box counted as a hit when it contained the centre of a true label and was no larger than about three labels. Each label could be counted once; other retained boxes were false boxes.

Getting images to Jev

On October 14, TypeSafe announced image support for Jev 1.14 as a preview feature. We tested it on a difficult construction drawing to see whether the model could find tiny, repeated spacing labels.

• Through OpenRouter, images had to be sent inline in the request state as a data URL. Sending a TypeSafe asset reference returned a successful request, but the model effectively answered blind; multipart uploads were rejected.
• Image dimensions were limited to 280–1,568 pixels per side. Larger images were rejected rather than resized.
• A working 800 × 600 image used about 1,600 tokens; drawing tiles used roughly 2,000–5,000 tokens.

Results: fast, inexpensive, and weak at placement

At the default 0.5 threshold, Jev correctly located 799 labels, missed 1,643, and produced 2,308 false boxes. Raising the threshold reduced false boxes faster than it reduced correct detections, but recall never rose above about one-third.

• Threshold 0.3: 4,190 boxes retained; 804 correct; 32.9% recall; 2,893 false boxes; 19.2% precision.
• Threshold 0.5: 3,571 boxes retained; 799 correct; 32.7% recall; 2,308 false boxes; 22.4% precision.
• Threshold 0.7: 3,168 boxes retained; 785 correct; 32.1% recall; 1,965 false boxes; 24.8% precision.
• Threshold 0.9: 2,278 boxes retained; 671 correct; 27.5% recall; 1,324 false boxes; 29.5% precision.

Every true label on the sheet: found and missed detections are spread evenly across the plan.

Close-up of the top-left bays: green marks found labels, red marks missed labels, and orange outlines show false boxes.

Counting is better than locating

The yes/no “at least k?” answers gave the exact count of fully visible labels on 406 of 588 tiles (69%). The difficulty was identifying which label was, for example, the fifth one. Later boxes drifted sideways or expanded to cover a row. The benchmark’s visual review showed hits and misses distributed throughout the plan, rather than concentrated in one easy or difficult region.

Cost and speed

• Tiles (requests): 588
• Questions answered: 18,816
• Input tokens: 2,207,202
• Output tokens: 330,456 (free on Jev)
• Cost per tile: $0.00016
• Total cost: $0.0927
• Wall-clock time: 31 seconds (12 concurrent requests)

Jev reads each image once and answers all 32 questions for that tile in parallel. At this price, it is practical to ask multiple questions of each tile and cross-check the answers—a useful direction for future experiments.

What the benchmark tells us

• Fast and cheap at sheet scale: 18,816 questions in 31 seconds for about nine cents.
• Promising on large, isolated features: warm-up tests found openings, tower squares, plan extent, title block, and sheet type. Some boxes were shifted or incomplete, so these results are not a general accuracy guarantee.
• Good at counting repeated small labels within a tile: exact counts on 69% of tiles.
• Not reliable yet for locating many tiny, near-identical labels: 32.7% recall at the default threshold, with more false boxes than true hits.
• For this vector sheet, the PDF text layer is the better tool: it returns all 2,442 target locations directly. Vision is more relevant for scanned sheets without usable text layers.

Next experiments

We would test smaller tiles containing only three or four labels, since the first boxes on a tile were generally the most accurate; request labels by joist line rather than by reading order; and use the count answers to validate the box answers.

Benchmark note: Jev 1.14 (served as jev-1.14-20261014) was accessed through OpenRouter’s /api/v1/systemone endpoint, with images enabled using the TypeSafe-Danger-Zone-Images header. Box and Images were marked “Dreamer-ready” preview features by TypeSafe at the time of testing, so results may change between releases. Source drawing: “Benchmark Test – BT-S01 – L1 Zone X Slab Shoring.pdf,” one page. Ground truth: PDF text spans reading exactly “150.”