{"schema_version":4,"intro":["No complete model design was printed (in round 3 Dan printed six roof sections cut from models' Estoa designs, to test the bridge check). Each design is a CadQuery program; the eval runs it, gets the 3D shape, and code measures that shape.","The measurements are compared with records of test prints that Dan, the eval's author, printed on his own printer (a Bambu Lab X1C, 0.4 mm nozzle, black high-speed PLA, 0.2 mm layers) and wrote down. A nut pocket is judged by what nuts did in the pockets Dan printed, a roof by which of his printed roofs came out clean, and so on.","Each task has a list of checks. A few run on every printed part (does the code run, is the part one piece, will its overhangs print); the rest are the task's own (does the bearing fit, does the bolt reach the camera).","A design passes only if every counted check passes; one failure fails it. Test prints the model adds get the every-part checks too (code runs, one piece, overhangs), but those results don't count. A test print can still count the other way, into a pass. Three checks below (the knob's nut fit, the wheel's bearing fit and the Estoa box's disk holes) pass a size the records reject or can't decide, when the design's own test print or template would have found the right size and a parameter applies it."],"claude":"Claude, the AI model that wrote much of this eval's code with Dan, made some of the scoring calls below where no print decided a case. By Dan's rule each one is labelled \"Claude's scoring call\" wherever it appears, so it's never mistaken for Dan's decision or for a recorded print.","states":{"pass":"The shape is inside what a recorded print showed working, or close enough to one to take its result (the nearest-print rule below), and the result says when it assumed.","fail":"The shape is where a recorded print failed, or plainly can't work (no hole for the bolt, a pocket the nut can't get into). A check whose own code crashed on the shape is also recorded as a fail, with the error as its finding.","not graded":"No recorded print decides this size or shape, so the code won't guess. Never counted as a pass; a design with a not-graded check and no failure is reported as not graded.","not reached":"The check had nothing to read because an earlier failure took away what it needs, usually code that didn't run. It isn't counted again. Not every pair of checks works this way; where one check still grades what another has failed, the section says so.","declined":"The model handed in no final part on purpose (for example only a test print). No check is counted.","no design":"The model never handed in a design (it ran out of turns or every reply stopped at the token limit)."},"denominators":"The counts in each section are of the counted first prints of the tasks the check applies to. That includes designs whose checks never ran, because the model declined or handed in no design; each section says how many there are. So a check's failures, passes and not-graded designs don't add up to the total.","nearest_print":"Where a size falls between two recorded prints, or just past the last one, the nearest recorded print within 0.2 mm decides (a tie goes to the tighter print), and the check says it assumed from that print (pbe/graders/assumed.py, nearest). Further than 0.2 mm from every print stays not graded.","checks":[{"anchor":"builds","check":"builds","task":"*","task_label":"Every printed part","title":"Code runs","tests":"Does the model's code actually produce a part?","how":["The eval runs each part's build(p) function in a separate, sandboxed Python process with a 120-second limit, with the design's own parameter values.","It exports whatever comes back as a mesh. If the code raises an error, times out, or exports something that can't be read, the part fails, and the error text is kept.","The task's checks that need that part are then not reached."],"record":"None needed. This is the one check that rests on no print.","not_graded":"Never; the code either produced a part or it didn't.","weak":"A design whose code didn't run says nothing about whether the model's idea would have worked. The crash replays on the site send each crashed design back once with its error.","inference":null,"code":[{"file":"pbe/graders/checks.py","function":"builds","line":71,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/checks.py#L71"},{"file":"pbe/parts/build.py","function":"build_parts","line":130,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/parts/build.py#L130"}],"photo":null,"counts":{"first_prints":466,"failed":39,"not_graded":0,"passed":410,"not_reached":0,"no_part_built":0,"declined":10,"no_design":7,"crashed":0,"via_test_print":0,"reasons":[{"key":"code-didn-t-run","label":"Code didn't run","kind":"failed","trials":39,"href":"./?reason=code-didn-t-run"}]},"examples_note":null,"examples":[{"outcome":"fail","label":null,"trial_id":"pilot-nut-knob/nut-knob/M5/can_ask/qwen__qwen3.8-flash/t1","trial_slug":"pilot-nut-knob--nut-knob--M5--can_ask--qwen__qwen3.8-flash--t1","href":"./?trial=pilot-nut-knob--nut-knob--M5--can_ask--qwen__qwen3.8-flash--t1","model_short":"Qwen 3.8 Flash","task":"nut-knob/M5","task_label":"Hand knob, M5","trial":1,"verdict":"fail","part":"knob","note":null,"state":"fail","detail":"NameError: name 'sqrt' is not defined","via_test_print":false,"view":null,"no_view_reason":"the code didn't run, so there is no shape to draw"}]},{"anchor":"bridges","check":"bridges","task":"*","task_label":"Every printed part","title":"Overhangs can be bridged","tests":"Will every overhang print, or will part of the roof come out as loose strands in mid-air?","how":["Stands the part the way its code builds it (the design's z axis is up, as it will print, with its lowest point on the bed) and cuts it into 0.2 mm layers from the bed up, the way the printer will lay it down.","In each layer, finds the area with nothing under it in the layer below. A region lying entirely within 1.5 mm of the layer below is an ordinary small overhang and is skipped.","Across every other region it lays parallel strands 0.4 mm apart (one nozzle width), trying a direction every 5 degrees. A strand counts as landing if both its ends are within 0.3 mm of the layer below, or if the whole strand lies within 1.5 mm of it (a stub at a corner that needs no bridge).","The region prints if, in some direction, at least 95% of strands land and none is longer than 50 mm; or if every strand is 6.8 mm or shorter and at least 24% land.","A hole cut through a wide roof fails, because strands end at the hole whichever way they run. That is the commonest failure in the eval. A small holed roof whose strands are all 6.8 mm or shorter can still pass by the short-strand rule.","A part the design's own instructions say to print with supports passes whatever its overhangs."],"record":"Dan's roof and bridge test prints (rounds 1 to 3, results/predictions/round1-coupons.yaml to round3b-roofs.yaml) and the first tripod plate he printed, whose roof over the bolt's hole tangled. Roofs up to 50 mm printed clean; plain bridges of 60, 80 and 100 mm all strung. The short-strand rule comes from two prints, a roof crop recorded as clean (44% landing, longest 6.8 mm) and ring roof 5, recorded as \"mostly ok; a few strands\" (24%, 6.0 mm). The limits are in catalog/printer.yaml.","not_graded":"A ledge wider than 1.5 mm but no wider than 2 mm that fails the strand test. The narrowest roof seen to tangle was a 2.0 mm ledge round a hole, and a 1.5 mm one printed, so anything in between isn't decided.","weak":"It models no cooling, speed or sag, only where strands can land. The short-strand rule rests on one print recorded as clean and one recorded as mostly ok, so it may pass a roof that prints with a few loose strands.","inference":"The short-strand rule, its 24% floor and counting a stub within 1.5 mm as landing are scoring calls (the first two Claude's, the last Dan's, 2026-11-13), not prints.","code":[{"file":"pbe/graders/bridges.py","function":"overhangs","line":169,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/bridges.py#L169"},{"file":"pbe/graders/bridges.py","function":"assess","line":131,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/bridges.py#L131"},{"file":"pbe/graders/checks.py","function":"bridges","line":103,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/checks.py#L103"}],"photo":{"url":"https://media.danoved.xyz/print-build-eval/nogit/51fb657f/test-print-bridge-tangle.jpg","thumb":"https://media.danoved.xyz/print-build-eval/site/1866fcd4/ref-test-print-bridge-tangle.jpg","width":1600,"height":2133,"kind":"test print","caption":"A roof over a hole, printed in the air: loose loops and tangles around the hole.","source":"README.md","reason":"holed-roof"},"counts":{"first_prints":466,"failed":126,"not_graded":4,"passed":290,"not_reached":0,"no_part_built":29,"declined":10,"no_design":7,"crashed":0,"via_test_print":0,"reasons":[{"key":"holed-roof","label":"A roof with the bolt's hole cut through it","kind":"failed","trials":105,"href":"./?reason=holed-roof"},{"key":"overhangs-that-won-t-bridge","label":"Overhangs that won't bridge","kind":"failed","trials":14,"href":"./?reason=overhangs-that-won-t-bridge"},{"key":"wide-roof","label":"A roof or ledge too wide to bridge","kind":"failed","trials":7,"href":"./?reason=wide-roof"},{"key":"a-roof-ledge-1-5-to-2-mm-wide","label":"A roof ledge 1.5 to 2 mm wide","kind":"not graded","trials":4,"href":"./?reason=a-roof-ledge-1-5-to-2-mm-wide"}]},"examples_note":null,"examples":[{"outcome":"fail","label":null,"trial_id":"pilot-tripod/tripod-plate/can_ask/anthropic__claude-haiku-5.5/t10","trial_slug":"pilot-tripod--tripod-plate--can_ask--anthropic__claude-haiku-5.5--t10","href":"./?trial=pilot-tripod--tripod-plate--can_ask--anthropic__claude-haiku-5.5--t10","model_short":"Haiku 5.5","task":"tripod-plate","task_label":"Tripod plate","trial":10,"verdict":"fail","part":"plate","note":null,"state":"fail","detail":"1 unbridgeable overhang(s); first at z = 4.4 mm, 157 mm²: best direction 40°, 39% of strands land on solid at both ends","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/7126f987/plate-bridges-fail.webp","caption":"Overhangs can be bridged (plate): fails. 1 unbridgeable overhang(s); first at z = 4.4 mm, 157 mm²: best direction 40°, 39% of strands land on solid at both ends (prints that came out clean had 95% of strands anchored, or strands no longer than 6.8 mm)","width":900,"height":977,"kind":"bridge-layer"},"no_view_reason":null,"drawing_note":"prints that came out clean had 95% of strands anchored, or strands no longer than 6.8 mm"},{"outcome":"pass","label":null,"trial_id":"pilot-tripod/tripod-plate/can_ask/google__gemini-4-argon/t3","trial_slug":"pilot-tripod--tripod-plate--can_ask--google__gemini-4-argon--t3","href":"./?trial=pilot-tripod--tripod-plate--can_ask--google__gemini-4-argon--t3","model_short":"Gemini 4 Argon","task":"tripod-plate","task_label":"Tripod plate","trial":3,"verdict":"pass","part":"qr_plate","note":null,"state":"pass","detail":"every overhang can be bridged","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/cc05c475/qr_plate-bridges-pass.webp","caption":"Overhangs can be bridged (qr_plate): passes. Every overhang can be bridged (the widest of 2 roof(s) the check assessed on this part; 100% of strands land on plastic at both ends, longest 20.0 mm)","width":900,"height":1017,"kind":"bridge-layer"},"no_view_reason":null,"drawing_note":"the widest of 2 roof(s) the check assessed on this part; 100% of strands land on plastic at both ends, longest 20.0 mm"}]},{"anchor":"connected_bodies","check":"connected_bodies","task":"*","task_label":"Every printed part","title":"Each part is one piece","tests":"Is each printed part one solid piece, not several that would fall apart or float in the air?","how":["Splits the part's mesh into its separate shells and counts those with positive volume. A sealed hollow inside a part shows up as an inside-out shell with negative volume, so it isn't counted.","A final part must be exactly one body.","A test print may be several pieces, such as three separate pockets, as long as every piece starts on the bed (its lowest point within 0.01 mm of the bed)."],"record":"None needed; this is geometry. It comes from the tripod build, where a hex peg floated free of the part it was meant to join.","not_graded":"Never.","weak":"It says nothing about strength. Two blocks joined by a hair-thin web are one body and pass.","inference":null,"code":[{"file":"pbe/graders/checks.py","function":"connected_bodies","line":40,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/checks.py#L40"}],"photo":null,"counts":{"first_prints":466,"failed":10,"not_graded":0,"passed":410,"not_reached":0,"no_part_built":29,"declined":10,"no_design":7,"crashed":0,"via_test_print":0,"reasons":[{"key":"a-part-in-pieces","label":"A part in pieces","kind":"failed","trials":10,"href":"./?reason=a-part-in-pieces"}]},"examples_note":null,"examples":[{"outcome":"fail","label":null,"trial_id":"pilot-nut-knob/nut-knob/M5/can_ask/google__gemini-3.8-flash/t4","trial_slug":"pilot-nut-knob--nut-knob--M5--can_ask--google__gemini-3.8-flash--t4","href":"./?trial=pilot-nut-knob--nut-knob--M5--can_ask--google__gemini-3.8-flash--t4","model_short":"Gemini 3.8 Flash","task":"nut-knob/M5","task_label":"Hand knob, M5","trial":4,"verdict":"fail","part":"hand_knob","note":null,"state":"fail","detail":"3 bodies","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/1f1e55e8/hand_knob-connected_bodies-fail.webp","caption":"Each part is one piece (hand_knob): fails. 3 bodies (red is drawn on top, so a piece inside another still shows)","width":900,"height":672,"kind":"pieces"},"no_view_reason":null,"drawing_note":"red is drawn on top, so a piece inside another still shows"},{"outcome":"pass","label":null,"trial_id":"pilot-nut-knob/nut-knob/M5/can_ask/anthropic__claude-opus-5.5/t1","trial_slug":"pilot-nut-knob--nut-knob--M5--can_ask--anthropic__claude-opus-5.5--t1","href":"./?trial=pilot-nut-knob--nut-knob--M5--can_ask--anthropic__claude-opus-5.5--t1","model_short":"Opus 5.5","task":"nut-knob/M5","task_label":"Hand knob, M5","trial":1,"verdict":"pass","part":"knob","note":null,"state":"pass","detail":"1 bodies","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/456219e6/knob-connected_bodies-pass.webp","caption":"Each part is one piece (knob): passes. 1 bodies","width":900,"height":625,"kind":"pieces"},"no_view_reason":null,"drawing_note":null}]},{"anchor":"nut-knob-R3","check":"R3","task":"nut-knob","task_label":"Hand knob (M3 and M5)","title":"Hex pocket holds the nut","tests":"Is there a hex-shaped pocket the nut can be pushed into from one side, that stops it turning and stops it falling out the other side?","how":["Cuts the knob into 0.2 mm layers along its print axis and looks in each layer for holes between 0.8 and 1.5 times the nut's width across flats (8.0 mm for M5, 5.5 mm for M3).","A hole is hex-shaped if a real nut that wide (corners trimmed to 95% of a sharp hexagon, as ISO 4032 nuts are) fits inside it at some angle, tried every 5 degrees. A round hole that wide can't hold the nut's corners, so the nut could spin.","Holes stacked one on another make a pocket; the one nearest the knob's middle is the nut's. It must be at least 1 mm deep.","A nut at the pocket's tightest size is tried through every layer above and below it. If it can get out of the bed side or the top side, the pocket opens that way.","Fails if there's no pocket, if it opens on neither side (closed in), if it opens on both (the nut can fall through), or if it's shallower than the nut's height less one layer."],"record":"None needed for the shape; this is geometry. Whether the nut fits snugly is the next check, R4.","not_graded":"When the check can't read the shape, a nut-sized hole running sideways (a knob printed on its side) or slits that split the walls into fingers.","weak":"It treats plastic as rigid. A pocket whose way in is narrower than the nut, meant to flex as the nut is pushed past (a snap lip, a springy mouth, slotted fingers), reads as closed when the nut, at the pocket's own size, can't pass the mouth. 12 of the 18 first prints that fail this check describe a nut snapping in like that, by their own notes. Round 3 printed two lip pockets a nut was pushed past (one went in snug, one wouldn't take the nut), but nothing like these mouths, so no record says whether they would work. One of the 18 is a crash of the check's own code on the shape, recorded as a fail. It also reads pockets only along the print axis.","inference":null,"code":[{"file":"pbe/graders/nuts.py","function":"pocket","line":377,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/nuts.py#L377"},{"file":"pbe/graders/nuts.py","function":"find_pockets","line":257,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/nuts.py#L257"}],"photo":null,"counts":{"first_prints":216,"failed":18,"not_graded":1,"passed":179,"not_reached":16,"no_part_built":0,"declined":0,"no_design":2,"crashed":1,"via_test_print":0,"reasons":[{"key":"the-pocket-is-closed-so-the-nut-can-t-be-pressed-in","label":"The pocket is closed, so the nut can't be pressed in","kind":"failed","trials":18,"href":"./?reason=the-pocket-is-closed-so-the-nut-can-t-be-pressed-in"},{"key":"a-pocket-shape-the-checker-can-t-read","label":"A pocket shape the checker can't read","kind":"not graded","trials":1,"href":"./?reason=a-pocket-shape-the-checker-can-t-read"}]},"examples_note":null,"examples":[{"outcome":"fail","label":null,"trial_id":"pilot-nut-knob/nut-knob/M5/can_ask/z-ai__glm-5.3-flash/t6","trial_slug":"pilot-nut-knob--nut-knob--M5--can_ask--z-ai__glm-5.3-flash--t6","href":"./?trial=pilot-nut-knob--nut-knob--M5--can_ask--z-ai__glm-5.3-flash--t6","model_short":"GLM 5.3 Flash","task":"nut-knob/M5","task_label":"Hand knob, M5","trial":6,"verdict":"fail","part":"-","note":"The model meant the nut to be pushed past a springy mouth narrower than its corners (its note calls it a spring cone). The check can't flex plastic, so it reads the pocket as closed.","state":"fail","detail":"pocket at z 3.8 to 10.5 is closed: a nut can't be pressed in from either face","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/3124bb83/design-R3-fail.webp","caption":"Hex pocket holds the nut: fails. Pocket at z 3.8 to 10.5 is closed: a nut can't be pressed in from either face","width":900,"height":659,"kind":"pocket-cut"},"no_view_reason":null,"drawing_note":null},{"outcome":"pass","label":null,"trial_id":"pilot-nut-knob/nut-knob/M5/can_ask/anthropic__claude-opus-5.5/t1","trial_slug":"pilot-nut-knob--nut-knob--M5--can_ask--anthropic__claude-opus-5.5--t1","href":"./?trial=pilot-nut-knob--nut-knob--M5--can_ask--anthropic__claude-opus-5.5--t1","model_short":"Opus 5.5","task":"nut-knob/M5","task_label":"Hand knob, M5","trial":1,"verdict":"pass","part":"-","note":null,"state":"pass","detail":"hex pocket 8.00 across flats, 5.0 deep (nut 4.7 tall), opens to the top","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/fe0d45d7/design-R3-pass.webp","caption":"Hex pocket holds the nut: passes. Hex pocket 8.00 across flats, 5.0 deep (nut 4.7 tall), opens to the top","width":900,"height":653,"kind":"pocket-cut"},"no_view_reason":null,"drawing_note":null}]},{"anchor":"nut-knob-R4","check":"R4","task":"nut-knob","task_label":"Hand knob (M3 and M5)","title":"Nut presses in and stays","tests":"Will the nut press in by hand and stay in when the knob is turned upside down?","how":["Finds the nut's pocket the same way the pocket check (R3) does, and reads its width across flats at its tightest layer. The clearance is that width less the nut's (8.0 mm for M5). A pocket R3 found closed is still graded here, so a design can fail both.","Looks the clearance up in Dan's recorded pocket prints for that nut and that pocket shape (plain hex, or with ribs or a lip). For M5 plain pockets, -0.15 and -0.10 mm wouldn't go in; -0.05, 0 and +0.10 held; +0.20 was loose. For M3, 0 wouldn't go in and +0.10 to +0.30 held.","At +0.30 the M5 record also says loose, but the check reads loose at +0.30 or more as fell out, because M5 and M8 nuts fell out of +0.30 pockets on Dan's earlier tripod build, printed in PETG.","An exact match takes that print's result. Between two prints that agree, it takes theirs. Tighter than the tightest print, when that one wouldn't go in, it won't go in either; looser than the loosest, when that one was loose or fell out, the same. Otherwise the nearest print within 0.2 mm decides.","A result of held passes; wouldn't go in and fell out fail. Loose doesn't settle it, because nobody turned that pocket upside down.","A second way to pass, for plain hex pockets only. The design's manual has the person print a test piece with at least one pocket (even a single one) of a size the records say holds, and changing one of the design's parameters by 0.2 mm moves the final pocket by at least 0.1 mm, so the person can apply what the test piece shows."],"record":"Two test prints. Round 1's B (results/predictions/round1-coupons.yaml, B_nuts): plain hex pockets from 0 to +0.30 mm in 0.1 mm steps, meant to be tested by pressing the nut in by hand, turning the piece over and shaking. Dan's notes say how each nut went in; the M5 pockets at +0.20 and +0.30 are recorded as loose, \"not yet checked upside down\". Round 3's I (round3-coupons.yaml, I_nuts): plain M5 pockets at -0.15, -0.10 and -0.05 mm (cells U1 to U3), and pockets with crush ribs or a lip. By thumb most of these didn't go in; pressed on with a bolt, U3 went in when pushed hard and stayed in upside down. Claude's scoring call counts that as held: going in by thumb alone isn't required.","not_graded":"A pocket size or shape no recorded print settles, for example ribs at a size round 3 didn't print, or an M5 plain pocket at +0.20 (loose, but never turned over).","weak":"One printer, one filament, one person's thumb. A few readings rest on Dan's notes rather than the recorded word (M3 at +0.10 is coded \"won't go in\", but he said it goes in with force), and those are listed in the code. Reading +0.30 as fell out borrows a result from a different filament.","inference":"Reading M5 loose at +0.30 as fell out comes from the PETG tripod build, not this print. Counting a nut pressed on with a bolt as going in is Claude's scoring call.","code":[{"file":"pbe/graders/nuts.py","function":"fit","line":476,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/nuts.py#L476"},{"file":"pbe/graders/nuts.py","function":"reading","line":316,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/nuts.py#L316"}],"photo":{"url":"https://media.danoved.xyz/print-build-eval/nogit/854da3b2/nut-test-print.gif","thumb":"https://media.danoved.xyz/print-build-eval/site/a1740757/ref-nut-test-print.jpg","width":300,"height":534,"kind":"test print","caption":"Pressing nuts into the nut-pocket test print, Oct 29, at 2.5 times speed.","source":"README.md; results/predictions/round1-coupons.yaml","reason":"nut-pocket-a-size-the-prints-don-t-take"},"counts":{"first_prints":216,"failed":63,"not_graded":33,"passed":96,"not_reached":22,"no_part_built":0,"declined":0,"no_design":2,"crashed":1,"via_test_print":22,"reasons":[{"key":"nut-pocket-a-size-the-prints-don-t-take","label":"Nut pocket a size the prints don't take","kind":"failed","trials":63,"href":"./?reason=nut-pocket-a-size-the-prints-don-t-take"},{"key":"nut-pocket-shape","label":"A nut pocket shape no recorded print settles","kind":"not graded","trials":24,"href":"./?reason=nut-pocket-shape"},{"key":"nut-pocket-size","label":"A nut pocket size no recorded print settles","kind":"not graded","trials":9,"href":"./?reason=nut-pocket-size"}],"overlap":{"check":"R3","trials":7}},"examples_note":null,"examples":[{"outcome":"fail","label":null,"trial_id":"pilot-nut-knob/nut-knob/M5/can_ask/anthropic__claude-haiku-5.5/t2","trial_slug":"pilot-nut-knob--nut-knob--M5--can_ask--anthropic__claude-haiku-5.5--t2","href":"./?trial=pilot-nut-knob--nut-knob--M5--can_ask--anthropic__claude-haiku-5.5--t2","model_short":"Haiku 5.5","task":"nut-knob/M5","task_label":"Hand knob, M5","trial":2,"verdict":"fail","part":"-","note":null,"state":"fail","detail":"pocket 7.80 across flats, -0.20 over the 8.0 nut: prints say won't go in; no test print in the manual","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/e5b7b732/design-R4-fail.webp","caption":"Nut presses in and stays: fails. Pocket 7.80 across flats, -0.20 over the 8.0 nut: prints say won't go in; no test print in the manual (pocket 7.80 mm across flats against the nut's 8.0 mm)","width":900,"height":880,"kind":"pocket-fit"},"no_view_reason":null,"drawing_note":"pocket 7.80 mm across flats against the nut's 8.0 mm"},{"outcome":"not graded","label":null,"trial_id":"pilot-nut-knob/nut-knob/M5/can_ask/qwen__qwen3.8-flash/t5","trial_slug":"pilot-nut-knob--nut-knob--M5--can_ask--qwen__qwen3.8-flash--t5","href":"./?trial=pilot-nut-knob--nut-knob--M5--can_ask--qwen__qwen3.8-flash--t5","model_short":"Qwen 3.8 Flash","task":"nut-knob/M5","task_label":"Hand knob, M5","trial":5,"verdict":"not graded","part":"-","note":null,"state":"not graded","detail":"pocket 8.29 across flats at its tightest, with ribs, clips or relief, +0.29 over the 8.0 nut: no round 3 print of that shape settles this size; test prints aren't counted for a pocket like this","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/ffa134c6/design-R4-not-graded.webp","caption":"Nut presses in and stays: not graded. Pocket 8.29 across flats at its tightest, with ribs, clips or relief, +0.29 over the 8.0 nut: round 3's prints of that shape say not measured; test prints aren't counted for a pocket like this (pocket 8.29 mm across flats against the nut's 8.0 mm)","width":900,"height":880,"kind":"pocket-fit"},"no_view_reason":null,"drawing_note":"pocket 8.29 mm across flats against the nut's 8.0 mm"},{"outcome":"pass","label":null,"trial_id":"pilot-nut-knob/nut-knob/M5/can_ask/anthropic__claude-opus-5.5/t1","trial_slug":"pilot-nut-knob--nut-knob--M5--can_ask--anthropic__claude-opus-5.5--t1","href":"./?trial=pilot-nut-knob--nut-knob--M5--can_ask--anthropic__claude-opus-5.5--t1","model_short":"Opus 5.5","task":"nut-knob/M5","task_label":"Hand knob, M5","trial":1,"verdict":"pass","part":"-","note":null,"state":"pass","detail":"pocket 8.00 across flats, +0.00 over the 8.0 nut: prints say holds","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/dcd8e553/design-R4-pass.webp","caption":"Nut presses in and stays: passes. Pocket 8.00 across flats, +0.00 over the 8.0 nut: prints say holds (pocket 8.00 mm across flats against the nut's 8.0 mm)","width":900,"height":880,"kind":"pocket-fit"},"no_view_reason":null,"drawing_note":"pocket 8.00 mm across flats against the nut's 8.0 mm"}]},{"anchor":"bearing-wheel-R3","check":"R3","task":"bearing-wheel","task_label":"Closet door wheel","title":"Bearings press in and stay","tests":"Will the two 608 skateboard bearings press into the wheel and stay in?","how":["Takes each final part's thinnest direction as its axle, cuts it into 0.2 mm layers along that axle, and follows the holes that run all the way through.","The bore is the one with the most layers between 20 and 26 mm wide. A run of such layers is a seat, and its width is the clear width (the largest circle that fits) held over the depth the bearings take up.","The width is compared with Dan's printed seats: 22.05 mm or less wouldn't take a bearing (fail); 22.1, 22.2 and 22.3 took it and held it upside down (pass); 22.4 and over were too loose (fail). Readings within 0.02 mm of those count as them. In the two gaps between them the nearest print decides, and both gaps are narrower than 0.2 mm, so every plain round seat gets a pass or a fail.","Every seat with a bearing is read, and the worst one decides.","As with the knob, a test print in the manual with a seat that holds, plus a parameter that moves the final seat, also passes."],"record":"Round 2's bearing seats (results/predictions/round2-coupons.yaml) and round 3's H (round3-coupons.yaml), black PLA on Dan's printer, a 608 pressed in, the block turned over and shaken. Round 3 records 22.4, 22.5 and 22.7 as \"too loose\"; it doesn't say the bearing fell out.","not_graded":"A seat with ribs, lobes or a slot (only plain round seats were printed), or no seat the check can read.","weak":"The bearing's real outside is 22 mm, and the whole pass band is 0.2 mm wide, finer than the drawing can show; the number line under the drawing carries it. The check's own words for 22.4 and over say the bearing \"fell out\"; the record says too loose. Whether the hub presses on the bearing's inner ring isn't checked.","inference":null,"code":[{"file":"pbe/graders/wheel.py","function":"fit","line":272,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/wheel.py#L272"},{"file":"pbe/graders/wheel.py","function":"_holds","line":169,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/wheel.py#L169"}],"photo":null,"counts":{"first_prints":110,"failed":27,"not_graded":1,"passed":65,"not_reached":16,"no_part_built":0,"declined":0,"no_design":1,"crashed":0,"via_test_print":7,"reasons":[{"key":"bearing-seat-a-size-the-prints-don-t-take","label":"Bearing seat a size the prints don't take","kind":"failed","trials":27,"href":"./?reason=bearing-seat-a-size-the-prints-don-t-take"},{"key":"a-bearing-seat-between-printed-sizes-or-ribbed","label":"A bearing seat between printed sizes, or ribbed","kind":"not graded","trials":1,"href":"./?reason=a-bearing-seat-between-printed-sizes-or-ribbed"}]},"examples_note":null,"examples":[{"outcome":"fail","label":null,"trial_id":"pilot-wheel/bearing-wheel/can_ask/anthropic__claude-haiku-5.5/t1","trial_slug":"pilot-wheel--bearing-wheel--can_ask--anthropic__claude-haiku-5.5--t1","href":"./?trial=pilot-wheel--bearing-wheel--can_ask--anthropic__claude-haiku-5.5--t1","model_short":"Haiku 5.5","task":"bearing-wheel","task_label":"Closet door wheel","trial":1,"verdict":"fail","part":"-","note":null,"state":"fail","detail":"21.89 mm: a 22.05 mm seat wouldn't take the bearing; no test print in the manual","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/ae06e43a/design-R3-fail.webp","caption":"Bearings press in and stay: fails. 21.89 mm: a 22.05 mm seat wouldn't take the bearing; no test print in the manual (the bearing is drawn at its catalog 22 mm; the difference is too small to see, so the line below carries it)","width":900,"height":799,"kind":"wheel-seat-fit"},"no_view_reason":null,"drawing_note":"the bearing is drawn at its catalog 22 mm; the difference is too small to see, so the line below carries it"},{"outcome":"pass","label":null,"trial_id":"pilot-wheel/bearing-wheel/can_ask/anthropic__claude-opus-5.5/t1","trial_slug":"pilot-wheel--bearing-wheel--can_ask--anthropic__claude-opus-5.5--t1","href":"./?trial=pilot-wheel--bearing-wheel--can_ask--anthropic__claude-opus-5.5--t1","model_short":"Opus 5.5","task":"bearing-wheel","task_label":"Closet door wheel","trial":1,"verdict":"pass","part":"-","note":null,"state":"pass","detail":"22.14 mm: it presses in and holds upside down","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/b735c56e/design-R3-pass.webp","caption":"Bearings press in and stay: passes. 22.14 mm: it presses in and holds upside down (the bearing is drawn at its catalog 22 mm; the difference is too small to see, so the line below carries it)","width":900,"height":788,"kind":"wheel-seat-fit"},"no_view_reason":null,"drawing_note":"the bearing is drawn at its catalog 22 mm; the difference is too small to see, so the line below carries it"}]},{"anchor":"tripod-plate-R2","check":"R2","task":"tripod-plate","task_label":"Tripod plate","title":"Fits the tripod's clamp","tests":"Will the plate slide into the tripod's clamp and be held by its angled lips?","how":["The plate is the final part with the largest footprint; its thinnest direction is its thickness.","Cuts it into 0.2 mm layers through that thickness (leaving out pads or marks standing on a face) and reads each layer's outline from edge to edge, on both sides of the plate.","The narrowing is how far both edges move in from the widest layer to the narrowest. One edge alone gives the lips nothing to pinch on the other side, so the smaller of the two counts.","Fails with straight sides, narrowing less than 0.9 mm (1 mm wanted, less half a layer for reading at layer middles). Then the widest layer must be 37 to 39 mm; any other width is not graded."],"record":"The one plate Dan printed, 38 mm wide with 45-degree edges, which fit his clamp.","not_graded":"A plate width outside 37 to 39 mm. How far the clamp's jaws travel isn't recorded, so a plate further off isn't failed.","weak":"It checks that both edges narrow, not the angle or which face is wide, so any taper of 1 mm passes. Which face goes into the clamp is checked under R3.","inference":"The 1 mm either side of 38 mm is an estimate (the clamp's jaws have travel), not a print.","code":[{"file":"pbe/graders/bolted.py","function":"fits_clamp","line":220,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/bolted.py#L220"},{"file":"pbe/graders/bolted.py","function":"_clamped_widths","line":178,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/bolted.py#L178"}],"photo":null,"counts":{"first_prints":110,"failed":21,"not_graded":5,"passed":62,"not_reached":9,"no_part_built":0,"declined":10,"no_design":3,"crashed":0,"via_test_print":0,"reasons":[{"key":"straight-sides-nothing-for-the-clamp-s-lips-to-grip","label":"Straight sides, nothing for the clamp's lips to grip","kind":"failed","trials":21,"href":"./?reason=straight-sides-nothing-for-the-clamp-s-lips-to-grip"},{"key":"a-plate-width-outside-37-to-39-mm","label":"A plate width outside 37 to 39 mm","kind":"not graded","trials":5,"href":"./?reason=a-plate-width-outside-37-to-39-mm"}]},"examples_note":null,"examples":[{"outcome":"fail","label":null,"trial_id":"pilot-tripod/tripod-plate/can_ask/anthropic__claude-haiku-5.5/t2","trial_slug":"pilot-tripod--tripod-plate--can_ask--anthropic__claude-haiku-5.5--t2","href":"./?trial=pilot-tripod--tripod-plate--can_ask--anthropic__claude-haiku-5.5--t2","model_short":"Haiku 5.5","task":"tripod-plate","task_label":"Tripod plate","trial":2,"verdict":"fail","part":"-","note":null,"state":"fail","detail":"straight sides: it narrows by only 0.00 mm on both edges (want 1.0), so the clamp's lips have nothing to bear on","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/6aa099c8/design-R2-fail.webp","caption":"Fits the tripod's clamp: fails. Straight sides: it narrows by only 0.00 mm on both edges (want 1.0), so the clamp's lips have nothing to bear on (widths 40.00 mm at the low face, 40.00 mm at the high face)","width":900,"height":704,"kind":"plate-profile"},"no_view_reason":null,"drawing_note":"widths 40.00 mm at the low face, 40.00 mm at the high face"},{"outcome":"pass","label":null,"trial_id":"pilot-tripod/tripod-plate/can_ask/anthropic__claude-opus-5.5/t5","trial_slug":"pilot-tripod--tripod-plate--can_ask--anthropic__claude-opus-5.5--t5","href":"./?trial=pilot-tripod--tripod-plate--can_ask--anthropic__claude-opus-5.5--t5","model_short":"Opus 5.5","task":"tripod-plate","task_label":"Tripod plate","trial":5,"verdict":"pass","part":"-","note":null,"state":"pass","detail":"37.8 mm wide, narrowing by 3.9 mm on both edges","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/9dc545ad/design-R2-pass.webp","caption":"Fits the tripod's clamp: passes. 37.8 mm wide, narrowing by 3.9 mm on both edges (widths 30.00 mm at the low face, 30.00 mm at the high face)","width":900,"height":718,"kind":"plate-profile"},"no_view_reason":null,"drawing_note":"widths 30.00 mm at the low face, 30.00 mm at the high face"}]},{"anchor":"tripod-plate-R3","check":"R3","task":"tripod-plate","task_label":"Tripod plate","title":"Bolt goes through, head on the clamp side","tests":"Does the camera's 1/4-20 bolt go through the plate by hand, with its head under the plate on the clamp side?","how":["If the design's hardware leaves out the salvaged webcam bolt and lists a socket-head (ISO 4762) bolt instead, it fails; leaving it out otherwise is read as printing its own screw, which is not graded (no printed thread was ever tested).","Follows every hole through all of the plate's layers, keeps the one the bolt fits through, and reads its tightest clear width.","Under 6.35 mm the bolt won't pass, and 15 mm or more is taken to let the head pull through; both fail.","The head's recess (a wider part at one end of the hole, at least 1 mm deep) must open on the plate's wide face, which goes into the clamp. On the narrow, camera-side face it fails.","Then the width is looked up in round 3's bolt-hole test print: at 6.35 mm the bolt only screwed through (fails, because the plate's own thread would stop the head pulling the camera down); at 6.5 and 6.6 mm it went through by hand (passes). Looser than 6.6 also passes, up to 13 mm, the widest hole known to stop the head; between 13 and 15 is not graded."],"record":"Round 3's test print F (results/predictions/round3-coupons.yaml, F_bolt_holes) for the hole, and Dan's printed plate, whose 13 mm recess stopped the head.","not_graded":"A hole between 13 and 15 mm, or a printed screw. Between 6.35 and 6.5 mm the nearest recorded hole is always within 0.2 mm, so that gap is decided by the nearest-print rule, and the check says so.","weak":"A plate with a groove in its side doesn't say which face is wide, so the face test is skipped for it.","inference":"That a 15 mm hole would let the head through is the task brief's inference from a head that fit a 15 mm recess, not a print. That a hole the bolt only screws through fails is Claude's scoring call.","code":[{"file":"pbe/graders/bolted.py","function":"bolt_hole","line":235,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/bolted.py#L235"},{"file":"pbe/graders/bolt_fits.py","function":"reading","line":67,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/bolt_fits.py#L67"}],"photo":null,"counts":{"first_prints":110,"failed":23,"not_graded":0,"passed":65,"not_reached":9,"no_part_built":0,"declined":10,"no_design":3,"crashed":0,"via_test_print":0,"reasons":[{"key":"dovetail-upside-down-the-bolt-head-s-recess-is-on-the-camera-s-side","label":"Dovetail upside down: the bolt head's recess is on the camera's side","kind":"failed","trials":13,"href":"./?reason=dovetail-upside-down-the-bolt-head-s-recess-is-on-the-camera-s-side"},{"key":"no-usable-bolt-hole","label":"No usable bolt hole","kind":"failed","trials":10,"href":"./?reason=no-usable-bolt-hole"}]},"examples_note":null,"examples":[{"outcome":"fail","label":null,"trial_id":"pilot-tripod/tripod-plate/can_ask/anthropic__claude-opus-5.5/t1","trial_slug":"pilot-tripod--tripod-plate--can_ask--anthropic__claude-opus-5.5--t1","href":"./?trial=pilot-tripod--tripod-plate--can_ask--anthropic__claude-opus-5.5--t1","model_short":"Opus 5.5","task":"tripod-plate","task_label":"Tripod plate","trial":1,"verdict":"fail","part":"-","note":null,"state":"fail","detail":"the head's recess opens on the narrow face: the head is on the camera's side, or the dovetail is upside down","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/1d5d949e/design-R3-fail.webp","caption":"Bolt goes through, head on the clamp side: fails. The head's recess opens on the narrow face: the head is on the camera's side, or the dovetail is upside down (tightest 6.80 mm clear (red line); recess 16.00 mm across, 5.0 mm deep)","width":900,"height":487,"kind":"plate-bolt-cut"},"no_view_reason":null,"drawing_note":"tightest 6.80 mm clear (red line); recess 16.00 mm across, 5.0 mm deep"},{"outcome":"pass","label":null,"trial_id":"pilot-tripod/tripod-plate/can_ask/anthropic__claude-opus-5.5/t5","trial_slug":"pilot-tripod--tripod-plate--can_ask--anthropic__claude-opus-5.5--t5","href":"./?trial=pilot-tripod--tripod-plate--can_ask--anthropic__claude-opus-5.5--t5","model_short":"Opus 5.5","task":"tripod-plate","task_label":"Tripod plate","trial":5,"verdict":"pass","part":"-","note":null,"state":"pass","detail":"6.80 mm clear at its tightest: the bolt goes through by hand (round 3)","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/3b82b2f7/design-R3-pass.webp","caption":"Bolt goes through, head on the clamp side: passes. 6.80 mm clear at its tightest: the bolt goes through by hand (round 3) (tightest 6.80 mm clear (red line); recess 16.20 mm across, 6.0 mm deep)","width":900,"height":748,"kind":"plate-bolt-cut"},"no_view_reason":null,"drawing_note":"tightest 6.80 mm clear (red line); recess 16.20 mm across, 6.0 mm deep"}]},{"anchor":"tripod-plate-R4","check":"R4","task":"tripod-plate","task_label":"Tripod plate","title":"Bolt head fits its recess","tests":"Does the bolt's head fit in its recess, deep enough to sit flush?","how":["Finds the bolt's hole the same way R3 does and reads the recess at the end of it, how wide and how deep.","Less than 1 mm deep counts as no recess (a chamfer), and is not graded, because nobody printed a plate with the head standing proud.","13 mm across or less fails, too tight for this head. 15 mm or more is wide enough. In between, the nearest of the two within 0.2 mm decides, otherwise not graded.","It must also be at least about 5 mm deep (4.69 mm, allowing for layers and the 0.2 mm tolerance); shallower is not graded."],"record":"Dan's printed plate and the session record behind it, where a 13 mm recess was too tight for the salvaged webcam bolt's head and a 15 mm by 5 mm one fit.","not_graded":"A recess under 5 mm deep, no recess at all, or a width between 13.2 and 14.8 mm. Only one depth was ever printed, so this is the tripod's commonest not-graded check.","weak":"The head's true size was never measured; the person in the trials says \"about 15 mm across, about 5 mm tall\". There's no upper limit on the recess's width.","inference":null,"code":[{"file":"pbe/graders/bolted.py","function":"head_recess","line":274,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/bolted.py#L274"},{"file":"pbe/graders/bolted.py","function":"Bore.recess","line":62,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/bolted.py#L62"}],"photo":null,"counts":{"first_prints":110,"failed":3,"not_graded":22,"passed":56,"not_reached":16,"no_part_built":0,"declined":10,"no_design":3,"crashed":0,"via_test_print":0,"reasons":[{"key":"a-tripod-head-recess-under-5-mm-deep","label":"A tripod head recess under 5 mm deep","kind":"not graded","trials":22,"href":"./?reason=a-tripod-head-recess-under-5-mm-deep"},{"key":"head-recess-too-small-for-the-bolt-s-head","label":"Head recess too small for the bolt's head","kind":"failed","trials":3,"href":"./?reason=head-recess-too-small-for-the-bolt-s-head"}]},"examples_note":null,"examples":[{"outcome":"not graded","label":null,"trial_id":"pilot-tripod/tripod-plate/can_ask/anthropic__claude-haiku-5.5/t1","trial_slug":"pilot-tripod--tripod-plate--can_ask--anthropic__claude-haiku-5.5--t1","href":"./?trial=pilot-tripod--tripod-plate--can_ask--anthropic__claude-haiku-5.5--t1","model_short":"Haiku 5.5","task":"tripod-plate","task_label":"Tripod plate","trial":1,"verdict":"fail","part":"-","note":null,"state":"not graded","detail":"recess 4.4 mm deep: only 5 mm has been printed","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/a90a8e1d/design-R4-not-graded.webp","caption":"Bolt head fits its recess: not graded. Recess 4.4 mm deep: only 5 mm has been printed (tightest 6.60 mm clear (red line); recess 15.60 mm across, 4.4 mm deep)","width":900,"height":653,"kind":"plate-bolt-cut"},"no_view_reason":null,"drawing_note":"tightest 6.60 mm clear (red line); recess 15.60 mm across, 4.4 mm deep"},{"outcome":"fail","label":null,"trial_id":"pilot-tripod/tripod-plate/can_ask/openai__gpt-6-luna/t5","trial_slug":"pilot-tripod--tripod-plate--can_ask--openai__gpt-6-luna--t5","href":"./?trial=pilot-tripod--tripod-plate--can_ask--openai__gpt-6-luna--t5","model_short":"GPT-6 Luna","task":"tripod-plate","task_label":"Tripod plate","trial":5,"verdict":"fail","part":"-","note":null,"state":"fail","detail":"recess 7.90 mm across: the 13 mm one was too tight for this head","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/9e797bed/design-R4-fail.webp","caption":"Bolt head fits its recess: fails. Recess 7.90 mm across: the 13 mm one was too tight for this head (tightest 3.40 mm clear (red line); recess 7.90 mm across, 5.4 mm deep)","width":900,"height":745,"kind":"plate-bolt-cut"},"no_view_reason":null,"drawing_note":"tightest 3.40 mm clear (red line); recess 7.90 mm across, 5.4 mm deep"},{"outcome":"pass","label":null,"trial_id":"pilot-tripod/tripod-plate/can_ask/anthropic__claude-opus-5.5/t5","trial_slug":"pilot-tripod--tripod-plate--can_ask--anthropic__claude-opus-5.5--t5","href":"./?trial=pilot-tripod--tripod-plate--can_ask--anthropic__claude-opus-5.5--t5","model_short":"Opus 5.5","task":"tripod-plate","task_label":"Tripod plate","trial":5,"verdict":"pass","part":"-","note":null,"state":"pass","detail":"recess 16.20 mm across and 6.0 mm deep","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/99126aa4/design-R4-pass.webp","caption":"Bolt head fits its recess: passes. Recess 16.20 mm across and 6.0 mm deep (tightest 6.80 mm clear (red line); recess 16.20 mm across, 6.0 mm deep)","width":900,"height":745,"kind":"plate-bolt-cut"},"no_view_reason":null,"drawing_note":"tightest 6.80 mm clear (red line); recess 16.20 mm across, 6.0 mm deep"}]},{"anchor":"estoa-light-R1","check":"R1","task":"estoa-light","task_label":"Estoa light box","title":"Disk screws go into the box","tests":"Will the two screws through the LED disk's holes find somewhere to go into the box?","how":["Puts the box together the way the design's assembly says, with the disk where the design places it. A disk facing the wall fails.","Dan measured the disk's two holes, but his readings leave each hole's center anywhere from 16.60 to 17.11 mm from the disk's center. The check treats that whole range as possible.","The disk's board is taken as 1.6 mm thick, the standard circuit board (never measured). Plastic where the board itself is fails; printed pins through its holes are not graded.","Behind each hole, it reads the plastic layer by layer. The first plastic must be within 0.6 mm (the disk rests on the box); none at all fails.","A screw can move inside a clearance hole, but a pilot hole it cuts its thread in, or a hole sized for a heat-set insert (3.9 to 4.9 mm, in a design that lists inserts), holds it at the hole's center. The disk's 3.3 mm holes give an M3 screw 0.15 mm of play.","At 11 points across the range, it asks whether a screw through the disk's hole there can go into the box. All 11, pass; none, fail; only some, not graded.","A second way to pass, when the holes are the only problem. Any acceptance criterion of the design's whose check method is a template or test print, and any parameter that criterion names: the check changes each such parameter by 1 mm (up to 8 of them), rebuilds the box, and if the plastic 0.3 mm behind the holes changes by more than 0.5 mm², it passes. It doesn't check that the criterion is about where the holes are."],"record":"Dan's caliper readings of the disk (2026-09-28) and his Estoa build, which placed the holes from a printout and fit.","not_graded":"A screw that goes in only if the hole is at one end of where Dan's readings put it (for example into a heat-set insert at a fixed spot) with no template rescue, printed pins through the disk's holes, a disk the design doesn't place exactly once, no assembly saying where the parts go, or a mesh the check can't read.","weak":"It trusts the design's own assembly placements. The template rescue is loose. Any test-print or template criterion counts, and any parameter it names that moves the plastic behind the holes, so a design can pass on a test of something else. Claude Opus 5.5's trial 2 below passed on an insert test (whether an insert melts in straight and holds an M3) and the insert hole's diameter, not on where the holes are. This was reported to the session that owns grading; the grader is unchanged here.","inference":null,"code":[{"file":"pbe/graders/enclosure.py","function":"board_holes","line":251,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/enclosure.py#L251"},{"file":"pbe/graders/enclosure.py","function":"_hole_verdict","line":152,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/enclosure.py#L152"},{"file":"pbe/graders/enclosure.py","function":"_template_rescue","line":236,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/enclosure.py#L236"}],"photo":null,"counts":{"first_prints":30,"failed":1,"not_graded":11,"passed":16,"not_reached":1,"no_part_built":0,"declined":0,"no_design":1,"crashed":0,"via_test_print":11,"reasons":[{"key":"inserts-hold-the-disk-s-screws-at-one-spot-and-nothing-checks-the-holes-first","label":"Inserts hold the disk's screws at one spot, and nothing checks the holes first","kind":"not graded","trials":9,"href":"./?reason=inserts-hold-the-disk-s-screws-at-one-spot-and-nothing-checks-the-holes-first"},{"key":"the-design-doesn-t-say-where-the-person-s-parts-go","label":"The design doesn't say where the person's parts go","kind":"not graded","trials":2,"href":"./?reason=the-design-doesn-t-say-where-the-person-s-parts-go"},{"key":"the-box-s-holes-miss-the-disk-s-holes","label":"The box's holes miss the disk's holes","kind":"failed","trials":1,"href":"./?reason=the-box-s-holes-miss-the-disk-s-holes"}]},"examples_note":"Most first prints that pass this check pass by the template rule, not on their holes as drawn; the count is below.","examples":[{"outcome":"not graded","label":null,"trial_id":"pilot-estoa/estoa-light/can_ask/anthropic__claude-haiku-5.5/t1","trial_slug":"pilot-estoa--estoa-light--can_ask--anthropic__claude-haiku-5.5--t1","href":"./?trial=pilot-estoa--estoa-light--can_ask--anthropic__claude-haiku-5.5--t1","model_short":"Haiku 5.5","task":"estoa-light","task_label":"Estoa light box (harder)","trial":1,"verdict":"fail","part":"-","note":null,"state":"not graded","detail":"a screw through the hole at +16.6 mm goes in only if the hole is at one end of where Dan's readings put it; no template or test print in the design would catch it","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/e68277de/design-R1-not-graded.webp","caption":"Disk screws go into the box: not graded. A screw through the hole at +16.6 mm goes in only if the hole is at one end of where Dan's readings put it; no template or test print in the design would catch it","width":900,"height":668,"kind":"disk-plan"},"no_view_reason":null,"drawing_note":null},{"outcome":"fail","label":null,"trial_id":"pilot-estoa/estoa-light/can_ask/deepseek__deepseek-v4-pro-0813/t3","trial_slug":"pilot-estoa--estoa-light--can_ask--deepseek__deepseek-v4-pro-0813--t3","href":"./?trial=pilot-estoa--estoa-light--can_ask--deepseek__deepseek-v4-pro-0813--t3","model_short":"DeepSeek V4 Pro","task":"estoa-light","task_label":"Estoa light box (harder)","trial":3,"verdict":"fail","part":"-","note":null,"state":"fail","detail":"printed plastic stands where the disk's board is","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/c4162dbc/design-R1-fail.webp","caption":"Disk screws go into the box: fails. Printed plastic stands where the disk's board is","width":900,"height":684,"kind":"disk-plan"},"no_view_reason":null,"drawing_note":null},{"outcome":"pass","label":null,"trial_id":"pilot-estoa/estoa-light/can_ask/openai__gpt-6.1-sol/t3","trial_slug":"pilot-estoa--estoa-light--can_ask--openai__gpt-6.1-sol--t3","href":"./?trial=pilot-estoa--estoa-light--can_ask--openai__gpt-6.1-sol--t3","model_short":"GPT-6.1 Sol","task":"estoa-light","task_label":"Estoa light box (harder)","trial":3,"verdict":"fail","part":"-","note":null,"state":"pass","detail":"a screw through the hole at +16.6 mm goes into the box wherever the hole really is","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/a15f0a15/design-R1-pass.webp","caption":"Disk screws go into the box: passes. A screw through the hole at +16.6 mm goes into the box wherever the hole really is","width":900,"height":668,"kind":"disk-plan"},"no_view_reason":null,"drawing_note":null},{"outcome":"pass","label":"Passes, by the template rule","trial_id":"pilot-estoa/estoa-light/can_ask/anthropic__claude-opus-5.5/t2","trial_slug":"pilot-estoa--estoa-light--can_ask--anthropic__claude-opus-5.5--t2","href":"./?trial=pilot-estoa--estoa-light--can_ask--anthropic__claude-opus-5.5--t2","model_short":"Opus 5.5","task":"estoa-light","task_label":"Estoa light box (harder)","trial":2,"verdict":"pass","part":"-","note":"Not an honest pass. A screw goes in only at one end of the range, and the template rule passed it on a criterion that tests whether a heat-set insert melts in straight and holds an M3, using the insert hole's diameter (insert_hole_d) as the parameter that moves the holes.","state":"pass","detail":"a screw through the hole at +16.6 mm goes in only if the hole is at one end of where Dan's readings put it; a template or test print checks the holes, and param insert_hole_d moves them to match","via_test_print":true,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/52c5e577/design-R1-pass.webp","caption":"Disk screws go into the box: passes. A screw through the hole at +16.6 mm goes in only if the hole is at one end of where Dan's readings put it; a template or test print checks the holes, and param insert_hole_d moves them to match","width":900,"height":682,"kind":"disk-plan"},"no_view_reason":null,"drawing_note":null}]},{"anchor":"estoa-light-R6","check":"R6","task":"estoa-light","task_label":"Estoa light box","title":"Hangs on the wall screws","tests":"Can the box be hung on the wall through keyholes in its back, on some ordinary nail or screw?","how":["Puts the box together from the design's assembly, takes the lowest point of any printed part as the wall, and cuts the back 0.3 mm from it.","Every hole in that cut is tested as a keyhole, a round entry (the largest circle that fits) with a slot running off it at least 1 mm narrower.","Which way is up comes from the slots themselves: the average direction they run. The person can hang the box any way round.","A keyhole hangs on a fastener if the fastener's head passes the entry with 0.4 mm to spare, its shank fits the slot, and the slot is narrower than its head. It tries five common ones: a wire nail (head 4.5 mm, shank 2.0), a common nail (6.7, 2.9), and #6, #8 and #10 pan-head screws (6.5 and 3.5, 7.9 and 4.17, 9.47 and 4.83). The person picks the fastener to suit the keyhole, so one is enough.","A keyhole fails if no fastener hangs it, if its slot runs more than 60 degrees from up, or if the fastener would rest inside the entry. A back with no hole at all fails.","Not graded if a slot leans more than 30 degrees, or if fewer than two keyholes are found."],"record":"Fastener sizes from published nail and screw sizes, not a print, by Claude's scoring call, after Dan said on 2026-11-13 that hanging on a common nail is fine and the screws can stand further out. Estoa's own keyholes (10.5 mm entry, 4.8 mm slot) hung it.","not_graded":"A slot that leans more than 30 degrees from the others, or a box hung some other way (open slots, a cleat). It reads only keyholes.","weak":"No weight, no wall, no pull-out strength, and no check of how thick the plastic is at the slot, since the person sets how far the nail stands out. Since this rule came in (2026-11-13) no counted first print fails R6.","inference":"The fastener list and the 0.4 mm margin are Claude's scoring call from published sizes.","code":[{"file":"pbe/graders/enclosure.py","function":"keyholes","line":926,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/enclosure.py#L926"},{"file":"pbe/graders/enclosure.py","function":"_keyhole_verdict","line":901,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/enclosure.py#L901"},{"file":"pbe/graders/enclosure.py","function":"_hangs_on","line":895,"url":"https://github.com/oveddan/functional-print-eval/blob/8a73e99a751d463b68945fc23a7a61c875d1e9e1/pbe/graders/enclosure.py#L895"}],"photo":null,"counts":{"first_prints":30,"failed":0,"not_graded":3,"passed":25,"not_reached":1,"no_part_built":0,"declined":0,"no_design":1,"crashed":0,"via_test_print":0,"reasons":[{"key":"it-hangs-some-other-way-and-the-checker-reads-only-keyholes","label":"It hangs some other way, and the checker reads only keyholes","kind":"not graded","trials":3,"href":"./?reason=it-hangs-some-other-way-and-the-checker-reads-only-keyholes"}]},"examples_note":"No counted first print fails R6 today. Claude Haiku 5.5's trial 2 failed it until 2026-11-13 (its 9.5 mm entry was too small for a","examples":[{"outcome":"not graded","label":null,"trial_id":"pilot-estoa/estoa-light/can_ask/deepseek__deepseek-v4-pro-0813/t1","trial_slug":"pilot-estoa--estoa-light--can_ask--deepseek__deepseek-v4-pro-0813--t1","href":"./?trial=pilot-estoa--estoa-light--can_ask--deepseek__deepseek-v4-pro-0813--t1","model_short":"DeepSeek V4 Pro","task":"estoa-light","task_label":"Estoa light box (harder)","trial":1,"verdict":"fail","part":"-","note":"The check's cut 0.3 mm from the wall finds only the rim of this box, one big opening and no keyhole, so it can't say how the box would hang.","state":"not graded","detail":"0 keyhole(s) found in the back, and this check reads only keyholes","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/dd32f5d6/design-R6-not-graded.webp","caption":"Hangs on the wall screws: not graded. 0 keyhole(s) found in the back, and this check reads only keyholes (1 hole(s) in the back, none of them a keyhole)","width":900,"height":875,"kind":"back-keyholes"},"no_view_reason":null,"drawing_note":"1 hole(s) in the back, none of them a keyhole"},{"outcome":"pass","label":null,"trial_id":"pilot-estoa/estoa-light/can_ask/anthropic__claude-haiku-5.5/t2","trial_slug":"pilot-estoa--estoa-light--can_ask--anthropic__claude-haiku-5.5--t2","href":"./?trial=pilot-estoa--estoa-light--can_ask--anthropic__claude-haiku-5.5--t2","model_short":"Haiku 5.5","task":"estoa-light","task_label":"Estoa light box (harder)","trial":2,"verdict":"fail","part":"-","note":null,"state":"pass","detail":"keyhole at (-60, 45) (entry 9.5, slot 4.60 running 7.7 mm, 0 degrees from up): hangs on a common nail or 2 other common fastener(s)","via_test_print":false,"view":{"url":"https://media.danoved.xyz/print-build-eval/site/e4ac4ced/design-R6-pass.webp","caption":"Hangs on the wall screws: passes. Keyhole at (-60, 45) (entry 9.5, slot 4.60 running 7.7 mm, 0 degrees from up): hangs on a common nail or 2 other common fastener(s)","width":900,"height":699,"kind":"back-keyholes"},"no_view_reason":null,"drawing_note":null}]}]}