Functional 3D Printing Eval

Can a language model design a part that prints and works?

Fully open source under the MIT licence: the code, the tasks, the checks and every saved run are in the GitHub repository.

Language models were asked, in a person's own words, for a part to 3D print. Of their designs, this many pass:

 on the first print   reprint after feedback the same first prints; each one that didn't pass was sent back once  test prints first fewer trials ran; in some the person couldn't say how a test piece came out

A pass means every check passes against test prints that Dan, the eval's author, printed and recorded, under the assumptions listed in the reference. No complete model design was printed (round 3 printed six roof sections cut from models' designs): code graded each design against those records, and where no record decides a check the design is not graded, never a pass.

See a failing design Browse the trials How the checks work

Some of the test prints the designs are graded against. No complete model design was printed.

A plate of small test pieces: an arc of the ring, the disc mount, insert posts
A plate of small test prints
A roof over a hole, printed in the air: loose loops and tangles around the hole
A roof over a hole: it printed as loose loops
The roof strip: square pockets roofed at 20 to 50 mm, walls on all four sides
Roofs over pockets 20 to 50 mm wide, to see which ones print clean
The box's wall with the USB-C openings, the power bank and ESP32 board lined up behind it
A wall with USB-C openings, with the real parts behind it

Why this eval

This is an open-source eval of how well LLMs design functional 3D prints: things that have to do a job once they're printed and put together. A functional print is rarely one shape. A robot arm, a light enclosure or a camera mount is several printed parts joined with screws, nuts, heat-set inserts or bearings, and it only works if every part prints, fits the hardware, and fits the other parts.

The tasks so far start smaller: a camera plate for a tripod, a hand knob with a nut pressed in, and a closet-door wheel on two bearings, each a printed part that has to print without supports and fit real hardware, plus a harder multi-part enclosure for a wall light. The assembly steps the model writes aren't graded yet.

Each task gives a model a request in a person's own words, with the measurements and hardware on hand. The model writes the CAD code for each part, picks the fasteners and writes the assembly steps. The eval then grades the design in code before anything is printed. Will each part print? Will the screws reach their inserts? Will the plug fit through its hole? The answers come from test prints I've actually made.

Printing is slow, and a wrong design costs hours and material before you find out. So the questions are:

  • Which models get a design right on the first print, and at what cost in tokens, material and waiting?
  • Can we tell whether a print will work before printing it?
  • When a model is told what went wrong, the way a test print would tell you, does its fix get closer?

How it works, on GitHub.

How to read a result
Loading…

How the checks work: what each one tests and how a failure is decided, with examples

One failing design, with what fails marked in red.

Why first prints don't pass

The most common reasons, by number of designs. A design can have more than one. Each opens the designs behind it.

    Scorecard

    Pass rate on the four scored parts. Every number opens exactly the trials it counts.

    By model and part

    Passes out of trials for each part, in all three conditions. Each bar opens its trials.

    Each model, part by part

    Trials

    Loading the trials…

    Filters