Contents: How this started · How the agent works · How a trial runs · Constraints · What the eval tests: failure classes · The test prints behind the checks · The tasks · Results · Run it · Limits · Prior work and comparable evals · Tools used · More
The code as submitted to the hackathon is at the hackathon-submission tag; this README describes the eval as it is now.
Browse every trial on the results site: each model's design, every check with its reason, the test prints it asked for, its code and its transcript.
This is an open-source eval of how well LLMs design functional 3D prints: things that have to do a job once they're printed and put together. A functional print is rarely one shape. A robot arm, a light enclosure or a camera mount is several printed parts joined with screws, nuts, heat-set inserts or bearings, and it only works if every part prints, fits the hardware, and fits the other parts.
The tasks so far start smaller: a camera plate for a tripod, a hand knob with a nut pressed in, and a closet-door wheel on two bearings, each a printed part that has to print without supports and fit real hardware, plus a harder multi-part enclosure for a wall light. The assembly steps the model writes aren't graded yet.
Each task gives a model a request in a person's own words, with the measurements and hardware on hand. The model writes the CAD code for each part, picks the fasteners and writes the assembly steps. The eval then grades the design in code before anything is printed. Will each part print? Will the screws reach their inserts? Will the plug fit through its hole? The answers come from test prints I've actually made.
Printing is slow, and a wrong design costs hours and material before you find out. So the questions are:
- Which models get a design right on the first print, and at what cost in tokens, material and waiting?
- Can we tell whether a print will work before printing it?
- When a model is told what went wrong, the way a test print would tell you, does its fix get closer?
The Estoa light's enclosure in OpenSCAD, an open-source modeling tool: the code, written with an LLM, and the part it describes.
Over the past month, I've been working on a bunch of 3D printing projects at home. A week before a gallery show, I was asked to make a piece for it (the post). Starting with an LED disc I'd had for 5 years, I worked with Opus 5.5 to 3D model, fabricate and program it: a light that rewards stillness, with a presence sensor, an ESP32 and frosted sides. With Opus 5.5 I was able to do a lot more than I could before; it told me about heat-set inserts, which I didn't even know about. I also printed parts to hang a lamp I bought at a flea market. A key part of all of this was doing test prints: to see how the frosting looked, and whether the inserts fit the holes the model designed. There were always mistakes on the way: a battery holder designed without knowing the battery's size, a back plate that took 6 hours to print where I couldn't reach the insert to screw it in, and cable holes too narrow for the cable to flex.
![]() |
![]() |
![]() |
|---|---|---|
| The gallery piece, coming apart and back together | Its frosted wall, after a test print showed the layer lines | A test print for the heat-set inserts |
An example of a print that failed, and cost lots of time and money: the back plate for an art installation, which had inserts. The poles for the inserts did not have enough give, causing them to warp when the inserts were pressed in. This was a costly and timely mistake. We fixed it later, but had to reprint the whole plate again.
Examples of functional printing tasks would be:
Or: "Can we design a custom robotic arm that mounts to my desk?"
The loop is in pbe/runner/agent.py. The person's answers come from measure.py (measurements) and coupon_loop.py (how test prints came out), and the checks on the final print are in pbe/graders/.
- A person asks in plain language for a printed fix: a camera plate, a knob with a nut pressed in, a wheel with bearings
- The model is given a 3d printer, filament, and list of available hardware. It writes each part as code (CadQuery, in Python), which it can't run or look at.
- The model decides if it needs to know more information, such as measurements of a servo. If it does, it requests this from the user.
- The user responds with measurements.
- The model decides if it needs to know if an aspect of a print will work, such as an opening, or a connection between hardware and the print. It can then design a test print that can help answer this. The goal is to get to an answer with less filament used or time spent than if we printed the final print without knowing this answer.
- We simulate the test print, and generate a response from the user based on the results. The model can test again.
- It prints a final version. The code then runs some automated tests on the resulting geometry, to simulate the results of the print.
- We simulate the result of this print, and provide the results to the model. If it has not done two prints, it can print again and we repeat that loop.
Some of my test prints:
Trials ran in three conditions. These names are used everywhere in the results:
- First print: the model can ask for measurements and hands in one design, with no feedback.
- Reprint after feedback: the same first prints. Each one that didn't pass was sent back once, the simulated person said how it came out, and the model handed in one more design.
- Test prints first: the model can ask for small test prints before its final design.
flowchart TD
A[Person asks for a part] --> B[Model designs it<br/>and can ask for measurements]
B --> Q{Is the model unsure<br/>part of it will work?}
Q -->|yes, its choice| T[Model asks for a test print]
T --> R[Person says how each piece came out,<br/>from Dan's recorded prints,<br/>or that they can't tell]
R --> B
Q -->|no| F[Final design, printed in simulation:<br/>checks compare it with parts Dan printed]
F --> P[Verified pass]
F --> X[Fail]
F --> N[Not graded:<br/>no recorded print decides it]
A trial where the model asked for a test print before its final design (Sonnet 5.5's M3 knob):
In each trial, nothing actually gets printed. We have checks, run by code, that simulate the model's parts and compare them with the results of my test prints to see if they would print successfully. Here's an example of a trial in the Reprint after feedback condition: it failed, then succeeded after we gave the model feedback about what failed (Gemini 4 Argon on the Estoa light, from the earlier loop, where the feedback came after its first print):
- The print shouldn't require any supports
- The model has up to 12 turns when it can ask for test prints, and 8 otherwise
- We are currently restricting to high-speed PLA.
Every one of our checks comes from failures that happened on my real builds. Going back through my past build sessions, I found 57 failures and categorized them into 10 classes (every check). Six of the classes were tested in these trials.
Not tested yet: C4 tool and hand access, C8 test-plan coverage, C9 manual correctness, C10 print settings.
Every failure has a test case. For example, C6's photo above is a failed first attempt at printing a tripod's Arca plate, as designed by a model. The roof over where the bolt's head goes printed as tangled strands. The command shown below runs the checks again on both the original design and the fixed one. The test fails the original design, just like it happened in real life, but the fixed design passes.
uv run python -m pbe.run examples/arca-plate/original.yaml examples/arca-plate/fix.yamlExpected: the original fails bridges and the fix passes every check.
These test prints provide the ground truth for parts of the 3D printing simulation. Each print tests the ground truth of one thing. For example, if there's a roof over a hole, will it print successfully? We printed nut pockets at different sizes, using different techniques, to see which size a nut would fit into snugly without falling out.
What each print tests, why we printed it, and what happened, from the test-print code (coupons/coupons.py) and the records of each round with my notes (results/predictions/):
| Round one, printing | Round two, printing |
|---|---|
![]() |
![]() |
Round three, on 2026-11-11 and 12, went after the designs the first two rounds couldn't grade: crush-ribbed nut pockets, narrow roof ledges, bolt holes and bearing seats between the sizes I'd printed, and six roofs cut straight out of the models' own Estoa designs. Five of the six roofs strung as the bridge check predicted.
Each task is based on a realistic prompt that a person would write. The reference solution is one I made by hand that I know passes all the checks, to show the task can realistically be done. The other pictures show the designs the models came up with.
"I borrowed my friend's tripod, and I can't seem to find the part that connects the camera to the tripod. How hard would it be to 3D print it, both the flat part and the part that screws in? Just as a temporary thing until I find the real one. I know it's metal normally."
More from the saved trials: four passes and four failures, failures seen from below with the failing roof or ledge in red.
"The roller wheel on my sliding closet door snapped. I've got two 608 skateboard bearings and an M8 bolt I could use. Could I 3D print a new wheel that spins on the bolt? The old one was about 50 mm across and about 16 mm wide."
More from the saved trials: four passes and four failures, failures seen from below with the failing roof or ledge in red.
"I want a hand knob for an M3 bolt, about 20 mm across, so I can tighten it without a tool. It's one printed piece. An M3 hex nut (ISO 4032) presses into the knob by hand and has to stay in when I turn the knob upside down, no glue."
More from the saved trials: four passes and four failures, failures seen from below with the failing roof or ledge in red.
"I'm making a round light to hang on my wall: an Adafruit DotStar LED disc, the 255-LED one. … for now I only need that box, with the disc screwed onto it."
| Reference solution | The real one |
|---|---|
![]() |
![]() |
| A box the disc screws onto; any shape that passes the checks works | Dan's build |
We ran out of time to make the test-print loop work on this task. Some example failures:
Browse every trial on the results site: pick a condition and a part, then open any model's design to see its render, each check with its reason, the test prints it asked for, its code and its transcript.
| Model | Tripod plate | Bearing wheel | Knob M3 | Knob M5 | All four tasks | Estoa light box (harder) | Print hours per pass | Filament per pass | Cost per pass |
|---|---|---|---|---|---|---|---|---|---|
| Gemini 4 Argon | 70% (7/10) | 75% (6/8) | 100% (5/5) | 100% (5/5) | 82% (23/28) | 0% (0/3) | 0.4 h | 13 g | $0.19 |
| GPT-6.1 Sol | 20% (2/10) | 70% (7/10) | 60% (6/10) | 100% (10/10) | 63% (25/40) | 67% (2/3) | 0.5 h | 18 g | $0.08 |
| Gemini 3.8 Flash | 20% (2/10) | 100% (5/5) | 60% (3/5) | 80% (4/5) | 56% (14/25) | 0% (0/3) | 0.4 h | 14 g | $0.26 |
| Opus 5.5 | 30% (3/10) | 40% (4/10) | 50% (5/10) | 90% (9/10) | 53% (21/40) | 0% (0/3) | 0.5 h | 17 g | $0.29 |
| Sonnet 5.5 | 10% (1/10) | 0% (0/10) | 90% (9/10) | 100% (10/10) | 50% (20/40) | 33% (1/3) | 0.7 h | 23 g | $0.21 |
| DeepSeek V4.1 Flash | 33% (3/9) | 40% (2/5) | 20% (1/5) | 57% (4/7) | 38% (10/26) | 33% (1/3) | 1.0 h | 35 g | $0.16 |
| Haiku 5.5 | 0% (0/5) | 0% (0/5) | 40% (2/5) | 20% (1/5) | 15% (3/20) | not run | 2.0 h | 74 g | $0.04 |
The rows are the Test prints first condition: the model can ask for test prints before its final design (all three conditions are defined in How a trial runs). Each row is a model, best first; the best in each column is bold. A pass means every check passes against my recorded test prints, under the assumptions listed in the reference. Print hours and filament are slicer estimates and cost is what the API charged; each counts everything the model asked to print, test prints included, divided by the designs that pass. Nothing a model designed was printed.
Best to worst, when the model can ask for test prints (7 models, 5 or more trials on each of the four tasks):
- Gemini 4 Argon, 82% of trials pass (23/28): $0.19 and 0.4 print hours per working part
- GPT-6.1 Sol, 63% of trials pass (25/40): $0.08 and 0.5 print hours per working part
- Gemini 3.8 Flash, 56% of trials pass (14/25): $0.26 and 0.4 print hours per working part
- Opus 5.5, 53% of trials pass (21/40): $0.29 and 0.5 print hours per working part
- Sonnet 5.5, 50% of trials pass (20/40): $0.21 and 0.7 print hours per working part
- DeepSeek V4.1 Flash, 38% of trials pass (10/26): $0.16 and 1.0 print hours per working part
- Haiku 5.5, 15% of trials pass (3/20): $0.04 and 2.0 print hours per working part
Not run with test prints, ranked by their first print without them: GLM 5.3 Flash 23%, GPT-6 Luna 8%, DeepSeek V4 Pro 5%, Qwen 3.8 Flash 3%. Grok 4.7 was left out because it was too expensive to run 10 trials on.
What asking for test prints changed. Across these 7 models, 37% of first prints pass without test prints, and 53% pass when the model can ask for test prints before its final design.
| Model | First print | Reprint after feedback | Test prints first | Change from the first print |
|---|---|---|---|---|
| Gemini 4 Argon | 50% (20/40) | 80% | 82% (23/28) | +32 points |
| GPT-6.1 Sol | 50% (20/40) | 53% | 63% (25/40) | +13 points |
| Gemini 3.8 Flash | 45% (18/40) | 63% | 56% (14/25) | +11 points |
| Opus 5.5 | 40% (16/40) | 68% | 53% (21/40) | +13 points |
| Sonnet 5.5 | 35% (14/40) | 50% | 50% (20/40) | +15 points |
| DeepSeek V4.1 Flash | 39% (16/41) | 61% | 38% (10/26) | -1 points |
| Haiku 5.5 | 3% (1/40) | not replayed | 15% (3/20) | +13 points |
In 155 of these 219 trials the person answered at least one test piece with "I can't tell you how that one came out", for a size my test prints never tried. 78 of the 116 passes came after such an answer, so some may owe part of the pass to moving to a size I did print (see Limits).
Some of the gain in the reprint after feedback comes from a gap in the simulation, not from what a builder would say. When no recorded print decides a part, such as a ribbed nut pocket, the person can only answer "I can't tell you how that one came out", and a model can then switch to a shape the test prints cover, which passes. The final design is verified, but a real builder would have said whether the part worked. New passes that came this way: Sonnet 5.5 1 of its 6. Gemini 4 Argon, Opus 5.5, Gemini 3.8 Flash, DeepSeek V4.1 Flash and GPT-6.1 Sol had none.
The Estoa light box, the harder task, isn't in the four-task total: 6 models, 30 trials. 5 of 30 designs pass on the first print. In the Reprint after feedback condition, after the person says how the box came out, 12 of the 26 designs replayed (every one that had failed when the replays ran) pass. With test prints first, 4 of 18 pass (its row in the table). We ran out of time to make the test-print loop work on this task (the Estoa section).
The models design a print without really understanding the constraints of 3D printing. A lot of them design roofs over open space that can't print without supports. They design a place to slot in a nut without any kind of gap, so the nut won't go in. The models do make test prints, but they test the wrong thing. On the wheel they tested if the bearing fits, and mostly got that right. Then the wheel failed on stuff they never tested, like not leaving room for both bearings, the bolt not fitting through, or a roof printed over air.
From a clone, with uv installed and no API key:
uv syncuv run python scripts/scorecard.pyscripts/reproduce.shThe scorecard prints the table above from the saved runs. reproduce.sh checks everything end to end: tests, a regrade of every saved trial, the results page and the README's tables (about 30 minutes). Building model-written parts needs a sandbox: macOS has one built in, and on Linux install bubblewrap. The reference has every step, the Linux setup and how to run new trials.
Run a trial yourself (needs an OpenRouter key: make an account, add credits and create a key in its settings; one trial costs a few cents):
OPENROUTER_API_KEY=... uv run python -m pbe.eval --run mine --condition can_test --task nut-knob/M3 --model anthropic/claude-sonnet-5.5 --budget 0.50--condition can_test is the mode in the results: the model can ask for test prints before its final design. can_ask has no test prints.
It saves the conversation, the design and every check's result under results/runs/mine/. The tasks are tripod-plate, bearing-wheel, nut-knob/M3, nut-knob/M5 and estoa-light, and --model takes any OpenRouter model id.
Some designs still can't be graded, because no test print I've made is close enough to decide them. They count as not graded, never as a pass; the reference lists each reason, how many trials it affects and the print that would settle it.
The replies only come from my test prints. For a size I never printed, the person can only say "I can't tell you how that one came out," so models switch to sizes I did print. That wouldn't happen in real life.
In my own 3D printing workflow, I'm constantly adding to a self-learning knowledge base of mistakes that gives the model context and failure cases when 3D modeling. The models in this eval didn't have it, which leads to more failures.
Our test prints were done on my Bambu Lab X1C, only in PLA, with a single print setting. To make the eval broader, we would do test prints in different materials and on different 3D printers.
The assumptions that let us grade some of the unknowns are in the reference.
| Eval | What it tests | How this one differs |
|---|---|---|
| BenchCAD | CadQuery code from images and text, across 106 part families | Grades whether a part works with real hardware on a real printer, not whether it matches a reference shape |
| CADEngBench | Parametric parts graded on valid solids and design-for-manufacturing checks | Fit limits come from test prints on one printer, and the model can ask for measurements and hear how a print came out |
| Hephaestus-CCX | Requirements written as typed checks | Borrowed: each task's requirements are typed checks (requirements.yaml) |
Code from BenchCAD (MIT) is used for its CadQuery sandbox; CADEngBench has no licence, so its layered grading is reimplemented and cited. Prior art has the full list.
- Models: tested through OpenRouter's API.
- Building and checking parts: CadQuery (the models write their parts in it), trimesh, manifold3d and shapely for the geometry, bubblewrap or
sandbox-execto run model code in a sandbox. - Print time and filament: Bambu Studio's slicer, offline.
- Test prints: a Bambu Lab X1C, PLA.
The full results, every check, what went wrong task by task and the prior art are in docs/reference.md. MIT licence, in LICENSE.
















































