OpenAddict

Context

Advice about how to prompt a model circulates faster than anyone checks it. A technique gets posted, it sounds reasonable, it gets repeated, and it becomes something people believe without a single run behind it.

I built OpenAddict to check. It is an independent evaluation lab that takes community techniques and tests them against pinned model versions across three vendors, and it publishes what it finds including the cases where the technique does nothing.

That last part is the reason the lab exists. A null result is the most useful finding available about a piece of prompting folklore and the least likely to be published, because it makes for a worse post than a claimed improvement.

Owned

Everything: the claim set, the harness, the scoring design, the run infrastructure, and the published write-ups. Sole builder.

Architecture

A claim becomes testable by being turned into a pair: one technique against one pinned model version. 18 claims tested^{Source note 2} against 3 models^{Source note 3} gives the 54 claim and model pairs^{Source note 1} the lab runs, and it records every run rather than every conclusion, which is what makes a verdict re-derivable instead of asserted. Two runs have produced 4,938 records^{Source note 4} in total, the launch run and wave two.

Model versions are pinned because an unpinned comparison measures the vendor's release schedule rather than the technique. A result against "Claude" means nothing six weeks later. A result against a named version stays true.

Scoring runs two ways. Most claims are scored by deterministic functions, which is cheap and repeatable. A minority need rubric grading, which the methodology page records as costing about twenty times a deterministic record, and that ratio is why the split exists rather than grading everything the expensive way.

Thresholds are frozen before the runs, and the methodology page is versioned. Both exist to stop the one failure mode an evaluation lab cannot survive: deciding what counts as an improvement after seeing which way the numbers went.

Decisions

• Pinned model versions, always. An unpinned test measures the release schedule.
• Null results published as first-class findings. The whole point. A lab that only publishes wins is a marketing channel with a methodology page.
• Frozen thresholds before running. Deciding the bar afterwards is how honest people produce dishonest results.
• Deterministic scoring by default, rubric only where necessary. The cost ratio makes rubric grading a budget decision, so it is spent where a function cannot do the job.
• Per-vendor configuration caveats recorded rather than smoothed over. The vendors are not interchangeable, and pretending they are would make every cross-vendor claim softer than it looks.

Outcome

The lab is running and publishing, with the methodology and the per-run record counts public so a reader can check a verdict rather than take it.

The finding I would point at first is cross-vendor: permitting a model to abstain measurably reduces fabrication on all three vendors. It is the kind of result that is only worth anything because the runs behind it are recorded and the thresholds were set in advance.

Revision

I would have versioned the methodology page from the first run rather than adding versioning after the launch run, because the value of a frozen threshold is only visible if you can show when it was frozen. I would also have decided the record-count reporting convention before publishing two runs, since a corpus that grows in waves needs its scope stated every time a number from it is quoted.