How we test our own ideas

How a claim is tested: timeline Five steps in order: hypothesis written, kill criteria frozen, hash anchored in a public log, cases scored blind, results published whatever they show. Hypothesis written Kill criteria frozen Hash anchored in a public log Cases scored blind Results published whatever they show
The order matters: the hypothesis and the conditions that would kill it are fixed before any case is scored.

We try to break an idea before we believe it. These are the rules we follow when we test a claim, and why.

We write down what would kill the idea first

Before we look at any data, we publish the hypothesis, the exact way cases are collected and scored, and the results that would make us abandon the idea. These kill criteria are fixed in advance. If one fires, the stated consequence follows, and we do not renegotiate it after seeing the numbers.

We also write down what we expect. Our published expectation for the current study is that the most probable outcome is a documented kill of the claim that the idea is new and worth building. Recording that in advance is what stops anyone, including us, from saying later that we always thought it was promising.

The scorer and the checker are different models

One AI model scores each case. A different AI model audits those scores, paying particular attention to the calls that favour our hypothesis, because those are the calls most likely to be too generous. A person settles the one question on which the two cannot agree.

Both models come from companies that have a stake in some of the cases. So we added a conflict-of-interest rule: when a case involves a model from either company and is scored as a match, a person reviews it as well. No reviewer is perfectly neutral. The rule lowers that risk; it does not remove it.

We fix the reference before we score

Where a judgment can lean on an outside reference, we name the reference in advance. For example, whether a published detection rule would have caught an event is checked against a pinned, named ruleset, not recalled from memory. If we later build a detector, its decisions are meant to be deterministic and explainable, with no AI model in the decision path.

Changes are dated and never rewrite history

Once the plan is frozen we do not edit it. If something must change, we add a dated amendment that says what changed and why, and the original text stays in place. Amendments include our own mistakes. When we found that a ruleset we had pinned was effectively empty for our purpose, because its newer release had renamed the relevant tags, we did not quietly swap it. We published the error, kept the original pin, and recorded the corrected reference.

Negative results are published

A result that kills our idea is a success for the method, and we publish it. We also say what the data cannot show: our collection of cases is built by search, so it can tell us whether a pattern exists, not how often it happens.

Claims must match artifacts

Every statement should point to something a reader can check: a quoted passage, a source link, a logged search, a test result. If we cannot point to the artifact, we do not make the claim. We also record what we did not run, what we could not read, and what we know only second-hand.

What you can check

The pre-registration is public, and its digest is anchored in Sigstore Rekor (log index: 3710392060), a public, append-only transparency log. You can look the entry up at https://rekor.sigstore.dev/api/v1/log/entries?logIndex=3710392060 (the address returns JSON, not a web page). Anyone can confirm that the text existed in this form before any case was scored, and that it has not changed since.