How do you prove a track record wasn't cherry-picked? Publish the kill condition first

G10Grid · research notes

You can't prove it afterwards. Once results exist, every choice you make about how to report them is contaminated by knowing what they are, and no amount of disclosure fixes that. A person choosing a horizon after seeing the outcomes will pick the flattering horizon every time, and they will not even feel dishonest doing it. That is what makes it hard to catch.

The only thing that helps is moving the choices to before the data. Medicine worked this out decades ago and calls it pre-registration. Trading mostly hasn't, which is odd, because the incentive to fool yourself is stronger here than in almost any field.

Concretely, a pre-registration is a document with a date on it, written before you look, that fixes five things.

The horizon. One number, decided in advance. Mine is four weeks from the week the signal was logged. Holding periods are still recorded per signal and can be reported separately, but never as the headline, because a free choice of horizon after the fact is a free choice of result.

The metric, including costs. Not "did it go up" but a specific calculation, net of spread and carry, with the risk unit named. If costs are not in the metric they will be discovered later, always at the moment they would have changed the answer.

The benchmark, and it must not be a coin flip. Beating fifty percent is not evidence of anything in a market with drift and carry in it. The comparison has to be something free that a person could run in two lines of code. Mine is the direction implied by carry plus twelve-week momentum, on the same pairs, same horizon, same costs. The whole question being asked is whether an expensive fundamental layer adds anything over that, and if it doesn't, the machine is a costly reproduction of something free.

The effective sample size, which is where most published records quietly break. Six signals in one week that are all short the same currency are not six bets, they are one bet written down six times. Counting them as six is how a person gets to a hundred observations in a month and a confidence interval that means nothing. So observations get clustered by week and dominant currency, and a cluster counts once.

And the kill condition. The number that would mean the thing is dead. This is the one everybody skips, and skipping it is the whole game, because a study with no failure condition cannot fail. Mine is written out: if the point estimate of the difference against the benchmark is zero or below once a hundred clusters have accumulated, the conclusion is that the fundamental layer adds nothing over carry and trend, and that conclusion gets published exactly like a positive one would. There is also a date lock. The main metric does not get looked at before the end of December 2026 or a hundred clusters, whichever lands later. I will say now that at the current rate of signals the cluster count is the binding one and the date will almost certainly move. Moving the date is allowed and moving the threshold is not, which is the only reason that admission is not just an excuse.

Two things I got wrong about this, in case they save someone else the trouble.

The first is that I assumed a locked file was enough. It isn't. A hash protects the code, not the behaviour. I had a scoring engine that was byte-identical across a change and still produced different tiers, because one of its input layers had gone dead and another was biased, and neither is visible in a checksum. So the pre-registration has to version the model separately from the code, and a model change has to reset the sample counter to zero even when not a single rule in the document was touched. I have already had to do it. The honest footnote is that it cost nothing: when I counted the rows in the pre-registered sample at the moment of the reset there were zero of them, so there was no result sitting there to be biased by the decision. That is the cheap version of this problem and I would not have found out how cheap it was without counting. Separately, the published research ledger holds eleven signals from an older version that the pre- registration excludes on its own terms, which is not the same thing and I do not get to claim credit for both.

The second is that a pre-registration is only real if you write down the peeking too. Checking early is not forbidden and pretending you didn't is. A second forward test has been running here since 24 August, with a fifty-two week minimum before the result may be read, and the rule attached to it is that looking before then gets logged as a peek. Not banned. Logged.

None of this makes a record trustworthy. It makes it falsifiable, which is a much weaker property and the only one available to someone without a broker statement and an auditor. If you are reading somebody's results and there is no document older than the results, the honest position is that you are looking at a story, not a test. That includes mine, for everything logged before the document existed.