Correction for looking twice

Suppose an idea has no edge at all. A test set at the usual level will still call it "real" one time in twenty, just by chance: that is the false positive rate of the test, five percent under a true null. Test twenty independent versions of ideas with no edge and the chance that at least one of them passes is about sixty four percent. Not a certainty, but far from the one in twenty you were promised. When the versions overlap, the arithmetic changes, but the direction does not: more looks, more false positives.

The correction is to raise the bar in proportion to the number of looks, or to compare the winner against the best that the same number of empty ideas would produce by luck. The logbook counts every attempt, including the ones from other weeks in the same family, and the newest test pays for all of them. It is why a family with several empty entries becomes very hard to reopen: the bar has climbed.

Sellers of strategies never show you how many versions they tried before the one on the chart. Testing many ideas and keeping the best-looking result, without accounting for that selection, is how a backtest gets overfitted. The correction is the price of that silence.