Multiple testing

Throw enough darts blindfolded and one of them will land near the bullseye. Nobody would call that aim. Multiple testing is the same situation in research: every extra version of an idea, a different window, a different filter, a different holding period, is another dart. The more versions you try, the more certain it becomes that at least one of them looks good by chance alone, even if none of them has any edge.

The danger is not the trying, it is the hiding. A result that came from the twentieth attempt and a result that came from the only attempt look identical on a chart. Only the count of attempts tells them apart, and that count is almost never printed next to a pitch. The count also adds up over time and across people: last year's variations, and every paper written about the same family of ideas, belong to the same pile of darts.

That is why the logbook writes each idea down before testing it and keeps a running tally for every family. A new test has to clear a bar that rises with every earlier attempt, the rule explained under correction for looking twice. In the momentum entry that tally is what closed the family: the last written reason to look again was spent, and a fresh attempt would have to pay for all the earlier ones.