How we test
What this page can do: show how we test our own strategies, so that “looked good” is not mistaken for “is good”. Every step has a real example with a date and a hash. What it cannot do: promise that a strategy passing these steps will make money in the future. So far: 24 preregistered families checked, 0 passed. "0 passed" does not mean that no edge exists. The cards are in the register, verdicts section.
The same rules are built into Tape Check: whoever stamps a forecast there commits in advance, and the forecast is later scored against the zero-skill forecast (methods paper ZSB-1).
-
Fix it in advance
Before we see a single number from real data, these are fixed: hypothesis and mechanism (who has to trade, and when), the exact rule, markets, periods (development, validation, holdout), cost per trade, minimum number of trades, error rate 1 % per family, and the conditions under which a family passes. Plus a cost check without price data: can the gross effect we would need cover the costs at all?
Example, round 5: the cost check removed two candidates before any run; for one of them costs alone would have been 0.8–0.9 R per trade. Nothing was computed unless it passed the check. Source: round 5 cost pre-check, 17 Oct 2026 10:26–10:31.
-
Time-stamp
The fixed text gets a fingerprint (sha256) and a time stamp in the bitcoin blockchain (OpenTimestamps). Nothing may be computed until the line “round x – before any run” is in the log.
Example, round 5: fixed 17 Oct 2026 11:50, sha256
531a2b3343b92c5e3d960784a365b2e7b559bb577d49b190713fc6a82f68f733, OpenTimestamps. The log line was in place at 22:02; until then no test ran on real data. -
Null seeds
Before the real run, the code runs on artificial data without an effect. There it may report “hits” at most as often as the error rate allows. On artificial data with a planted effect it has to find the effect. This check never touches real data.
Example, round 5: family “thin order book”: 0 of 24 null seeds below the threshold (smallest p 0.25), planted effect found. 24 runs are too few to check a 1 % error rate precisely; a smallest p of 0.25 over 24 runs is unusually high, so the test was probably too strict here. Source: null-seed run 17 Oct 2026 11:19–11:50.
-
Holdout exactly once
The rule may be built on the development data. We look at the validation data once. We open the holdout (the most recent years) exactly once, with a log line before, and recompute nothing afterwards. If we find a bug in our own code beforehand, it is corrected openly, with the reason and a new hash, before any result is visible.
Example, round 4: log line “holdout opened” 17 Oct 2026 09:26, holdout runs afterwards, once each. Example, round 5: before the real run our code counted Sunday rows; the fix was documented with a diff and a new hash before any result was visible. Holdout computed once on 18 Oct 2026 11:50–11:56: n 301, mean −0.065 R.
-
Forecast log
Before every round we write down how likely we think an outcome is, with a time stamp. Later the forecast is scored (Brier score). That shows whether we trust ourselves too much.
Example, round 4: forecast before the run: “at least one of five families passes” at 0.35. Outcome: no (0 of 5). Example, round 3: for the market weighting, “fails, noise” was in the log beforehand; that is what happened.
-
Publish whatever comes out
Every result goes on a card, the same size whether good or bad. Nothing is added afterwards: no new thresholds once the result is known. What could not be tested is called “not testable”, not “failed”.
As of 19 Oct 2026: 24 preregistered families checked (19 failed, 1 undecided, 2 not testable, 2 still running), 0 passed; 43 cards in the graveyard. One stream is on probation (holdout n 348, +0.138 R per trade, t 1.65, not significant; last 12 months −0.05 R at n 72; checkpoint 28 Feb 2027). All cards: register, verdicts section.
What these steps do not do
- They prevent self-deception, not losses. A family that passes all six steps has a finding, not a guarantee.
- We do not correct across families: with 24 families at 1 % each, one chance hit is expected about 21 % of the time; a future “passed” will be shown with that figure.
- All rounds share the same holdout period; later ideas know earlier results.
- The holdout is untouched exactly once: we open it a single time. For variants of our own stream, the period from 10.2025 plus the future remains.
- We publish prereg texts and code with each method version; until then the hash and time stamp show that the text existed at that time. That nothing was computed before is recorded in our log; the stamp cannot prove it. Rounds 1–3 carry a hash but no time stamp.