What survives a hostile search of cheap public data?

9 featureFeature The thing that might predict: a weather anomaly, a positioning z-score, a funding rate, a team's recent form. Transformed and lagged so it was knowable before the target. glossary families × 16 targetsTarget The thing being predicted: the forward return of a price over 5, 10 or 21 days, or a game's margin, win, or result against the line. glossary × horizonsHorizon How far ahead the target looks. A 21-day horizon means the return from today to 21 trading days from now. glossary 5/10/21 days × lagsLag How old the feature is allowed to be. A lag of 3 days means the feature value used is the last one published at least 3 days before the target's start. glossary 0/1/3/5 days, pruned by a physical pair policy (weather may touch commodities, not crypto). Enumerated before any result is seen; that is the pre-registrationPre-registered Written down before the result was seen. The miner's grid of features, targets, horizons and lags is enumerated in code before any test runs, so the search cannot be steered toward what looks good. glossary.

A non-overlapping SpearmanSpearman correlation Correlation on ranks instead of raw values: do high values of one series go with high values of the other? Runs from −1 to +1 and ignores outliers' size, only their order. glossary on at least 120 as-of-joinedAs-of (point in time) Reading the data as it looked on a given date: only observations published by then, and the version that was current then. Backtests that read today's cleaned data are reading the future. glossary rows, Benjamini-HochbergBenjamini-Hochberg A rule for testing many hypotheses at once: sort the p-values, and accept only the ones small enough given their rank, so the share of false discoveries stays below a target (here 10%). Without it, screening 34,000 specs would 'find' thousands of relationships by chance. glossary within each target-and-horizon familyFamily The group of tests corrected together: every feature tested against one target at one horizon. The family size is stored with each result so nobody can shrink it later. glossary at qq-value (false-discovery rate) The p-value corrected for how many tests were run in the same family. q = 0.05 means that among findings called significant at this level, about 5% are expected to be flukes. The Benjamini-Hochberg procedure computes it. glossary < 0.1. Survivors get HAC regressionHAC (Newey-West) errors Overlapping forward returns are not independent: tomorrow's 21-day return shares 20 days with today's. HAC standard errors widen the uncertainty to account for that, so the p-value is not fake-small. glossary, an embargoedEmbargo A gap of one horizon between the training and test data in a walk-forward, so a 21-day return that starts in training cannot end in test and leak the answer. glossary walk-forwardWalk-forward Fit on the first chunk of history, test on the next, then grow the training window and repeat. Four folds here, with an embargo gap so overlapping returns cannot leak across the boundary. glossary scored as excess over the fold meanOOS excess The average target return when the rule fired, minus the average over the whole test period. Measuring against zero lets a rule 'win' just because prices drifted up or home teams win more often. glossary, a regime splitRegime split Testing the relationship separately in different periods (calendar thirds). A relationship that only exists in one regime is probably that regime, not a law. glossary, influence testsInfluence (leave-one-out) Drop each observation in turn and recompute. If one extreme day is doing all the work, the result flips when it is removed, and it fails. glossary and, where a price exists, a cost-adjusted backtest.

One grading path: the reviewer checklistReviewer checklist Seventeen questions a hostile statistician would ask (sample size, pre-registration, multiplicity, out of sample, costs, look-ahead, survivorship, revisions, regimes, autocorrelation, single observations, confounders), each answered from the numbers, never from prose. glossary, then the holdoutHoldout The newest quarter of every sample, set aside before any screening or fitting. The development rule is applied to it once, at the end. Passing means the relationship kept its sign and beat the holdout's own average. It is the closest thing to a fresh test the data allows. glossary must agree. robustGrade robust: significant after correction, out of sample, costs, stable, holdout agrees. promising: significant with out-of-sample excess, holdout not decisive. weak: screened but failed out of sample or the holdout. failed: was promising, no longer. Retired rows stay visible. glossary needs significance after correction, out-of-sampleOut of sample (OOS) Tested on data the rule was not fitted on. We fit a rule on earlier data, then score it on later data it never saw. Anything can fit the past; only out-of-sample results count. glossary excess, costs, regime stabilityStability Split the history into three periods: in what share of them does the relationship keep the same sign as the full sample? 1.0 means all three agree; 0.33 means it flips around. glossary and a passing holdout. Game insights that predict the result rather than the result against the line are tagged baseline_onlybaseline_only A game insight that predicts the result (who wins, by how much) rather than the result against the betting line. Real, but the line already prices it, so it is not an edge. glossary: real, but not edge.