One Significant Result Among Many: Multiple Testing in Neurofinance Research

The research question: how many opportunities produced the result?

A study may report one striking association while having examined many physiological measures, time windows and behavioral outcomes. The research question is how much evidential weight that selected association deserves given the full analytical search. The answer cannot be read from a single p-value in isolation. A result found through a broad exploratory process has a different interpretation from a precisely specified test evaluated once under its declared assumptions. Simmons, Nelson and Simonsohn demonstrated how flexibility in data collection, analysis and reporting can increase false-positive findings. Their work used simulations and experiments to show the consequences of undisclosed analytical choices. [1] The lesson is not that every positive finding is false. It is that a conventional significance threshold does not protect an analysis whose effective search process is hidden. Readers need to know which decisions were made, when they were made and what alternatives were examined.

The search can be implicit rather than a visible list

Gelman and Loken discussed data-dependent analytical paths: choices can depend on what the researcher sees even without a deliberate plan to test dozens of hypotheses. [2] A person might select one transformation because the distribution looks unusual, one subgroup because its pattern seems clearer and one time window because the effect appears strongest there. Each choice may sound defensible in isolation. Together, they create a path shaped by the data that should not be described as a fully prespecified test. In neurofinance, the number of possible comparisons can grow quickly. Researchers might consider heart-rate metrics, skin conductance, reaction time, confidence and several definitions of decision quality. The appropriate response is not to avoid rich data. It is to distinguish primary questions from exploratory analyses and to document the selection process. Otherwise a reader cannot tell whether the highlighted result was the intended target or the survivor of many unreported alternatives.

Different error controls answer different questions

Benjamini and Hochberg developed an approach to controlling the false discovery rate in multiple testing, with a proof under the assumptions stated in their original paper. [3] False discovery rate concerns the expected proportion of false rejections among rejected hypotheses, under the relevant framework. It is not the probability that a particular published claim is false, and it is not identical to controlling the probability of any false positive within a family of tests. Choosing an error-control strategy requires defining the family of tests and the purpose of the analysis. A broad discovery screen and a single confirmatory primary outcome need not use the same approach. Dependence among tests, model assumptions and subsequent selection also matter. A correction applied only to the small subset of results chosen for the paper cannot automatically account for a larger undocumented search. Statistical labels should describe an actual procedure rather than serve as a general assurance of rigor.

Design an NSI analysis that remains interpretable when results disappoint

A prospective NSI protocol could identify one primary behavioral outcome and a limited set of prespecified predictors, then label additional analyses as exploratory. It should record how physiological artifacts are handled and whether alternative preprocessing choices were examined. The discussion of EEG error-related negativity shows why these choices matter: a derived signal depends on a measurement and processing workflow. Selecting the workflow because it produces the desired relationship can entangle measurement development with hypothesis testing. Exploratory findings can still be useful if they lead to a new test on independent data. The report should describe the original search and avoid treating a later, more specific hypothesis as though it had existed from the beginning. This connects to the requirements for NSI validation and replication. NSI remains a proposed interdisciplinary framework, not a validated medical or financial prediction system. A collection of statistically significant associations would not, by itself, establish diagnostic value, practical usefulness or profitable forecasting.

A strong report also includes effect estimates and uncertainty, the direction and size of results that did not meet the selected threshold, and the limitations of the sample. A statistically detectable relationship may be too small or unstable to support a useful decision tool. Conversely, a non-significant result may be inconclusive if the study is imprecise. Neither interpretation should be reduced to a binary badge. The design and the range of compatible effects deserve attention alongside the statistical test. The conclusion is that a significant result must be read together with the process that found it. Prespecify primary questions, disclose analytical flexibility and use an appropriate strategy for multiplicity when the design requires it. Treat exploratory patterns as proposals for further testing rather than completed validation. This approach does not eliminate uncertainty, but it makes uncertainty visible and prevents a large search from being mistaken for a single decisive experiment.