One dataset can generate more than one defensible answer

Researchers often describe the processing of neural data as if raw measurements mechanically yield one statistical result. In reality, functional imaging and other high-dimensional signals usually require a sequence of decisions about preprocessing, exclusions, alignment, modeling and statistical thresholds. Some choices are plainly justified by quality control; others may have several reasonable alternatives. The research question is how much those alternatives change the conclusion. This is not an argument that every analysis is arbitrary. It is an argument that scientific confidence must account for sensitivity to analytic choices, especially when a proposed decision-state indicator is being evaluated. Carp submitted the same functional MRI experiment to numerous combinations of analysis procedures and demonstrated substantial variation in resulting maps and statistical conclusions [1]. The experiment illustrates why a single attractive image of brain activation cannot be treated as the unavoidable output of the raw observations. A researcher might achieve different reported regions or significance levels while using analysis strategies that appear defensible in isolation. These differences have implications for replication, but they do not prove that every reported neural response is false.

This is related to NSI's earlier discussion of [Temporal Leakage: How a Nervous-System Model Can Learn Tomorrow by Accident](/publications/nsi-2026-10-30-research-03). Temporal leakage concerns the transfer of forbidden future information into a model's evaluation. Analytic flexibility concerns the range of defensible workflows from the same dataset. Both can distort confidence, but they are different methodological failures and require different controls.

What happens when independent teams analyze exactly the same observations?

Botvinik-Nezer and colleagues coordinated a large multi-team investigation in which independent groups analyzed the same neuroimaging data while testing the same predefined hypotheses [2]. The teams selected different workflows and produced meaningfully varied outcomes for some hypotheses. Their results do not justify treating scientific disagreement as mere incompetence. The teams operated within a genuine space of methodological choices, and intermediate products could be similar even when final thresholded claims diverged. The study provides unusually direct evidence that investigator decisions can materially influence the meaning of a reported statistical finding. A fixed data collection does not eliminate analysis uncertainty. Decisions about motion correction, nuisance variables, spatial smoothing, model specification and multiple-comparison adjustment can interact. If researchers report only one path, readers may be unable to judge whether the result is robust. Sharing analysis code and configurations can make the path reproducible, but reproduction of one exact pipeline is not identical to robustness across reasonable pipelines. Both matter. A model that repeats one result reliably may still depend strongly on an unjustified preprocessing choice.

In neurofinance, the analogous issue might arise when a laboratory records physiological features alongside a risk-choice task. Should noisy trials be excluded? How should an individual's baseline be normalized? Which behavioral labels count as impaired decisions? Which time window precedes the response? Choosing these answers after examining the results invites optimistic interpretations. A serious NSI project would specify them in advance or disclose a sensitivity analysis. It would not advertise the most favorable result from many silent alternatives.

Multiverse analysis and the boundaries of robustness

Steegen and colleagues introduced a multiverse analysis approach that systematically examines the conclusions obtained across alternative reasonable data-processing choices, illustrating the method with empirical data [3]. The logic is transparent: when more than one construction of a dataset is defensible, examine the family of results rather than presenting one convenient member as uniquely correct. A multiverse does not make every specification equally plausible or exempt the researcher from theory. Implausible alternatives should not be included merely to inflate the appearance of uncertainty. A good multiverse protocol must state which alternatives were justified, which combinations were logically incompatible and how outcomes would be summarized. It should also separate preprocessing options from major changes in the research hypothesis. If one pipeline studies attention and another studies reward learning, combining their results as if they were interchangeable would create conceptual confusion. The purpose is to expose analytic sensitivity within a defined question, not to generate endless possibilities until every interpretation seems equally valid.

Comparable logic applies to proposed physiological indicators. An NSI score may depend on filters, feature extraction, normalization and model thresholds. If changing one defensible assumption reverses a conclusion, that sensitivity belongs in the research report. If results remain stable across plausible alternatives and an independent external sample, confidence may improve—but such evidence is not a substitute for prospective tests. See [A Confidence Percentage Is Not Proof of Calibration](/publications/nsi-2026-10-30-research-04) for the separate problem of how probability claims must be evaluated.

A reproducibility protocol for proposed NSI studies

Before collecting data, researchers should register the question, target construct, planned exclusions, preprocessing steps, participant-level splits and primary effect measure. Each analysis must preserve the separation of training and evaluation data. Artifact identification, missing observations and sensor failures should be handled according to rules that do not depend on which outcome makes the final plot most impressive. The protocol should also specify uncertainty intervals and all relevant covariates, not just an attractive p-value. When exploratory choices are unavoidable, researchers should preserve their record and publish a reasonable range of analyses. External teams should be able to reproduce the published pipeline and test alternatives without reverse engineering hidden settings. The related NSI article [Cross-Participant Validation](/publications/nsi-2026-10-29-research-01) explains why testing entirely new people adds a separate requirement beyond analytic robustness. Stability on one dataset is weaker than generalization to new participants, devices, sessions and environments.

The conclusion is that a neural result is partly a product of data and partly of declared analytical decisions. The empirical findings of Carp [1] and the multi-team evidence [2], together with multiverse methodology [3], justify systematic sensitivity reporting. They do not invalidate neuroscience or establish a financial forecasting method. NSI remains a proposed interdisciplinary research framework, not a clinically or financially validated prediction system. Its most defensible standard is transparency about all material paths from observation to claim.

Scientific bibliography & sources

  1. [1] Carp J. On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments. Front Neurosci (2012). https://pmc.ncbi.nlm.nih.gov/articles/PMC3468892/
  2. [2] Botvinik-Nezer R et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature (2020). https://www.nature.com/articles/s41586-020-2314-9
  3. [3] Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W. Increasing Transparency Through a Multiverse Analysis. Perspectives on Psychological Science (2016). https://www.psychologicalscience.org/journals/perspectives/1745691616658637/

Research note

NSI is a proposed research framework, not an independently validated medical or financial prediction system. This article is research and education, not individualized medical or investment advice. Market observations are timestamped historical context, not live prices.