The difference between holding out trials and holding out people

The research question is whether a decision model that performs well on unfamiliar trials from previously observed participants can be expected to work for entirely new people. These are different kinds of generalization. A study may contain hundreds of heart-rate, reaction-time or gaze recordings from each participant and still have only a small number of independently sampled people. When observations from one person are distributed across training and test sets, the model may recognize stable person-specific patterns instead of identifying a generalizable psychological state. When a Model Knows Too Much discussed data leakage; cross-participant validation adds the crucial question of whose information has remained unseen.

Varoquaux's analysis of cross-validation in neuroimaging showed why small samples create considerable uncertainty in estimates of predictive performance [1]. A model may appear excellent in one fold yet generalize poorly to a new group. Poldrack and colleagues argued for transparent, reproducible neuroimaging pipelines so researchers can evaluate how analyses were performed [2]. Neither source proves that an NSI model can or cannot work. Their methodological relevance is that a predictive claim must specify the intended population and evaluation procedure before a model's accuracy is announced. Reliability on repeated observations within one person is not interchangeable with success across people.

How identity and recording conditions create misleading prediction

Imagine twenty people providing two hundred recordings apiece. The resulting four thousand records may look like a large training dataset, but the participant-level sample is twenty. A classifier can potentially exploit stable differences in heart-rate baseline, electrode placement, habitual movement, skin characteristics, room lighting or camera calibration. If a participant-specific feature correlates with the study's labels, the model may achieve high accuracy without discovering anything about threat response, recovery capacity or decision quality. Random trial splitting does not prevent this problem when the same participant is present on both sides of the boundary.

Another danger arises from preprocessing. Standardizing features across the complete dataset before separation allows information from the test set to influence model preparation. Choosing sensors after reviewing every participant's performance has the same effect. More subtle leakage may occur when repeated sessions are mixed, or when labels depend on information that would not exist at the time of an actual prediction. Button and colleagues described how small-sample neuroscience results can overestimate effect sizes and undermine reproducibility [3]. A model that appears accurate in a convenience sample should therefore be evaluated as a provisional finding rather than a verified population tool.

A validation design that measures actual transfer

One defensible evaluation keeps every recording from particular participants out of model training, normalization and feature selection. Researchers can use nested grouped cross-validation for development and retain a final untouched cohort for evaluation. They should separately report generalization to unseen sessions, unseen days and unseen people, because each represents a distinct practical claim. A person-specific model may be defensible when its intended application is expressly limited to that person, but developers should not describe that result as universal. What Would Validate NSI? explains why baselines, external samples and replication are indispensable before any public performance claim.

Evaluation should include the prevalence of each behavioral outcome, participant-level uncertainty intervals and sensitivity to equipment changes. Reporting only average accuracy can conceal a classifier that performs well for a few participants while failing for others. The basic comparison might be a majority-class predictor or a straightforward behavioral measure. An advanced model should demonstrate value beyond those simpler alternatives and preserve the costs of false alarms. Before the Result describes how preregistered definitions protect against choosing a flattering metric after seeing test outcomes. All model revisions and exclusions should be reported, especially when a previously held-out cohort has influenced development decisions.

Why external validity still remains even after good cross-validation

Passing a grouped validation test does not automatically transfer a finding to live financial markets. A laboratory task may differ from actual trading in incentives, interface complexity, emotional salience and uncertainty. Market outcomes also depend on price movements unrelated to the individual's cognitive performance. Therefore an NSI research design should measure a clearly defined behavioral decision process separately from whether a trade happened to make money. The existence of an attractive physiological correlation does not establish a causal mechanism, and a usable instrument would need reliability, construct validity and independent evaluation under the conditions in which it is intended to operate.

The conclusion is that a model must be tested against the particular claim its creators want to make. Unseen trials, unseen sessions, unseen people and unseen environments are separate evidentiary challenges. Negative results are scientifically valuable because they reveal boundaries of a proposed relationship. NSI remains an interdisciplinary research proposal rather than an independently validated medical diagnosis or financial prediction system. It should not be presented as a tool that detects fear or predicts XAU/USD merely because a classifier achieved high internal accuracy. Good validation reduces the distance between a compelling demonstration and knowledge that others can reproduce.

Scientific bibliography & sources

  1. [1] Varoquaux, Cross-validation failure: Small sample sizes lead to large error bars (2018). https://pubmed.ncbi.nlm.nih.gov/28655633/
  2. [2] Poldrack et al., Scanning the horizon: transparent and reproducible neuroimaging research (2017). https://pubmed.ncbi.nlm.nih.gov/28053326/
  3. [3] Button et al., Power failure: why small sample size undermines neuroscience reliability (2013). https://doi.org/10.1038/nrn3475

Research note

NSI is a proposed research framework, not an independently validated medical or financial prediction system. This article is research and education, not individualized medical or investment advice. Market observations are timestamped historical context, not live prices.