When a Model Knows Too Much: Data Leakage in NSI Prediction Research

The research question: what information was legitimately available?

A model can produce impressive historical predictions for the wrong reason: it has access, directly or indirectly, to information unavailable when the prediction would have been made. The research question is whether the evaluation reproduces the intended information boundary. This is not merely a software issue. It is a question about what the model is being asked to predict, for whom, at what time and with which observations already available. Kaufman and colleagues described leakage as the introduction of target-related information that should not legitimately be available for the data-mining task. [1] The wording is useful because it focuses on the intended use rather than one particular coding mistake. A variable can be legitimate for describing an event after it ends and illegitimate for predicting the event before it begins. The same dataset can therefore support one research question while invalidating another.

The training boundary begins before the model fit

A common error is to select features or calculate transformations using the full dataset, then divide the resulting data into training and test sets. The later split does not remove information already borrowed from the test observations. Scikit-learn's documentation illustrates this problem and recommends learning preprocessing operations from the training portion only, including within cross-validation. [2] The principle applies to imputation, scaling and feature selection whenever their learned properties could influence the evaluation. In a proposed physiological study, a researcher might choose the signal features most associated with the outcome across all participants before evaluating a predictive model. That procedure lets the held-out outcomes influence what the model receives. A pipeline can help enforce the boundary, but only if the researcher places every learned selection step inside it. A technically correct estimator cannot repair a dataset whose construction has already used the answers.

Time and participant identity create additional boundaries

Randomly splitting rows can be inappropriate when repeated observations from the same person appear in both training and evaluation sets. Whether that is leakage depends on the intended claim. Predicting a new observation for a known person differs from predicting performance for an unseen person. A model may exploit stable individual characteristics without learning a relationship that generalizes across people. The evaluation should match the deployment question rather than using whichever split produces the strongest score. Kapoor and Narayanan examined leakage in machine-learning-based science and proposed a taxonomy and reporting tools to help identify it. [3] Their analysis shows why apparently sophisticated models still require scrutiny of data construction and evaluation. For time-dependent decisions, timestamps add another boundary. A physiological summary calculated over an interval extending beyond the decision cannot be used honestly as a predictor available before that decision. Clear feature availability times are therefore as important as the mathematical form of the model.

A defensible evaluation begins with a prediction contract

A prediction contract would state the target, prediction time, eligible input window, population and intended use. It would specify whether the goal is within-person monitoring, generalization to new people or forecasting later periods. The dataset would then be assembled to respect that contract. For an NSI study, a model intended to anticipate a rule departure should not include a self-report completed after the departure unless the claim is explicitly retrospective rather than predictive. The requirements for validating NSI include more than holding back a final dataset. Researchers must prevent repeated access to that dataset from turning it into another development resource. Hyperparameter choices, feature revisions and threshold selection belong to the development process. Once the final evaluation influences those choices, a fresh independent evaluation is needed for the revised claim. NSI remains a proposed framework, not a validated financial or medical prediction system, and no performance figure is asserted here.

Measurement quality also remains independent of leakage control. A perfectly separated dataset can contain an unreliable construct, as discussed in the limitations of skin conductance as an emotion measure. Preventing future information from entering the model does not establish that the model measures fear, decision quality or any other psychological concept. Researchers should assess the measurement, the prediction procedure and the practical meaning of performance as separate parts of the study. An informative report would describe the split strategy, participant overlap, temporal ordering, preprocessing boundary and the number of times the evaluation data were consulted. It would compare against simple baselines and report uncertainty rather than only the best metric. If a leakage correction removes an apparent advantage, that is a useful result. The correction shows that the earlier performance estimate answered an easier or different question than the intended application.

The conclusion is that a model must be evaluated with the knowledge it would actually possess at the moment of use. Establish that boundary before preprocessing, preserve it across people and time where the claim requires it, and document every step that learns from data. Only then can a predictive result begin to support a claim about future decisions rather than a sophisticated reconstruction of information already known.