From the Laramide Intelligence Blog
Much ado
Have you gotten the impression that new software is likely to affect how well we can use data to make scientific predictions? By software I mean systems that implement machine learning algorithms, do automation, and are backed by substantial compute resources, systems that are commonly referred to as AI these days. It seems likely that you have come across sources that have implicitly or explicitly promoted the idea that machine learning, plus automation agents, harnesses, and whatnot, will allow us to analyze data much more effectively, to learn considerably more. For example, prominent AI boosters claim that AI will help us cure cancer and all diseases in short order. Meanwhile, large language models, agents, and harnesses are increasingly effective for automating code generation and some mathematics. Investors have bet that these systems will generate lots of profit for the associated companies and many hiring managers are adding AI capabilities to their position descriptions. Maybe analyses based on AI could benefit scientific predictions generally.
And before all of this AI business, we were in some ways already awash in data, with things like automation and high-throughput methods for data gathering raising expectations for what’s possible. So what of all that? How will big data and AI affect scientific prediction? Here I address this question and highlight a useful way of thinking about what to expect as computer systems develop further.
Reducible and irreducible error
In science, one of the things we do is construct models for expected outputs of interesting processes. We make predictions. The details of how we construct and formalize models vary. We can obtain descriptive, statistical learning models based on previous observations, such as the expected size of an organism after a number of days of growth, or which word should be emitted next from a large language model. Likewise, models can be derived from process understanding that can be formalized mathematically to generate expectations. Regardless of their form, most of these models, particularly in multi-faceted systems, are regarded as an approximation or estimate of what we expect in future observations. The quality of the prediction can be measured by how far off the estimate is from a known truth. With big data and AI, we might get closer to the truth.
Deviations of predictions from the true value can be partitioned into reducible and irreducible components (James, Witten, Hastie, and Tibshirani. 2013. An Introduction to Statistical Learning). When we choose different statistical and computational methods for using information about a system to generate outputs, these choices affect the reducible component of our prediction error. The reducible component is the fraction that can make use of the existing data to characterize a system and minimize errors in expectations. The key question is what fraction of errors in predicting outcomes of an interesting scientific process are associated with measured values or features, and what fraction lies beyond the measured and knowable aspects of the system. That remainder is the irreducible error.
Right off the top, before getting specific about what learning method we might use to construct a model of a focal process, we can do some thinking about how much of the error in prediction of future data is likely reducible, by measurement and model choices, and how much lies beyond our grasp and is not characterized by what we can measure. Phenomena in science provide examples that span the range. Predictions for some processes are readily constrained by measurable features, with low prediction error, allowing us to put satellites into desired orbits, run nuclear reactors, do analytical chemistry, and the like. Scientists seeking to predict cancer incidence or organismal lifespan are grappling with processes that have multifarious causes and that are less well constrained by measurable features.
Methods for predicting expected outputs will certainly differ in how close to optimally they estimate the relationship between outputs and features and minimize reducible error. Yet even with optimal methods, there will still be prediction error that is irreducible. The irreducible fraction of error is caused by processes that are not captured through measurable features. In evaluating prospects for using data for predictions, I think it would be worthwhile to state explicitly what size of irreducible error likely exists. There will be optimistic and pessimistic scientists, but it seems good to be explicit. For many interesting processes, my impression is that the irreducible error is likely to be appreciable. Knowing all of the genomic variants in the cells, the true abundance of all proteins and other molecules at multiple points in time, in different cells types and tissues, combined with socioeconomic measures, along with the most ideal computational method, arguably will yield predictions of the lifespan of a typical healthy individual that are associated with high error, in part because of the many possible causes of death. In the context of disease, the development of biomarkers and improved models will likely improve predictions of lifespan, but substantial irreducible error is likely to remain because available, measurable features are insufficient to further constrain our predictions. Yes, in principle, with perfect knowledge of all causes (see Laplace’s Demon), no such limit need exist. But, in practice there is a limit and irreducible error pervades.
While the true irreducible and reducible error are unknown for a system, we are not completely helpless. In simulations, we can construct generative models with known amounts of irreducible error and see how big data and computational methods work out for us. More on that in the Some numerical illustrations section, below.
In evaluating the future of prediction with big data and AI, we could discuss explicitly what fraction of the prediction error is likely irreducible, and what the benefits of reducing error through further data acquisition and method refinement will be in the context of our interesting processes. Simply believing and asserting that AI will be transformative, even for problems with an inherently large irreducible error, seems like a bad idea and likely wasteful of time and effort for science.
Stated another way
The average squared difference between true and predicted values (total error) can be decomposed into reducible and irreducible variance components: E(Y-\hat{Y})^2 = [f(X) - \hat{f}(X)]^2 + \text{Var}(\epsilon) (Equation 2.3 in James, Witten, Hastie, and Tibshirani, 2013). The left side of this equation is a measure of the total error in prediction, the expected (mean) squared difference between the true (Y) and estimated values (\hat{Y}). The right side formalizes that one portion of error will be a function of information about the system (X) and the form of the prediction equations (\hat{f} as an estimate of the ideal functional structure f). Better choices of a system for using information will make [f(X) - \hat{f}(X)]^2 and the total prediction error smaller. That is, choices of statistical learning tools can shrink this reducible component of error. Prediction error remains even when \hat{f} is a perfect estimate of the ideal f, due to the irreducible error that is independent of available information (X) about the system (\text{Var}(\epsilon)).
Getting additional data would change X and might affect the total error and its components. A very big piece of recent advances in the quality of predictions from statistical learning has come from very large amounts of additional training data, rather than refinements in the methods for estimation (\hat{f}). The return on increasing X with more samples, more features, or both, is commonly an open and difficult question. For example, additional samples will not necessarily reduce prediction error, as new samples might come from new strata. As noted above, evaluating the future of big data and AI involves evaluating how adding samples from additional years and cohorts, or new features, affects the fraction of the prediction error that is reducible. If the fraction of prediction error that is irreducible is stubbornly large, different learning strategies cannot make much of a difference.
It seems reasonable to remind everyone that the prediction error we are considering is for out-of-sample, test data, not the training data that were used to pick or fit a model. Whereas it is common in lots of area of science to stop evaluating the model once it has been fit to training data, that adds additional types of error and precludes recognition of reducible and irreducible error (see James, Witten, Hastie, and Tibshirani, 2013).
Some numerical illustrations
As an exercise to understand reducible and irreducible error, as well as the effects of data dimensions and statistical learning methods, a group of us used simulations with known causal graphs and true values to make contrasts (Jahner et al. 2026). While we considered many combinations for data attributes, for the purpose of evaluating prediction quality, the simulations were simple.
The output variable (Y) was always a linear function of ten input variables (features or covariates), each with a mean of zero and a standard deviation of one. Simulations differed in that the coefficient for each of the input variables (the causal effect) was 0.1, 0.3, or 0.8. The output of causal features was added to an error variance with a mean of zero and a standard deviation of one to yield the true output (Y). Consequently, a major contrast among the simulations was in the fraction of the total output variation that was simulated to arise from the causal variables, relative to the additional, constant variance that was independent of covariates (irreducible error). Throughout we used a set of observation to train models (varying the number of observations for training as a dimension to study), but used a constant number of observations (n=500) for measuring prediction error in held-out test data.
As a tool to characterize the irreducible error, we measured the squared correlation of predicted output based on the ten causal features and the observed output \text{corr}(\hat{Y}, Y)^2 as the ideal fraction of the output variance explained by causal factors (the Out-of-sample R^2 in Figure 1), with the complement approximating the fraction of the variance associated with error that is independent of all measured features 1-\text{corr}(f(X), Y)^2. Mathematically this is not directly equivalent to the reducible and irreducible components of the error variance, but the quantity functions analogously. It captures the difference in the simulations with respect to what fraction of the output variance was due to the causal features and what was the remainder variance that was inaccessible from knowledge of covariates. It was also a measure that we could apply to different statistical learning methods, to understand how well they performed in predicting outputs relative to the ideal of using only the known causal features and their coefficients.
Across the different sizes of causal effects, the maximum, ideal squared correlation between predicted and true values varied systematically (gray boxes for \beta_{\mathrm{causal}}= 0.1, 0.3, and 0.8 in Figure 1 correspond to the average empirical R^2 for a set of simulations), reflecting the decrease in fraction of the prediction error that is irreducible (and increase in the fraction that is reducible) with increasing causal effect size. For simulations with large causal effect size (\beta_{\mathrm{causal}}=0.8) and large training data (N=500), several computational methods performed nearly ideally, regardless of how many covariates were available (P; though more covariates caused trouble for several methods). Additional measured features (covariates) can act as potential distractors through their spurious associations with the output variable in the training data. That complexity was a principal focus in Jahner et al. 2026. Additional non-causal variables typically increased ambiguity and caused statistical learning approaches to fall farther short of the maximal reducible error (for example, see \beta_{\mathrm{causal}}=0.3 and N=150 in the center). In these circumstances, data are the limitation and computational methods do not differ that much (similarly see Shaktah et al. 2026).
These simulations are contrived, but they are illustrative as points of reference for evaluating prospects in prediction. For any real system, we might want to know which of the \beta_{\mathrm{causal}} and P regimes in Figure 1 pertain and set limits on reducible and irreducible error. With more time and money we can sometimes increase the number of samples (N) relative to the number of covariates (P). Simply measuring more covariates can dig us a deep hole of erroneous predictions (all DNA variants, expression of all genes at multiple time points, proteome, et cetera to predict lifespan) if sample size is not also large. Prediction error was high in a lot of parameter space of these simulations, regardless of the choice of statistical learning approach. This held true even with follow-up analyses of some of these simulations with a state-of-the-art tabular foundation model (TabPFN v3; see Hollman et al. 2025 and code on github).
Prospectus
If nothing else, it seems good to be aware that even with ideal scientific and computational methods irreducible error will exist. Maybe you were already there, but you did not have a name for it (irreducible error is not a commonly used label in science papers). Now in discussing a scientific plan, you can ask (yourself) how big the irreducible error is likely to be. Beyond that, as an exercise one can profile the system of interest with simulations and learn what we do and do not know about sources of error. Simulations would allow a sensitivity analysis to quantify what improvements in prediction error will likely result from different scales of additional data or from adopting different statistical learning strategies. To some extent those questions could be addressed verbally too.
These questions matter only for scientists who are not having absurd amounts of money thrown at them by investors. It feels a bit odd to be writing this when companies are spinning up with substantial funding, promising to use AI to get better predictions, in systems where data is undoubtedly the bottleneck and the irreducible error is profound. Even when there are new large investments in gathering biology data, like Biohub, or within AI and other tech companies, I wonder how big the irreducible error, or the prediction error overall, will be after $1.5 billion have been spent.
Citation
For attribution, please cite this work as:
Buerkle, Alex. 2026. “How Big Is the Irreducible Error Likely to
Be?” October 27. https://doi.org/10.5281/zenodo.23696555.