When can an ensemble of models perform worse than its best individual member?
Answer
Ensembles win through diversity, not through headcount. The ambiguity decomposition makes this exact: the ensemble’s error equals the average member error minus the average member disagreement, so combining helps only to the extent that members err on different examples. When errors are highly correlated, averaging or voting adds nothing and can dilute a genuinely better member. Stacked ensembles add a second failure surface: a meta-learner trained on too little or leaked validation data overfits the members’ quirks. And in production, the extra accuracy may not survive contact with engineering constraints, which is why Netflix never deployed the million-dollar Grand Prize ensemble.
(1) Diversity Is the Mechanism: majority voting only helps when member mistakes are at least partially independent; perfectly correlated members vote identically, and the ensemble is just the average member.
(2) Combination Failures: a strong model blended with weak-but-confident ones, or a stacking meta-learner fit on a tiny validation slice, can land below the best member; out-of-fold predictions and held-out weight tuning are the standard guards.
(3) Production Lesson (Netflix): the 2009 Grand Prize ensemble blended hundreds of models for a 10% RMSE gain, but Netflix reported the incremental accuracy “did not seem to justify the engineering effort” and shipped only two of the simpler Progress-Prize algorithms instead.
Figure 1: The diversity condition: with independent errors (left) majority vote exceeds every member; with correlated errors (right) the vote inherits the shared blind spot and can fall below the best single model.
Two subtler traps complete the picture. First, aggregation weights: an average is optimal only when members are comparably accurate and comparably calibrated; one badly calibrated but overconfident member can dominate a soft-vote average and drag it below the best member. Second, the objective mismatch: members tuned individually for accuracy may combine poorly if their errors concentrate on the same hard slice; deliberately trading a little individual accuracy for decorrelation (different feature views, architectures, or training objectives) is how production ensembles are actually designed.
Mathematical Formulation:
![\bar{A} = \frac{1}{M}\sum_{m=1}^{M} \mathrm{E}_x\left[ (f_m(x) - \bar{f}(x))^2 \right]](https://s0.wp.com/latex.php?latex=%5Cbar%7BA%7D+%3D+%5Cfrac%7B1%7D%7BM%7D%5Csum_%7Bm%3D1%7D%5E%7BM%7D+%5Cmathrm%7BE%7D_x%5Cleft%5B+%28f_m%28x%29+-+%5Cbar%7Bf%7D%28x%29%29%5E2+%5Cright%5D&bg=ffffff&fg=000&s=3&c=20201002)
Where:
is the squared error of the average prediction 
, and 
is the mean of the members’ individual squared errors.
is the ambiguity: the average squared disagreement of each member 
from the ensemble mean, i.e. the diversity dividend.- The ensemble beats the average member exactly by ; if all members make identical errors then
and ensembling gains nothing (Krogh-Vedelsby decomposition).
is the squared error of the average prediction 
, and 
is the mean of the members’ individual squared errors.
is the ambiguity: the average squared disagreement of each member 
from the ensemble mean, i.e. the diversity dividend.
and ensembling gains nothing (Krogh-Vedelsby decomposition).| Failure Mode | Mechanism | Guard |
|---|---|---|
| Correlated Errors | Same architecture, features, and data produce shared blind spots | Diversify feature views, model families, training objectives |
| Overfit Stacking | Meta-learner trained on in-sample or tiny validation predictions | Out-of-fold predictions; simple meta-model (logistic, weighted mean) |
| Miscalibrated Member | Overconfident weak model dominates the soft vote | Calibrate members before averaging; weight by validation skill |
| Engineering Cost | Hundreds of members multiply latency, memory, and failure surface | Netflix lesson: ship the simple models that capture most of the gain |
Leave a Reply