Exact divide-and-recombine method brings negative binomial regression to massive hospital datasets

Hospital length of stay is a notoriously awkward outcome for statistical modelling. It is a right-skewed count variable that is heavily overdispersed relative to the Poisson benchmark, which means ordinary least-squares and Poisson regression become inappropriate once datasets reach the scale of modern health administrative records. A new methodological paper in BMC Medical Research Methodology tackles this problem head-on with a divide-and-recombine framework for negative binomial regression that makes full-data maximum likelihood inference possible even when a dataset exceeds the memory available on a single machine.

The core idea is to partition observations into sequential chunks and, at each pass, accumulate what the author calls asymptotically lossless compressed representations of the data. Rather than approximating the fit from summaries of subsets, the approach accumulates information in a way that preserves the exact estimating equations of the full-data problem.

The theoretical centrepiece is a proof that the resulting estimator is algebraically identical, not merely asymptotically equivalent, to a fully specified centralised alternating Fisher-scoring or iteratively reweighted least-squares algorithm at every iteration. Conditions for recovering the full-data maximum likelihood estimate are stated separately, and the paper also derives the pooled asymptotic variance of the dispersion parameter, the quantity that governs how much the counts vary beyond what a Poisson model would allow.

The computational gains are substantial. Relative to the widely used MASS::glm.nb implementation in R, active memory falls from order n times p to order n times S times p plus p squared per iteration, where S is the number of chunks. In a Monte Carlo study with five million observations and one thousand replications, the new estimator agreed with MASS::glm.nb, treated as the empirical reference, to within 1.8 times ten to the minus ten for the regression coefficients and 8.0 times ten to the minus ten for the dispersion, leaving relative efficiency indistinguishable from 1.000.

Peak memory fell by 96 percent and computation time dropped by a factor of 3.75, of which partitioning alone contributed 2.35; on the SPARCS application the partitioning contribution was 1.15 out of 2.58. A forced 50-iteration trace confirmed the algebraic identity at every iterate, and in a separate study at a sample size of 200,000 the plug-in standard error for the dispersion achieved 0.947 coverage. Supplementary experiments showed the exactness holds even an order of magnitude below the recommended chunk-size floor, and that divide-and-recombine remains faster than a direct fit across sample sizes from 10,000 to 5,000,000.

The estimated dispersion parameter of 1.797 confirmed severe overdispersion, with a variance-to-mean ratio of 13.4 for length of stay. The model’s most striking finding concerns patient severity: the incidence rate ratios for length of stay were 4.26 for Extreme, 2.16 for Major and 1.55 for Moderate APR-severity admissions compared with Minor ones, so severity of illness dominates the length-of-stay gradient.

Because discharges cluster within facilities, the author also reports facility-clustered robust standard errors. These inflate the model-based errors by a mean factor of 17.8 and move eleven of twenty-nine terms out of nominal statistical significance, a sobering illustration of how ignoring institutional clustering can overstate confidence in large administrative datasets. Even so, the severity and mortality-risk gradients remain robust under clustering.

For health services researchers, the practical message is twofold. First, overdispersed count outcomes such as length of stay demand negative binomial or similar models at scale, and the dispersion estimate here shows just how far real hospital data depart from Poisson assumptions. Second, exactness and scalability need not trade off: the divide-and-recombine framework reproduces the full-data maximum likelihood fit iterate for iterate while making million-row fits feasible on ordinary hardware.

The author argues that these results establish divide-and-recombine as a principled, interpretable and computationally tractable foundation for large-scale count regression in health services research.

Subject of Research: A divide-and-recombine framework for exact negative binomial regression on large-scale overdispersed count data, applied to New York hospital length-of-stay records.

Article Title: Exact divide-and-recombine estimation for negative binomial regression: a simulation study and application to SPARCS hospital length-of-stay data

Article References: Nayem, M. M. H. (2026). Exact divide-and-recombine estimation for negative binomial regression: a simulation study and application to SPARCS hospital length-of-stay data. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-026-03041-6

Image Credits: AI Generated

DOI: 10.1186/s12874-026-03041-6

Keywords: negative binomial regression, divide and recombine, big data statistics, hospital length of stay, overdispersion, SPARCS, maximum likelihood, memory-efficient computation, health services research, simulation study, robust standard errors, Exact