Widely cited 2004 hemostasis screening study was circular, re-analysis finds

A widely cited 2004 study that reported strikingly high sensitivities for laboratory tests used to detect primary hemostasis disorders before surgery has been re-examined, and the verdict is damning: the diagnostic accuracy figures it produced cannot be used as evidence at all. In a Perspective article published in Frontiers in Medicine, researchers from Ludwig-Maximilians-Universität München and Labor Becker MVZ in Munich show that the study’s estimates of sensitivity and specificity were generated by circular validation, in which the tests being evaluated helped define the very condition they were judged against.

The 2004 study, based on a prospective cohort of 5,649 patients scheduled for elective surgery, evaluated a preoperative screening strategy combining a standardized bleeding questionnaire with laboratory testing. The questionnaire and clinical assessment identified a positive bleeding history in 628 patients, or 11.2 percent of the cohort. In these patients, additional testing included bleeding time and von Willebrand factor antigen, alongside platelet function analysis with the PFA collagen–epinephrine and collagen–ADP cartridges, activated partial thromboplastin time, prothrombin time and platelet count, which had been measured in all patients.

Of the 628 patients with a positive bleeding history, 256 showed at least one abnormal test result and were classified as having “impaired hemostasis.” Among these, 250 had an abnormal PFA collagen–epinephrine result, 199 an abnormal PFA collagen–ADP result, 188 a prolonged bleeding time, 39 an abnormal von Willebrand factor antigen level, 24 an abnormal aPTT, 10 an abnormal prothrombin time and 2 an abnormal platelet count. The study then reported that the PFA collagen–epinephrine test identified 97.7 percent of patients with primary hemostasis disorders, the collagen–ADP cartridge 77.7 percent, and von Willebrand factor antigen 15.2 percent.

The problem, according to the new analysis, lies in how “impaired hemostasis” was defined. The condition was established using the results of the same assays whose performance was then calculated against it. In diagnostic accuracy research this is known as incorporation bias: the index test forms part of the reference standard against which its own performance is evaluated. As the authors put it, agreement between the test and the reference standard is built into the design rather than measured, and no reference standard independent of the index tests ever existed. In plain terms, the validation was circular.

The re-analysis also scrutinizes a second set of performance figures reported in the 2004 study, in which a sensitivity of 90.8 percent and a specificity of 86.6 percent were claimed for the PFA collagen–epinephrine test on the basis of a receiver operating characteristic analysis. The 2004 publication itself does not describe in detail the reference used for this calculation, but the manufacturer’s instructions for use describe it as impaired platelet function based on all laboratory data except PFA results, together with patient medication history. Crucially, the re-analysis notes, these other tests were in part performed after, and with knowledge of, the PFA result. Antiplatelet drug use was in 14 patients suspected only after testing and confirmed upon requestioning. A patient with an abnormal PFA result therefore had a considerably greater chance of being investigated further and confirmed as having impaired hemostasis than a patient with a normal result, a problem known in QUADAS-2 terminology as verification bias. The cut-off underlying the figures was also not stated and was selected by ROC analysis within the same dataset, an optimized post hoc choice that often inflates performance metrics.

The bias extends to the questionnaire itself. Table 3 of the 2004 study reported sensitivity, specificity and predictive value for each of the 12 questions of the bleeding questionnaire with respect to “impaired hemostasis.” But a positive answer to at least one question, or clinical evidence of bleeding, triggered the additional testing that could establish “impaired hemostasis,” while patients who answered all questions negatively were not investigated further. Each questionnaire item was thus evaluated against a reference standard influenced by its own result, and the assays factoring into the reference standard were performed depending on the responses, compounding incorporation and verification bias.

To demonstrate how little such figures mean, the authors ran a striking experiment. They generated a synthetic dataset of 256 “patients” and seven “assays” in Microsoft Excel, with every cell filled by random numbers uniformly distributed between 0 and 1. Cut-offs were then defined, with the help of an artificial intelligence tool, so that each assay produced exactly the same proportion of positive results as reported in the 2004 study, and the dataset was minimally adjusted so that each “impaired hemostasis” case had at least one abnormal result. When the detection of “impaired hemostasis” was calculated from this self-referential construction, the random numbers reproduced the reported detection rates almost perfectly: 97.7 percent for the simulated collagen–epinephrine test, 77.7 percent for collagen–ADP, 73.4 percent for bleeding time, 15.2 percent for von Willebrand factor antigen, and down to 0.8 percent for platelet count. Given a suitable cut-off and a self-referential case definition, the authors conclude, random numbers can produce any desired “sensitivity,” which they call a statistical perpetuum mobile. They stress that this demonstrates the statistical problem of self-referential approaches and is not a statement about the raw data of the 2004 study or about PFA as a method.

The stakes are high because the 2004 study has been cited roughly 150 times according to Web of Science, including by articles in the British Journal of Anaesthesia, Anesthesiology, the American Journal of Hematology, the British Journal of Haematology and the European Journal of Anaesthesiology, as well as in clinical practice guidelines, multidisciplinary recommendations and educational articles. Most consequentially, its findings remain incorporated into the instructions for use of the PFA collagen–ADP and collagen–epinephrine cartridges, which state that a standardized questionnaire combined with the collagen–epinephrine cartridge provided a high detection rate of impaired hemostasis and that 98 percent of all hemostasis defects were identified by that cartridge. Both claims derive from the biased calculations, and the authors recommend that the manufacturer reassess the use of this evidence; the same citation also appears in the US-market instructions for the PFA-100 reagents.

The authors are careful to limit the scope of their critique. They state that their aim is not to assign blame to the 2004 authors, whose transparent reporting of their reasoning they credit, and whose questionnaire and observational data from the large cohort they consider to remain relevant. Their evaluation concerns only the diagnostic accuracy estimates of the 2004 study. It does not imply that PFA assays lack diagnostic value, and it does not affect the findings of other published studies on these assays, whose methodological rigor they did not evaluate. They also note that several citations of the 2004 study, such as recommendations to obtain a patient’s bleeding and medication history before surgery or to seek specialist consultation when hemostasis abnormalities are suspected, do not rely on the problematic aspects of the article.

Other citations, however, do appear to rest on the inflated figures. A recent review on the PFA method cites the 2004 study to support the claim that normal closure times can serve as a marker of low surgical bleeding risk as a replacement for skin bleeding time, and a review on total hip arthroplasty in patients with coagulopathy states that a standardized hemorrhage history together with the PFA-100 test can reliably detect most cellular hemostasis disorders. The broader lesson, the authors argue, is that widely cited evidence must be continuously scrutinized for methodological validity, and that key references in review articles and clinical guidelines should be critically appraised. Findings that seem to offer practical solutions to pressing clinical problems, they warn, may attract less critical scrutiny from authors, reviewers and readers alike.

Subject of Research: Circular validation and incorporation bias in a widely cited study of preoperative primary hemostasis screening

Article Title: Circular validation in diagnostic research: re-examination of a widely cited study on primary hemostasis screening

Article References: Calatzis, A., Spannagl, M., & Wittmann, G. (2026). Circular validation in diagnostic research: re-examination of a widely cited study on primary hemostasis screening. Frontiers in Medicine, 13, Article 1909484. https://doi.org/10.3389/fmed.2026.1909484

Image Credits: AI Generated

DOI: 10.3389/fmed.2026.1909484

Keywords: incorporation bias, verification bias, PFA-100, platelet function testing, primary hemostasis, diagnostic accuracy, bleeding questionnaire, preoperative screening, reference standard, von Willebrand factor, ROC analysis, hemostasis