New Scoring System Tracks How Nurse Anesthetists Grow, Month by Month

Every new nurse anesthetist faces a daunting transition: one day they are finishing their pre-employment preparation, and the next they are helping to keep patients safely unconscious on an operating table. Yet hospitals have long lacked a rigorous, specialty-specific way to measure how quickly these newcomers actually become competent, and whether their own sense of progress matches what their supervisors see. A research team at Beijing Tsinghua Changgung Hospital, affiliated with Tsinghua University’s School of Clinical Medicine, has now built and tested a detailed competency assessment framework designed to answer exactly those questions, tracking new hires month by month across their first half year in anesthesia practice.

The study, published in BMC Nursing, was led by Xiaobei Ma and Xueyan Fan, who contributed equally, together with colleagues including Fengli Gao, Xin Liu, Yi Duan, and corresponding author Zhifeng Gao. Rather than relying on a generic nursing checklist, the team constructed a framework tailored specifically to the role of the nurse anesthetist, a profession in which errors in airway management, drug dosing, or vigilance during the post-anesthesia care phase can have immediate and serious consequences. The result is a hierarchical instrument comprising five domains, twenty second-level indicators, and one hundred individual scored elements, each anchored to explicit behavioral criteria.

Building such an instrument required a multi-stage methodology that blends structured expert consensus with quantitative weighting. The researchers began with a literature review and semi-structured interviews with experts, then ran a two-round Delphi consultation, a technique in which a panel of specialists independently rates proposed items and receives feedback between rounds until agreement stabilizes. To determine how much weight each domain should carry in the final score, they applied the analytic hierarchy process, a mathematical method that converts pairwise expert judgments into a consistent set of weights, checking that the judgments did not contradict one another through a consistency ratio threshold.

The preliminary measurement evaluation focused on content validity and reliability. Experts rated the relevance of each item, yielding a scale-level content validity index average of 0.95, very close to the perfect score of 1.0, while item-level modified kappa values, which correct expert agreement for chance, ranged from 0.831 to 1.000. To estimate inter-rater reliability, five nurses were independently scored by two prespecified evaluators at program entry; the total-score intraclass correlation coefficient using the two-way mixed-effects absolute-agreement single-measure definition, ICC(A,1), was 0.889, with a 95 percent confidence interval of 0.388 to 0.988. The authors are candid that the narrow confidence interval’s lower bound reflects the tiny reliability subsample, making this preliminary, low-precision evidence rather than comprehensive validation.

The longitudinal component is where the study becomes genuinely distinctive. The team retrospectively analyzed a census of all eligible nurses entering the training program between August 2016 and August 2025, defining T0 as program entry after pre-employment preparation and T1 through T6 as the completion of each of the first six months. Of 77 screened records, 68, or 88.3 percent, had complete T0 through T6 data and entered the analysis; five nurses resigned during the period and four had at least two consecutive missing monthly assessments. All analyses were performed in R version 4.5.2, with repeated-measures analysis of variance serving as the primary within-evaluator test of change over time.

The trajectory of improvement was striking. Mean self-assessment scores climbed from 35.95 at entry to 96.83 at month six, while faculty ratings rose from 29.20 to 95.32. After applying the Greenhouse-Geisser correction, which guards against violations of the statistical assumption of sphericity in repeated-measures designs, the time effects were highly significant for both perspectives, with P values below 0.001 and partial eta-squared values of 0.977 for self-ratings and 0.984 for faculty ratings, meaning that time explained nearly all of the variance in scores. In practical terms, new nurse anesthetists moved from roughly a third of the maximum score to near-ceiling performance within six months under this framework.

Perhaps the most psychologically interesting finding concerns the gap between self-perception and external judgment. At every single time point, self-ratings exceeded faculty ratings, but the size of that gap changed systematically. A participant-level mixed-effects model, which accounts for repeated observations nested within individuals, estimated the model-based difference between evaluator trajectories as largest at month two, at 9.51 points, and shrinking to just 1.50 points by month six. The time-by-evaluator interaction was significant in a likelihood ratio test yielding a chi-square statistic of 110.67 with six degrees of freedom and P below 0.001. The pattern suggests that new nurses begin with a degree of overconfidence relative to their supervisors’ assessments, then calibrate their self-appraisal as real clinical experience accumulates.

The framework also showed meaningful concurrent associations with established assessment tools. At month three, self-ratings correlated with written examination scores at a Spearman coefficient of 0.815 and faculty ratings at 0.582, while the corresponding correlations with objective structured clinical examination scores, the standardized practical tests widely used in medical education, were 0.580 and 0.558. By month six these associations remained positive, at 0.752 and 0.532 for the written examination and 0.689 and 0.340 for the OSCE. Notably, at program entry the correlations with the written examination were negligible, which makes sense: before any anesthesia-specific training, general knowledge tests say little about how a nurse will perform in the operating room, so the framework’s scores capture something the written test does not.

The authors also report an exploratory analysis in a separate prespecified five-person subset examining whether every individual improved from T0 to T6. All five showed positive change, but an exact two-sided sign test produced a P value of 0.0625, just shy of the conventional 0.05 threshold, a mathematical inevitability when only five participants are tested, since the smallest achievable two-sided P value for a sign test with five observations is exactly 0.0625. The team treats this honestly as suggestive rather than confirmatory, and the supplementary materials go further, documenting acceptability questionnaires completed by all 68 respondents, sensitivity analyses, and exploratory safety surveillance, an unusually transparent level of statistical disclosure for a single-centre education study.

What the study does not claim is perhaps as important as what it does. The authors explicitly state that their results provide preliminary, low-precision measurement evidence rather than comprehensive validation or causal proof that the training program itself drove the score increases, and they call for prospective multicenter evaluation before the framework is used for any high-stakes decisions such as hiring, promotion, or certification. Still, the implications are considerable. With anesthesia teams worldwide facing persistent staffing pressures, a validated, behaviorally anchored, month-by-month competency map could help educators target support precisely where new nurses lag, and the finding that self-assessment converges with faculty judgment over time offers a measurable signal of professional maturation. If larger multicenter studies replicate these trajectories, the humble scoring rubric developed in a Beijing hospital could become a template for how surgical specialties everywhere turn the anxious first months of a new clinician’s career into a quantified, coachable journey.

Subject of Research: Development and longitudinal evaluation of a competency assessment framework for newly recruited nurse anesthetists

Article Title: Development, preliminary measurement evaluation, and longitudinal application of a competency assessment framework for newly recruited nurse anesthetists

Article References: Ma, X., Fan, X., Gao, F., Liu, X., Duan, Y., & Gao, Z. (2026). Development, preliminary measurement evaluation, and longitudinal application of a competency assessment framework for newly recruited nurse anesthetists. BMC Nursing. https://doi.org/10.1186/s12912-026-05424-y

Image Credits: AI Generated

DOI: 10.1186/s12912-026-05424-y

Keywords: nurse anesthetists, competency framework, Delphi method, longitudinal assessment, mixed-effects model, content validity, inter-rater reliability, OSCE, nursing education, self-assessment, analytic hierarchy process, BMC Nursing