New Robot Usability Questionnaire Passes Key Validation Test With AI and Non-AI Robots

A laboratory study with 64 participants has largely confirmed the convergent validity of a usability questionnaire built specifically for robots, showing that it works consistently whether users believe they are operating an AI-based system or a conventional one. The instrument, called the Usability Questionnaire for Robotic Systems (UQRS), is grounded in the interaction principles of the ISO 9241-110 standard, which breaks usability down into seven ergonomic design guidelines for human–technology interaction: suitability for the user’s tasks, controllability, learnability, conformity with user expectations, self-descriptiveness, user error robustness and user engagement. The new work, published in Discover Robotics by Eva Heinold, Patricia Helen Rosen and Sascha Wischniewski, extends a validation process that began with earlier studies on a stationary dual-arm robot and now covers mobile robots in an in-person interaction setting.

The motivation is practical. Human–robot interaction is spreading through workplaces, with mobile robots deployed in healthcare, education and public infrastructure, even though industry remains the dominant sector. At the same time, artificial intelligence is being folded into industrial robotics, holding potential for smoother, more intuitive interaction and better-performing production systems, yet adoption in manufacturing remains scarce. Under the Industry 5.0 framing, which emphasizes human-centered AI that is reliable, safe and trustworthy, usability occupies a special position: a system’s usability can influence both short-term task performance and longer-term acceptance or rejection of robotic technology. Despite this, usability is not always assessed systematically during design, and there has been little research on whether established assessment tools remain reliable when applied to AI-based robotic systems.

To test the questionnaire, the researchers ran a between-subjects laboratory study at the German Federal Institute for Occupational Safety and Health (BAuA) in which participants performed a robot-assisted inspection task. Each participant worked with one of two mobile robots, a Spot or a MiR 250, to inspect nine sensors distributed around the laboratory. Each sensor displayed four data points: its name, temperature, humidity and operational status. Participants instructed the robot to navigate to the sensors, photograph them and transmit the images and extracted readings back, then supervised the robot’s evaluations, judged their correctness and compiled a complete inspection report. After a supervised training phase with three sensors, participants completed three inspection rounds alone, while a researcher retained emergency control from a separate panel for safety.

The crucial twist was an artificial intelligence priming manipulation. Both groups used robots that functioned identically, but the instructions differed. Participants in the AI condition were told the robot employed machine learning-based perception for navigation and reinforcement learning for data processing, and that their input would train the AI through an iterative learning process; questionnaires referred to “the AI-based robot.” Participants in the not-AI condition were told the robot ran on newly developed camera systems with manually adjusted settings refined through human feedback, and questionnaires referred simply to “the robot.” The priming worked: every participant in the AI condition strongly agreed the robot was AI-based, and every participant in the control group rejected that statement. All participants were debriefed afterward and informed that the robots functioned identically.

The final sample comprised 64 participants, 30 male and 34 female, aged 18 to 64 with a mean age of 35.67 years, after four participants were removed for incomplete data or non-compliant behaviour. The AI group numbered 33 and the not-AI group 31, with no significant demographic differences between them and an even split of robot types. Alongside the 41-item UQRS, which rates items on a five-point Likert scale across eight factors, participants completed the ten-item System Usability Scale (SUS), a fast but shallow general usability measure, and the perceived usefulness (PU) and perceived ease of use (PEOU) questionnaires derived from the Technology Acceptance Model.

Reliability analysis using Cronbach’s alpha and McDonald’s omega showed largely good factor consistencies. All UQRS factors except suitability for the user’s task and user engagement reached acceptable to good internal consistency, with controllability and individualisation the strongest and most stable scales, bordering on excellent. Suitability for the user’s task showed questionable alpha and dropped to poor omega, while user engagement was the weakest scale, showing questionable reliability in both conditions and both metrics. The authors suggest the engagement items, which touch on trust and aesthetic enjoyment, may not capture a unidimensional construct, and that a single, relatively short interaction may have been too brief for engagement to fully manifest, though they note this interpretation is speculative. Among the comparison instruments, PU and SUS performed acceptably to well, while PEOU showed questionable consistency.

Convergent validity was largely confirmed. The UQRS correlated significantly and positively, above the conventional convergent validity threshold of r ≥ 0.30, with all three other constructs, and PU, PEOU and SUS also intercorrelated as expected. Regression analyses sharpened the picture: the UQRS significantly predicted PEOU and PU, explaining up to 23 percent of the variance in perceived ease of use and 30 percent of perceived usefulness, but did not significantly predict overall SUS scores. At the factor level, seven of eight UQRS factors significantly predicted some variance in PU (all except self-descriptiveness, with individual R² values of 0.07 to 0.18), and seven predicted PEOU (all except learnability, R² 0.09 to 0.25), indicating that no single standalone factor drives these perceptions. For the SUS, only suitability for the user’s task showed a significant relationship, accounting for roughly 40 percent of its variance, consistent with the idea of task–technology fit: a robot may be explainable and controllable, but if it does not suit the task it fails to support the user’s goal.

The group comparison produced the study’s most nuanced finding. A two-way MANOVA revealed a significant multivariate effect of AI condition, Pillai’s Trace = 0.282, F(8, 54) = 2.653, p = 0.016, with no significant effect of robot type or its interaction. However, no individual dependent variable reached significance in the univariate follow-up tests, and the SUS, PU and PEOU showed no differences between groups. The authors therefore describe only tentative evidence that perceived AI presence may shape the overall pattern of usability perceptions without localizing to any single dimension. On the application side, the result demonstrates that the UQRS is sensitive enough to reveal differences in perceived usability that the SUS, PEOU and PU did not detect, even though the robots performed identically, suggesting that merely believing a system is AI-based can colour usability assessments.

The authors are careful about limits. The manipulation was a priming or labelling intervention rather than a comparison of genuine technological differences, so perceptual differences may reflect responses to the label rather than the underlying technology. Demand characteristics and social desirability bias remain risks for self-report measures, mitigated here through standardized scripted instructions, a between-subjects design and minimized variance in instruction administration, though future work with implicit measures such as eye-tracking could strengthen inference. Individual experiences of the interaction were also not fully uniform despite standardized introductions. The study analyzed data in SPSS Version 29, and the underlying data are held by BAuA and available upon reasonable request with the institute’s permission.

The takeaway for the field is that the UQRS now stands validated for stationary and mobile robotic systems in industrial workplaces, including AI-labelled robots, making it a distinctive instrument in the usability assessment landscape. Its strength lies in explaining why systems may be perceived as more useful or easier to use, tracing perceptions back to specific ISO 9241-110 interaction principles, and it can be deployed during system design to flag problems before robots reach deployment. The analysis also showed that the UQRS and the SUS overlap only partially, so neither captures perceived usability in its entirety; the authors suggest researchers use them as complementary tools rather than treating either as redundant. Future directions include testing the questionnaire in social human–robot interaction scenarios, in longer-term or repeated interactions where engagement can develop more fully, and in closer studies of how expectations toward AI-based robots shape usability judgments.

Subject of Research: Validation of a robot-specific usability questionnaire in an in-person study with AI-labelled and conventional mobile robots

Article Title: Usability assessment of AI-enabled robots through convergent validity testing of a new questionnaire

Article References: Heinold, E., Rosen, P. H., & Wischniewski, S. (2026). Usability assessment of AI-enabled robots through convergent validity testing of a new questionnaire. Discover Robotics, 2(1), Article 18. https://doi.org/10.1007/s44430-026-00031-7

Image Credits: AI Generated

DOI: 10.1007/s44430-026-00031-7

Keywords: human-robot interaction, usability, questionnaire validation, ISO 9241-110, artificial intelligence, mobile robots, System Usability Scale, convergent validity, Industry 5.0, industrial workplaces, Cronbach's alpha, perceived ease of use