Dynamic checklist developed for web designers to work more efficiently, creatively

According to the Checklist Manifesto, the New York Times bestseller by Atul Gawande, professionals who do highly technical work such as surgeons, airplane pilots, and architects use checklists to help guard against making avoidable mistakes.

One of the challenges traditional checklists pose in more creative industries, such as web design, is that they employ a highly defined set of tasks, which can limit creativity, especially as tasks change according to project.

Virginia Tech researchers, in collaboration with Manhattan-based web design startup B12, wondered whether it would be possible to take Gawande’s concept and apply it to a scenario with less rigidly defined tasks such as web design, and apply checklists more broadly to creatively driven tasks like writing or programming.

The research team will present their findings at the Association of Computing Machinery’s Human-Computer Interaction conference on Human Factors in Computing Systems in Glasgow, Scotland, on May 7, 2019.

Aditya Bharadwaj, a Ph.D. candidate in the Department of Computer Science in the College of Engineering and first author on the study, collaborated with B12 to study the workflow habits of designers. Many of B12’s design experts work remotely, making face-to-face review an impossibility.

Bharadwaj, who previously worked as a software developer at Paypal, first needed to identify where and what was causing the designers at B12 to overlook aspects of their work before passing it on for review by another team member.

The team used a framework called Critter as part of Orchestra, an open-source platform for managing teams of experts and machines. Critter gave the web designers at B12 the ability to make the checklists dynamic and either prune the checklists manually, or use the ability of the checklists to self-prune tasks.

One of the other benefits of using Critter was its AutoQA functionality, an interface embedded in B12’s browser-based website editor. AutoQA allowed designers to check for common errors without bringing in a reviewer, improving on the website so their reviewers could focus on the more challenging aspects of the project.

The Critter checklist pruned 20.4 percent of checklist items on each project and checked for things like image resolution and layout, and also displayed feedback on previous projects so designers could reference older projects for compliance.

“We saw a lot of qualitative results that indicated using the software helped junior designers in particular deliver a website with fewer errors,” said co-author Pao Siangliulue, tech lead at B12 and Bharadwaj’s mentor.

Early in Bharadwaj’s research he shadowed designers, which allowed him to easily identify their challenges in adhering to a flat and plain checklist. He used checklists as a rubric to provide contextual feedback and reinforce the importance of checklists as part of the design process. He saw that developers benefited from the hierarchically presented dynamic checklists, and they valued contextually provided feedback.

“Designers value human feedback and Critter helped a lot by allowing reviewers to focus on the most challenging parts of the design work after our automated approaches took care of low-hanging fruit,” said co-author Adam Marcus, co-founder and chief technology officer at B12.

The added flexibility of dynamic checklists paired with the efficiency of automated quality assurance allowed the company to scale its efforts while improving quality. The checklist-based feedback simultaneously gave B12’s design experts a better sense of where they needed to concentrate efforts and made them feel more like a cohesive unit.

“Web design is the digital outlet of commerce and communication today, so having the ability to give designers tools that could be helpful to them is a great feeling of satisfaction,” said Kurt Luther, assistant professor of computer science and co-author on the paper being presented.

While the team’s research mainly focused on website design, the research results from their work apply to other creative domains, such as writing and programming. The Critter system can be used to add structure to many broadly defined creative tasks while maintaining the dynamism and nonlinearity of the complex workflows associated with them.

###