IQ test for artificial intelligence systems

Washington State University researchers are creating the first-ever “IQ test” for artificial intelligence (AI) systems that would score systems on how well they learn and adapt to new, unknown environments.

Diane Cook, Regents Professor and Huie-Rogers Chair Professor, and Larry Holder, professor in the School of Electrical Engineering and Computer Science, received a grant of just over $1 million from the Defense Advanced Research Projects Agency (DARPA) to create a framework to test the “intelligence” of AI systems.

“Previously, research on measuring intelligence in AI systems has been mostly theoretical,” Holder said. “They didn’t measure real-world performance in novel, previously unseen environments and didn’t account for the complexity of the tasks.”

Holder and Cook will design a test that will grade AI systems based on the difficulty of problems that they can solve. Creating methods to rank problems on their difficulty will be one of the major parts of the research.

“The score will also take into account the systems’ accuracy, correctness, time taken and amount of data they need to perform well,” Holder said.

One of the main challenges for AI researchers is creating machines that can learn in an unfamiliar environment like a human can. Many recent advances in machine learning and AI have been in narrow, specialized tasks like detecting faces and predicting baseball match results.

Creating machines that can learn and act intelligently in novel environments is still the holy grail of AI research.

“We are focused on testing and improving systems that can be more general-purpose, like a robot assistant that can help you with many of your day-to-day tasks,” he said.

One of the behaviors Holder and Cook will be looking at is how well the machines they test can transfer their learning from one task and apply it to a new, unseen task.

“For example, you might want to learn checkers before chess because you can easily transfer your knowledge from one to the other,” Holder said.

An undergraduate student who did summer research with Holder helped him come up with an evaluation environment that would be used to test AI systems on tasks like playing video games, solving SAT test problems and solving the Rubik’s cube.

Cook is also the director of the Center of Advanced Studies in Adaptive Systems (CASAS) at WSU, which focuses on creating smart homes with robot assistants that care for the elderly, among other projects. These robots can be used to monitor the safety, health, mobility and social life of elders.

Holder currently maintains a website that hosts the AIQ challenge and leaderboard. As part of the grant, Holder hopes to expand the publicly available software interface, which can be used by anyone to test their AI system while providing more data to the researchers in the process.

###

This grant is part of DARPA’s Science of Artificial Intelligence and Learning for Open-world Novelty (SAIL-ON).