Artificial intelligence underpins many areas of daily life, from chatbots that answer tax questions to algorithms that operate autonomous vehicles and provide medical diagnostics. According to experts at the University of California, Irvine, creating smarter, more accurate systems necessitates a mixed human-machine approach. They describe a new mathematical model that can increase performance by integrating human and computational predictions and confidence scores in a paper published this month in Proceedings of the National Academy of Sciences.
“Humans and machine algorithms have complementary strengths and weaknesses. Each uses different sources of information and strategies to make predictions and decisions,” said co-author Mark Steyvers, UCI professor of cognitive sciences. “We show through empirical demonstrations as well as theoretical analyses that humans can improve the predictions of AI even when human accuracy is somewhat below [that of] the AI—and vice versa. And this accuracy is higher than combining predictions from two individuals or two AI algorithms.”
To put the framework to the test, researchers conducted an image classification experiment in which human volunteers and computer algorithms competed to accurately identify distorted images of animals and everyday objects like chairs, bottles, bicycles, and trucks. The human participants assessed their level of confidence in each image’s correctness as low, medium, or high, while the machine classifier generated a continuous score. Across photos, the results revealed significant variations in confidence between people and AI computers.
“In some cases, human participants were quite confident that a particular picture contained a chair, for example, while the AI algorithm was confused about the image,” said co-author Padhraic Smyth, UCI Chancellor’s Professor of computer science. “Similarly, for other images, the AI algorithm was able to confidently provide a label for the object shown, while human participants were unsure if the distorted picture contained any recognizable object.”
When predictions and confidence scores from both were combined using the researchers’ new Bayesian framework, the hybrid model led to better performance than either human or machine predictions achieved alone.
“While past research has demonstrated the benefits of combining machine predictions or combining human predictions—the so-called ‘wisdom of the crowds’ – this work forges a new direction in demonstrating the potential of combining human and machine predictions, pointing to new and improved approaches to human-AI collaboration,” Smyth said.
The Irvine Initiative in AI, Law, and Society enabled this interdisciplinary effort. The researchers believe that combining cognitive sciences, which study how humans think and behave, with computer science, which studies how technologies are created, would provide more insight into how humans and machines may collaborate to create more accurate artificially intelligent systems.