GPT-3 performs on par with college undergraduates in reasoning tests, making similar mistakes, study finds

livemint.com:

OpenAI's widely recognized AI-powered tool, GPT-3, has been discovered to demonstrate reasoning abilities on par with college undergraduate students, according to scientific research. During the study, the artificial intelligence large language model (LLM) was presented with reasoning problems commonly found in intelligence tests and standardized exams like the SAT, which are crucial in the admissions process for colleges and universities in the United States and other nations, reported PTI. The findings highlight the impressive capabilities of GPT-3 in tackling complex cognitive tasks.

As per the report, at the University of California - Los Angeles (UCLA), researchers conducted an experiment involving GPT-3. The AI was tasked with predicting the subsequent shape in intricate arrangements of shapes and answering SAT analogy questions, both of which were entirely new to the AI. Additionally, the researchers invited 40 UCLA undergraduate students to attempt the same problems.

In the shape prediction test, GPT-3 achieved an accuracy of 80%, surpassing the human participants' average score, which was slightly below 60%, and even outperforming their highest scores. "Surprisingly, not only did GPT-3 do about as well as humans but it made similar mistakes as well," said UCLA psychology professor Hongjing Lu, senior author of the study published in the journal Nature Human Behaviour. Regarding the SAT analogies, GPT-3 exhibited superior performance compared to the average score of the human participants.