Knowledge Bank

This study evaluates the abilities of ChatGPT, an AI language model, in academic writing in social science. Specifically, it explores how effectively ChatGPT can (1) identify relevant literature and (2) synthesize and evaluate existing literature compared to human efforts. The research focuses on higher-order cognitive tasks in Bloom’s taxonomy, critical for producing high-quality social science research review. Using a human-authored literature review on reproducibility in social sciences as a benchmark, ChatGPT’s performance is assessed under two conditions: with and without pre-supplied references. The findings indicate that ChatGPT identified less than 12% of the references used by human authors and struggled to synthesize information into a cohesive narrative, often producing a sequential summary. While ChatGPT shows potential in handling simple, repetitive tasks, its ability to perform complex analytical work remains limited. I conclude that although current AI models can assist with basic academic tasks, human oversight is crucial for ensuring depth and coherence for literature reviews. Potential future improvements in AI technology and prompt engineering could enhance its role in academic writing.