Comparing AI-led to Human-led Chat-based Interviews: Motivations, Initial Results, and Challenges

Abstract: Chatbots powered by large language models (LLMs) have been proposed as AI-interviewers capable of collecting large-scale qualitative interview data. This paper asks whether data collected through AI-led interviews systematically differs from human-led chat interviews. In a randomized experiment (N=40), participants were assigned to synchronous text-based interviews conducted by either human interviewers or a locally hosted AI system. Human interviewers elicited longer responses per question, whereas AI interviewers conducted longer interviews overall due to faster question delivery. No significant differences in response quality—measured by specificity and relevance—were observed. These findings support the viability of AI-interviewing as a method for large-N qualitative data collection, particularly when implemented with locally hosted, open-weight LLMs that improve ethical standards, data security, and reproducibility.