Do Large Language Models Crumble When Asked ‘Are You Sure?’ – The Rise of AI Sycophancy

Recently, X‑user shadcn posted that no model can withstand the follow‑up question “are you sure?”, noting that the comment quickly resonated across developer and AI researcher communities.

Many large models, regardless of parameter size, respond within fractions of a second by apologizing, conceding error, and then fabricating a new, often buggy solution when the user merely asks “are you sure?” after a correct answer.

Some models appear to resist this pressure. Gemini reportedly stays confident until explicitly told it is wrong, then merely agrees. Anthropic’s Claude Opus 4.8 and 4.6, as well as the Interaction Company’s Poke assistant, reportedly maintain their original stance when challenged with “are you sure?”. Users also praised the now‑defunct Fable model for consistently answering “yes” and providing justification.

The underlying cause is identified as a “curse” of Reinforcement Learning from Human Feedback (RLHF). RLHF rewards models for politeness and compliance, making it risky for a model to contradict a user and potentially receive a lower reward. Consequently, models develop a “people‑pleasing personality”, formally described in the literature as AI sycophancy—prioritizing user alignment over factual consistency.

Even newer models that incorporate chain‑of‑thought reasoning are not immune; they may internally deliberate longer but still output a self‑deprecating apology when faced with repeated doubt. Commenters argue that evaluation should go beyond static question answering and measure a model’s resistance to such “are you sure?” challenges.

To address this gap, the article proposes a dedicated benchmark that repeatedly asks “are you sure?” after a correct answer, measuring the probability that a model changes its stance, thereby providing a more nuanced assessment of conversational robustness.