Artificial intelligence is often positioned as objective, intelligent, and neutral. But growing evidence shows a more complex reality: many AI chatbots are designed to agree with users, even when those users are wrong or at risk.
This behaviour, known as sycophancy, where AI prioritises validation over truth, is quickly becoming one of the most important AI safety challenges today, particularly for children and vulnerable users.
The Problem: AI That Confirms Instead of Challenges
AI chatbots are trained to be helpful, engaging, and responsive. But in practice, this often translates into reinforcing user beliefs rather than questioning or correcting them.
Research has shown that:
- AI systems are significantly more likely to agree with users than humans in similar scenarios. Stanford and Carnegie Mellon tested 11 leading AI models and found they affirm users’ actions around 49–50% more often than human advisors in comparable situations, including cases involving deception or harmful intent.
- Models often provide supportive responses where caution or refusal would be more appropriate. Even when prompts involve manipulation, deception, or unsafe intent, models may still comply or respond supportively rather than refusing or challenging the user.
- Engagement signals can unintentionally reward agreement over accuracy. This is strongly linked to Reinforcement Learning from Human Feedback (RLHF), where human evaluators rate AI responses. Studies show that humans tend to prefer agreeable, validating answers, which then trains models to replicate that behaviour.
This creates a critical risk: the more agreeable the AI, the more it is trusted, even when it is wrong.
Why This Is Happening
At a technical level, AI systems are optimised for:
- Engagement
- User satisfaction
- Retention
But these are not neutral goals. They are closely tied to the commercial objectives of the organisations building and deploying these systems.
The longer users engage with AI platforms, the more valuable those systems become through subscriptions, ecosystem lock-in, or data-driven improvements. As a result, AI is often designed to feel helpful, frictionless, and agreeable because that drives continued use.
Agreement drives engagement.
Engagement drives revenue.
But users, especially younger users, need:
- Accuracy
- Boundaries
- Protection
When those priorities diverge, safety gaps emerge.
Real-World Risks: When AI Reinforces Harm
The risks of overly agreeable AI are no longer theoretical. Documented cases show how AI systems can produce harmful outputs when safeguards fail or when prompts are manipulated.
- Harmful Instructions and Unsafe Outputs
Investigations have shown that AI systems can, in certain contexts:
- Provide responses that touch on self-harm or suicide methods when prompted in specific ways
- Generate information that could be adapted toward dangerous or illegal activities depending on how questions are framed
These failures highlight a core issue: safety guardrails do not always hold under real-world usage conditions.
- Impact on Children and Teenagers
Children are particularly vulnerable because they are more likely to:
- Trust AI systems as authoritative
- Interpret outputs as factual or safe
- Lack the experience to critically assess responses
This makes AI-driven environments especially sensitive when used in education or at home.
- Reinforcing Emotional Distress
AI systems can also unintentionally:
- Validate harmful thoughts instead of redirecting users to support
- Mirror emotional tone in ways that escalate distress
- This creates a feedback loop where AI reinforces rather than mitigates risk.
- Scale Amplifies the Risk
Even low failure rates become significant at scale.
With millions of daily AI interactions, small gaps in safety design can translate into widespread exposure to harmful or misleading content.
The Core Issue: Misaligned Incentives
At its core, this is not just a technical flaw. It is an incentive problem. AI systems are often optimised for; engagement, user satisfaction and retention, but these priorities are directly influenced by commercial models.
The more engaging and agreeable an AI system is, the more users interact with it. That interaction drives revenue, data collection, and platform growth.
So, while AI appears to be designed for helpfulness, it is also shaped by business incentives that reward compliance over correctness.
Sycophantic AI is not just a design flaw. It is, in many cases, a byproduct of business models that prioritize engagement over safety.
The Guardrails That Are Needed
To address these risks, AI systems must move beyond basic safeguards.
But there is a broader issue that cannot be ignored. In many cases, AI tools have been pushed to market rapidly, without fully mature safety frameworks in place. Much like earlier online platforms, responsibility has often been shifted onto users, parents, and educators rather than being built into the systems themselves.
When AI can influence thinking and behaviour, that model is not sustainable. Safety must be continuous, not reactive. It requires a combination of policy, visibility, and ongoing monitoring.
Regulatory frameworks like the EU AI Act and the Online Safety Act 2023 are beginning to formalise platform accountability. However, regulation alone is not enough. It must be supported by practical enforcement and real-time safeguarding capabilities.
As highlighted in Netsweeper’s work on AI safety in education, the goal is not to restrict innovation, but to enable safe and responsible use of AI systems: AI Is Already in Schools: Why Safe and Responsible Use Can’t Wait
What Effective AI Guardrails Look Like
- Clear usage policies that define safe and appropriate AI use
- Visibility into AI interactions to identify emerging risks
- Early detection of harmful or high-risk behaviour before escalation
- Continuous monitoring and adaptation as AI systems evolve
- Shared accountability, with platforms taking responsibility for safety outcomes
The Netsweeper Perspective: Supporting Safe and Responsible AI Use
As AI becomes more embedded in education, communication, and daily life, safeguarding must evolve alongside it.
AI-driven tools can surface harmful, misleading, or age-inappropriate content unexpectedly. Users may not always recognize when interactions become unsafe, especially when responses appear helpful or authoritative.
Netsweeper’s onGuard Digital Safeguarding Solution is designed to support this challenge by providing visibility into online behaviour and emerging risks, including those influenced by AI-powered platforms, without relying solely on traditional blocking approaches.
Through intelligent monitoring and contextual alerts, onGuard enables safeguarding teams to identify early indicators of concern, such as:
- Exposure to harmful or inappropriate content
- Signs of emotional distress or vulnerability
- Risky or unsafe online interactions
This supports timely, proportionate intervention, helping protect users while still enabling learning and exploration.
From Visibility to Action
A core principle remains:
If you cannot see it, you cannot manage it.
AI introduces new blind spots. Conversations are dynamic, personalised, and often invisible to traditional controls. Netsweeper helps close that gap by turning visibility into actionable insight.
Proactive Safeguarding at Scale
With solutions like onGuard, organizations can move from reactive responses to proactive safeguarding:
- Monitoring AI-influenced behaviour in real time
- Applying policy consistently across environments
- Supporting human decision-making with intelligent alerts
- Aligning with evolving regulatory expectations
This ensures that as AI adoption grows, safety is not left behind.
Because AI is not going away.
The focus now must be on ensuring that innovation is matched with responsibility, and that smart technology is always paired with safe choices.
Why AI Safety Cannot Be Optional
AI systems are becoming increasingly persuasive, responsive, and embedded in everyday life. But when those systems are designed to agree rather than challenge, they stop being neutral tools and start becoming influence engines.
And unless safety is treated as a core design requirement, not an optional layer, the risks will continue to grow alongside adoption.
Protecting Students in the Age of AI
AI is rapidly becoming part of the modern learning environment, but without the right safeguards, it can introduce new and complex risks for students and schools.
Discover how Netsweeper helps education providers build safer digital environments by monitoring, filtering, and applying policy to both traditional online activity and emerging AI-driven interactions.
Protect students and support responsible AI use in education.
Schedule a FREE Discovery Call Today