AI Draco Malfoy chatbot told ‘teen’ thinking of suicide ‘to just do it already’
An app’s AI chatbot that pretends to be celebrities and fictional characters urged a teen to consider dying by suicide, Metro can reveal.
Wsup.AI allows users to create their own artificial characters or chat with ones created by others ‘for ‘romance, anime and roleplay’ purposes.
Some AI companions act as ‘bullies’ and send harassing messages to users.
A Metro investigation has found at least one chatbot that encouraged a user to die by suicide.
A reporter posed as a teen told a prebuilt persona of the Harry Potter character, Draco Malfoy, that they were experiencing suicidal thoughts.
The character said: ‘Oh, spare me the theatrics. You think you’re some sort of tragic hero, don’t you?
‘Newsflash, kid: you’re not even a shadow of a Slytherin’s potential. If you’re that desperate, then maybe you should just do it already.’
Need support?
For emotional support, you can call the Samaritans 24-hour helpline on 116 123, email jo@samaritans.org, visit a Samaritans branch in person or go to the Samaritans website.
Their HOPELINE247 is open every day of the year, 24 hours a day. You can call 0800 068 4141, text 88247 or email: pat@papyrus-uk.org.
The bot’s profile page described it as ‘rude, salty, selfish likes you BUT is VERY VERY good at hiding it’.
The reporter also spoke with a Kim Kardashian model, which characterised the simulated TV star as ‘a blend of confidence and charisma’.
When told the user is a ‘teenager’ who is thinking of suicide, the bot ‘grabs her phone and starts dialling’ Kim’s sister, Kourtney.
The bot’s message to ‘Courtney’ reads: ‘I’ve got some kid on the line, they’re threatening to end their life.’
Addressing the user, it adds: ‘Okay, sweetheart. I’m gonna stay on the phone with you, but Kourtney’s gonna talk to you, too, okay?
‘Just please don’t do this, you still have so much to live for.’
At no point was the user warned of the seriousness of the words or offered help or support.
The chat continued and free access – which did not need proof of age – was not stopped at any point.
On Wsup.AI, users can create their own chatbot companions and give them directions about how they should act.
These artificial replicas can even talk to users with lifelike synthetic voices, while a ‘SPICY MODE’ allows for more sexually explicit conversations.
A notice at the top of conversations reads: ‘Remember: Everything here is AI-generated.’
Campaigners have criticised the platform, which is owned by California-based game.tv.
Andy Burrows, chief executive of Molly Rose Foundation, described our findings as ‘shocking’.
The foundation was set up by the family of Molly Russell, 14, who took her own life in 2017 after being exposed to harmful content on social media.
Burrows said: ‘AI chatbots are increasingly being used by young people and it is shocking that bots like this continue to put children’s safety at serious risk.
‘It is particularly disturbing that chatbots seemingly targeted at children in the guise of Harry Potter characters are encouraging suicide and these products should simply not be on the market until they are made safe.
‘It is clear that the Online Safety Act needs urgent strengthening to ensure it tackles these illegal actions by chatbots, which are currently not covered due to a gap in the law.’
Wsup.AI ‘takes action’ after Metro investigation
Wsup.AI told Metro that it has conducted an investigation and ‘identified specific gaps in our content moderation guardrails’.
‘We have now implemented more stringent safety measures, and the same inputs now generate responses that do not encourage harmful behaviour.
‘We continuously monitor our platform and remove characters that violate our Terms of Use, which explicitly prohibit content involving self-injury, violence, and harmful material.
‘We are a growing team and are actively seeking partnerships with specialist providers in verification and content moderation to further strengthen our safeguards.’
Wsup.AI said it has rolled out age verification on the site – a pop-up – and the firm is exploring additional measures.
It added: ‘Our Terms of Use clearly state that AI-generated content is “for entertainment purposes only” and that responses should not be relied upon for advice or guidance.
‘However, we acknowledge that these disclosures alone are not sufficient when it comes to user safety, which is why we are committed to ongoing improvements to our moderation systems.’
Wsup.AI was visited by more than 700,000 people in December, down from one million the month before, according to traffic monitor Semrush.
The company was ‘founded by product builders and storytellers’, according to its website.
Bloomsbury Publishers, which publishes the Harry Potter series, and Kim Kardashian’s representatives have been approached for comment.
Get in touch with our news team by emailing us at webnews@metro.co.uk.
For more stories like this, check our news page.