AI-generated images of young girls, some as young as five, are spreading on TikTok and Instagram, drawing inappropriate comments from a troubling audience consisting mostly of older men, Forbes uncovers. These images depict children in provocative outfits, sparking serious concerns, and while the images are not illegal, they are highly sexualised, prompting child safety experts to warn about their potential to lead to more severe exploitation.
It is no wonder they are causing a sense of imminent danger, with platforms like TikTok and Instagram, popular with minors, struggling to address this issue. One popular account, “Woman With Chopsticks,” had 80,000 followers and viral posts viewed nearly half a million times across both platforms. A recent study by Stanford revealed that the AI tool Stable Diffusion 1.5 was developed using child sexual abuse material (CSAM) involving real children collected from various online sources.
Under federal law, tech companies must report suspected CSAM and exploitation to the National Center for Missing and Exploited Children (NCMEC), which then informs law enforcement. However, they are optional to remove the type of images discussed here. Nonetheless, NCMEC believes that social media companies should remove these images, even if they exist in a legal grey area.
TikTok and Instagram assert that they have strict policies against AI-generated content involving minors to protect young people. TikTok bans content showing anyone under 18, while Meta removes material that sexualises or exploits children, whether real or AI-generated. Both platforms removed accounts and posts identified by Forbes. However, despite strict policies, the ease of creating and sharing AI-generated images will certainly remain a significant challenge for safeguarding children online.
Why does it matter?
The Forbes story reveals that such content, which has increasingly become easy to find due to powerful algorithm recommendations, worsens online child exploitation, acting as gateways to severe material exchange and facilitating offender networking. A 13 January TikTok slideshow of young girls in pyjamas found by the investigation showed users moving to private messages. The Canadian Centre for Child Protection stressed that companies need to look beyond automated moderation to address how these images are shared and followed.
The EU regulators announced on Thursday that Meta Platforms’ social media platforms, Facebook and Instagram, will undergo investigation for potential violations of the EU online content rules about child safety, potentially resulting in significant fines. The scrutiny follows the EU’s implementation of the Digital Services Act (DSA) last year, which places greater responsibility on tech companies to address illegal and harmful content on their platforms.
The European Commission has expressed concerns that Facebook and Instagram have not adequately addressed risks to children, prompting an in-depth investigation. Issues highlighted include the potential for the platforms’ systems and algorithms to promote behavioural addictions among children and facilitate access to inappropriate content, leading to what the Commission refers to as ‘rabbit-hole effects’. Additionally, concerns have been raised regarding Meta’s age assurance and verification methods.
Why does it matter?
Meta, formerly known as Facebook, is already under the EU scrutiny over election disinformation, particularly concerning the upcoming European Parliament elections. Violations of the DSA can result in fines of up to 6% of a company’s annual global turnover, indicating the seriousness with which the EU regulators are approaching these issues. Meta’s response to the investigation and any subsequent actions will be closely monitored as the EU seeks to enforce stricter regulations on tech giants to protect online users, especially children, from harm.
OpenAI is sparking debate by considering the possibility of allowing users to generate explicit content, including pornography, using its AI-powered tools like ChatGPT and DALL-E. While maintaining a ban on deepfakes, OpenAI’s proposal has raised concerns among campaigners who question its commitment to producing ‘safe and beneficial’ AI. The company sees potential for ‘not-safe-for-work’ (NSFW) content creation but stresses the importance of responsible usage and adherence to legal and ethical standards.
The proposal, outlined in a document discussing OpenAI’s AI development practices, aims to initiate discussions about the boundaries of content generation within its products. Joanne Jang, an OpenAI employee, stressed the need for maximum user control while ruling out deepfake creation. Despite acknowledging the importance of discussions around sexuality and nudity, OpenAI maintains strong safeguards against deepfakes and prioritises protecting users, particularly children.
Critics, however, have accused OpenAI of straying from its mission statement of developing safe and beneficial AI by delving into potentially harmful commercial endeavours like AI erotica. Concerns about the spread of AI-generated pornography have been underscored by recent incidents, prompting calls for tighter regulation and ethical considerations in the tech sector. While OpenAI’s policies prohibit sexually explicit content, questions remain about the effectiveness of safeguards and the company’s approach to handling sensitive content creation.
Why does it matter?
As discussions unfold, stakeholders, including lawmakers, experts, and campaigners, closely scrutinise OpenAI’s proposal and its potential implications for online safety and ethical AI development. With growing concerns about the misuse of AI technology, the debate surrounding OpenAI’s stance on explicit content generation highlights broader challenges in balancing innovation, responsibility, and societal well-being in the digital age.
Social media platforms such as Facebook, Instagram, and TikTok face proposed measures in the UK to modify their algorithms and better safeguard children from harmful content. These measures, outlined by regulator Ofcom, are part of the broader Online Safety Act and include implementing robust age checks to shield children from harmful material related to sensitive topics like suicide, self-harm, and pornography.
Ofcom’s Chief Executive, Melanie Dawes, has underscored the situation’s urgency, emphasising the necessity of holding tech firms accountable for protecting children online. She asserts that platforms must reconfigure aggressive algorithms that push harmful content to children and incorporate age verification mechanisms.
The utilisation of complex algorithms by social media companies to curate content has raised serious concerns. These algorithms often amplify harmful material, potentially influencing children negatively. The proposed measures seek to address this issue by urging platforms to reevaluate their algorithmic systems to prioritize child safety by providing children with a safer online experience tailored to their age.
UK’s Technology Secretary, Michelle Donelan, called for social media platforms to engage with regulators and proactively implement these measures, cautioning against waiting for enforcement and potential fines. After a consultation, Ofcom plans to finalise its Children’s Safety Codes of Practice within a year, with anticipated enforcement actions, including penalties for non-compliance, once parliament approves.
Kyrgyzstan has banned TikTok following security service recommendations to safeguard children. The decision comes amid growing global scrutiny over the social media app’s impact on children’s mental health and data privacy.
The Kyrgyz digital ministry cited ByteDance’s failure to comply with child protection laws, sparking concerns from advocacy groups about arbitrary censorship. The decision reflects Kyrgyzstan’s broader trend of tightening control over media and civil society, departing from its relatively open stance.
This decision stems from legislative text approved last summer aimed at curbing the distribution of ‘harmful’ online content accessible to minors. Such content encompasses material featuring ‘non-traditional sexual relationships’ and those that undermine ‘family values,’ as well as promoting illegal conduct, substance abuse, or anti-social behaviours. Chinese officials have not publicly commented on this decision, although in March, Beijing accused the US of ‘bullying’ over similar actions against TikTok.
A convicted sex offender in the UK has been banned from using ‘AI-creating tools’ for five years, marking the first known case of its kind. Anthony Dover, 48, received the prohibition as part of a sexual harm prevention order, preventing him from accessing AI generation tools without prior police permission. This includes text-to-image generators and ‘nudifying’ websites used to produce explicit deepfake content.
Dover’s case highlights the increasing concern over the proliferation of AI-generated sexual abuse imagery, prompting government action. The UK recently introduced a new offence making it illegal to create sexually explicit deepfakes of adults without consent, with penalties including prosecution and unlimited fines. The move aims to address the evolving landscape of digital exploitation and safeguard individuals from the misuse of advanced technology.
Charities and law enforcement agencies emphasise the urgent need for collaboration to combat the spread of AI-generated abuse material. Recent prosecutions reveal a growing trend of offenders exploiting AI tools to create highly realistic and harmful content. The Internet Watch Foundation (IWF) and the Lucy Faithfull Foundation (LFF) stress the importance of targeting both offenders and tech companies to prevent the production and dissemination of such material.
Why does it matter?
The decision to restrict an adult sex offender’s access to AI tools sets a precedent for future monitoring and prevention measures. While the specific reasons for Dover’s ban remain unclear, it underscores the broader effort to mitigate the risks posed by digital advancements in sexual exploitation. Law enforcement agencies are increasingly adopting proactive measures to address emerging threats and protect vulnerable individuals from harm in the digital age.
European regulators have demanded a risk assessment from TikTok within 24 hours regarding its new app, TikTok Lite, recently launched in France and Spain. The European Commission, under the Digital Services Act (DSA), is concerned about potential impacts on children and users’ mental health. This action follows an investigation initiated two months ago into TikTok for potential breaches of the EU tech rules.
Thierry Breton, the EU industry chief, emphasised the need for TikTok to conduct a risk assessment before launching the app in the 27-country EU. The DSA requires platforms to take stronger actions against illegal and harmful content, with penalties of up to 6% of their global annual turnover for violations. Breton likened the potentially addictive and toxic nature of ‘social media lite’ to ‘cigarettes light,’ underlining the commitment to protecting minors under the DSA.
TikTok Lite, targeted at users aged 18+, includes a ‘Task and Reward Lite’ program that allows users to earn points by engaging in specific platform activities. These points can be redeemed for rewards like Amazon vouchers, PayPal gift cards, or TikTok coins for tipping creators. The Commission expressed concerns about the app’s impact on minors and users’ mental health, particularly potential addictive behaviours.
Why does it matter?
TikTok has been directed to provide the requested risk assessment for TikTok Lite within 24 hours and additional information by 26 April. The Commission will analyse TikTok’s response and determine the next steps. TikTok has acknowledged the request for information and stated that it is in direct contact with the Commission regarding this matter. Additionally, the Commission has asked for details on measures implemented by TikTok to mitigate systemic risks associated with the new app.
Meta CEO Mark Zuckerberg has secured the dismissal of certain claims in multiple lawsuits alleging that Facebook and Instagram concealed the harmful effects of their platforms on children. US District Judge Yvonne Gonzalez Rogers in Oakland, California, ruled in favour of Zuckerberg, dismissing claims from 25 cases that sought to hold him personally liable for misleading the public about platform safety.
The lawsuits, part of a broader litigation by children against social media giants like Meta, assert that Zuckerberg’s prominent role and public stature required him to fully disclose the risks posed by Meta’s products to children. However, Judge Rogers rejected this argument, stating it would establish an unprecedented duty to disclose for any public figure.
Despite dismissing claims against Zuckerberg, Meta remains a defendant in the ongoing litigation involving hundreds of lawsuits filed by individual children against Meta and other social media companies like Google, TikTok, and Snapchat. These lawsuits allege that social media use led to physical, mental, and emotional harm among children, including anxiety, depression, and suicide. The plaintiffs seek damages and a cessation of harmful practices by these tech companies.
Why does it matter?
The lawsuits highlight a broader concern about social media’s impact on young users, prompting legal action from states and school districts. Meta and other defendants deny wrongdoing and have emphasised their commitment to addressing these concerns. While some claims against Zuckerberg have been dismissed, the litigation against Meta and other social media giants continues as plaintiffs seek accountability and changes to practices allegedly detrimental to children’s well-being.
The ruling underscores the complex legal landscape surrounding social media platforms and their responsibilities regarding user safety, particularly among younger demographics. The outcome of these lawsuits could have significant implications for the regulation and oversight of social media companies as they navigate concerns related to their platforms’ impact on mental health and well-being.
The Belgian EU Council Presidency has introduced a compromise text aimed at detecting and preventing online child sexual abuse material (CSAM). The proposal refines risk categorisation thresholds and outlines data retention obligations for service providers.
However, it has faced criticism for potentially allowing authorities to scan private messages on platforms like WhatsApp or Gmail. The draft legislation enables service providers to flag potential abuse, triggering detection orders mandating active searching for abusive content. Providers are also required to assist the newly established EU Centre by conducting audits at the source code level to combat such material.
Additionally, the compromise introduces specific risk thresholds for categorising service providers. It emphasises adherence to data processing principles, particularly focusing on lawfulness, purpose limitation, and data minimisation in age verification measures.
Why does it matter?
The EU’s proposed legislation to employ surveillance technologies for detecting CSAM in digital messaging faced further scrutiny as the Commission’s ombudsman criticised the lack of transparency in communications with a child safety tech company early this year. Critics argue that the proposal poses risks to privacy and fundamental freedoms and suggest lobbyists influenced it.
Since the law needs approval from the Commission, Council, and Parliament, the next step with the CSAM proposal remains to be determined.
Meta, Instagram’s parent company, has announced plans to trial new features aimed at protecting teens by blurring messages containing nudity. This initiative is part of Meta’s broader effort to address concerns surrounding harmful content on its platforms. The tech giant faces increasing scrutiny in the US and Europe amid allegations that its apps are addictive and contribute to mental health issues among young people.
The proposed protection feature for Instagram’s direct messages will utilise on-device machine learning to analyse images for nudity. It will be enabled by default for users under 18, with Meta urging adults to activate it as well. Notably, the nudity protection feature will operate even in end-to-end encrypted chats, ensuring privacy while maintaining safety measures.
Meta is also developing technology to identify accounts potentially involved in sextortion scams and is testing new pop-up messages to warn users who may have interacted with such accounts. These efforts come after Meta’s previous announcements regarding increased content restrictions for teens on Facebook and Instagram, particularly concerning sensitive topics like suicide, self-harm, and eating disorders.
Why does it matter?
The company’s actions follow legal challenges, including a lawsuit filed by 33 US states alleging that Meta misled the public about the dangers of its platforms. The European Commission has also requested information on Meta’s measures to protect children from illegal and harmful content in Europe. As Meta continues to navigate regulatory and public scrutiny, its focus on enhancing safety features underscores the ongoing debate surrounding social media’s impact on mental health and well-being, especially among younger users.
