Interview: X’s new restrictions fail to protect users from AI image manipulation

Being online, sharing pictures and videos of ourselves with our friends, increasingly comes with the risk that our personal content can easily be stolen and altered in a matter of seconds.

Image theft, manipulation and reproduction has become more accessible than it has ever been, all due to AI tools that are evolving far more quickly than regulation mechanisms.

The results often affect lives offline just as much as online.

The increasingly accessible technology requires no expertise or advanced technical capacity, with users given access at their fingertips and near total freedom to take or generate any type of image they want.

The issue has come into sharp relief this month in light of X’s AI chatbot, Grok, which users have used to generate non-consensual, fake and often sexualized or violent versions of images posted online. Many of these images were of women and children.

As the images became a trend, the platform’s timeline was quickly flooded with users tagging Grok in comments under other people’s images, asking the bot to put the subjects in revealing clothing, to remove their clothing entirely or to physically disfigure the people depicted, with some being given fake bruises or injuries, for example.

Amid backlash from users, who contested the growing practice and the platform’s failure to regulate the use of Grok’s features, X’s first response earlier this month was to limit the use of image generation and editing AI tools to its paid subscribers.

But the measure proved to be lacking. As paid users continued to generate sexualized images, others mounted their criticism of the platform and were joined by several officials who began taking legal actions against the platform, with some launching investigations.

A day after the Californian state attorney general began an investigation into whether xAI — the firm owning X — had violated state law, the firm announced that they had rolled out “technological measures” to prevent Grok from allowing the “editing of images of real people in revealing clothing such as bikinis.”

The platform added it had “zero tolerance for any forms of child sexual exploitation, nonconsensual nudity, and unwanted sexual content.”

X’s Safety account said that the new regulations include steps to “remove high-priority violative content” and take “appropriate actions” against users who violate X rules. It also said that it will geoblock users from generating sexualized content for users in jurisdictions where the practice is illegal.

The platform did not specify the jurisdictions in question, whether the restrictions would apply to the Grok application itself, or only the Grok feature on X. It did not mention channels for affected users to submit related complaints.

The company's owner, Elon Musk, made a post on the same day, saying he was not aware that Grok was generating “naked underage images.”

And days after the restrictions were rolled out, global news outlets began reporting that the practice was still accessible to users on the Grok application.

While governments in Europe and the United States have taken action against the use of the Grok feature, women in Egypt have also been affected by the abuse of the feature.

Eman Mohamed told media outlets that a selfie she had posted on the platform earlier this month was manipulated by another user using the Grok feature to depict her in a bikini. She reported the generated picture to X and filed a police report against the user. X removed the image, but there’s no guarantee it doesn’t exist elsewhere.

And the repercussions of image theft and manipulation can be dire, exposing the subject of the images to various forms of community or social violence, ranging from shame to blackmail and extortion.

In 2022, Bassant Khaled, a 17-year old living in Gharbiya, took her own life after a man she had refused to go on a date with published images of her online. She said the images had been taken without her consent and digitally altered.

To understand more about the implications of X’s Grok feature, and whether the new restrictions on its use will protect users in Egypt, Mada Masr spoke with Tasneem Mounir, a digital rights expert and digital security helpdesk coordinator at SuperWomen Initiative — a hotline that works on digital security literacy and protection for women in Egypt.

Mounir talked us through how the tool, and its potential for abuse, remains widely accessible, while there is still little to no support for women or other users whose images are altered without their consent.

As the use of automation and AI becomes more and more prevalent, complainants have less and less access to in-person support.

***

Mada Masr: xAI says it has rolled out restrictions and policy updates based on the wave of complaints about Grok manipulating images to create unwanted sexual content or facilitate child exploitation through image theft and manipulation. Based on your experience in the field of supporting women facing digital violence, how effective are the new measures?

Tasneem Monir: The decision explicitly says that it will prohibit the generation of images with very revealing clothing, such as bikinis and “extra revealing” bikinis, and similarly revealing outfits.

This raises a broader question: when we speak of digital defamation — as an umbrella term for these kinds of abuses — revealing images or those that emphasize certain parts of a woman’s body may be relevant in some Western contexts, but in other settings, including across the Middle East and more conservative or religious societies, different forms of image alteration can constitute equally unwanted and harmful “exposure.”

For example, if I create an image of a girl wearing a cropped T-shirt while she is veiled or even wearing a niqab — then clearly X's decision does not protect us from that scenario. What about, for example, a crop top and a short skirt? This clothing is not prohibited, as per X’s new rules, but it is read across as implying that the wearer is being sexually suggestive, even if the woman herself has not written or said anything that indicates that.

[Unwanted exposure] can be indirect, through a pose, or whatever.

In the case of the TikTok girls [who were digitally pursued and arrested], people considered them to be emphasizing “feminine” areas of the body regardless of their clothing.

And what is the mechanism X will use to prevent other issues, poses and images that involve violence, for example?

xAI also announced that it will ban the production of sexualized images “in jurisdictions where it is illegal.” Fine — but what about countries that cannot issue orders to X? How will you protect users there?

What about jurisdictions where there isn't digital monitoring or oversight? In Egypt, there is digital monitoring. But for who?

MM: So how will these restrictions apply to people in Egypt? Where are the gaps and what can be done about them? How can X ensure that its new safeguards take cultural contexts into account — namely conservative societies — and not solely rely on narrow definitions of nudity and revealing clothing?

TM: Exactly. I want to understand what this will look like in practice.

You are telling us: fine, we will no longer generate images that involve nudity or defamation, and so on. But if I come to you and say: this image does not contain nudity, it contains normal clothes, but in the Middle East this is considered [defamation] — what will you do then?

And what is your plan to improve the idea of trusted partners who run helplines? Will you listen to us or not? How will you measure this? Will you issue periodic reports about the use of AI for nudity, revealing clothing, and related issues? Will you improve response mechanisms and speed? Will you listen to us about local context, or will we mostly end up speaking to AI bots?

What about women who do not even know that trusted partners exist? They will contact the platform itself. In that case, how long will it take before a human agent speaks to me — not an AI system that does not understand the risks posed to me in my context?

Let us move away from revealing clothing. You generate an image of me, a veiled woman, wearing my headscarf, but you also place me sitting with my former fiancé — or even fabricate an image of us sitting together in a public place. This can be interpreted as infidelity.

What if the images are not explicitly nude, but contain suggestions, or give the impression that this is a romantic place or a private home? What if I am fully dressed and you place me in an image sitting and drinking tea, but with a man, inside a house?

For me, the key issue here is context. Where is the human, contextual element? The human element that can assess the situation and understand whether this is defamation?

MM: So if I ask you, what is the best way for women to report these types of incidents given the contextual differences you just mentioned, would you say the trusted helplines?

TM: Exactly, yes. If there is a trusted partner that has an open channel with the platform, whatever that channel may be, that would be ideal.

Of course I wish these helplines were advertised everywhere so that all women would know about them — but the reality is that many do not know about these services.

And even if all women in Egypt did know about them, it would be a disaster: these helplines would collapse and would not be able to handle the volume of calls from all over the country in all kinds of contexts — like Bassant, who was from a small village in Gharbiya. The core issue is the spread of these images [everywhere]. How are you going to control that?

MM: The recent decision to prevent Grok from generating sexualized images does not address content already produced by the tool. What about what has already happened — what is the current situation regarding images that have already been generated?

TM: They mention government regulations in the decision. Ok, they will respond to some removal requests, but as I said, the removals will be limited.

We normally refer women to tools like Stop NCII (non-consensual intimate images). This is a very useful tool that uses a nice technique where it does not actually save or store the images themselves. For example, when I submit a report on a published  image, they don’t store it, they create a file — a hash file, let’s say — like a fingerprint, a unique identifier. Even if I later rename it, change its format, or manipulate it in different ways, the hash remains identifiable. If I upload a file hash to NCII for an image of a woman being exposed or sexualized in a way she is not comfortable with, they can prevent the circulation of that image.

If I took the image from X, they would block it there, on Facebook and any other platforms they have access to. But NCII is still limited to images of nudity or partial nudity.

And about three quarters of sharing happens in inboxes. So are you also going to do something about inboxes? Or encrypted platforms like Telegram?

So far, I don’t see clear commitments beyond limited definitions of “revealing clothing.”

Do you have fast reporting mechanisms from individuals themselves — not only through helplines — that you can actually act on?

The speed at which the content spreads is faster than your ability to control it.

What about remedies for the harm done to individuals? What exactly are you doing in terms of redress for these people? For example, do you prevent offenders from creating new accounts? What tools do you have for that?

They also talk about blocking these features based on users’ geographic locations, in places where this is legally prohibited. But what if I turn on a VPN? I don’t really see any measurable procedures here.

MM: Are there any targets that you think should be in place to improve the situation?

For me, we need people from the Middle East who can engage with these cases at scale, because you have to listen to people. Algorithms have limits when it comes to verification, protecting users and understanding context.

I did not see anything clear in this decision about human involvement, confrontation, protection, or fast reporting.

For me, there should be something to prevent the re-upload of content involving humiliation or degrading treatment before we can say that these measures are effective, and there should be location-specific measures, so that at the very least this content is not circulated in the woman’s own country.

If we cannot control everything worldwide, which is genuinely difficult, then at least it should not be spread in her country, for her safety.

I do not feel [the step taken so far] will provide the maximum possible protection for women here.

The decision protects against the production of some of these images, but I do not see them taking responsibility when it comes to preventing re-circulation or re-uploading, even on other platforms — and that, for me, is 100 percent their responsibility, because the AI bot belongs to them.