CHAT CONTROL II. - What the EU’s Chat Control 2 Proposal Would Actually Require

A system cannot know whether a message contains illegal material until it has examined the message.

It may compare an image against a database of known material, use AI to estimate whether a previously unseen image depicts abuse, or analyse a conversation for language that might indicate grooming.

Whatever method is chosen, the system must first gain access to the content.

The surveillance therefore comes before the suspicion.

Chat Control 2

Under the Commission’s original Chat Control 2 proposal, providers could be ordered to detect known child sexual abuse material, previously unknown material and possible solicitation of children.

These are three very different technical tasks.

1 - Matching a file against a known digital fingerprint is relatively specific.

2 - Deciding whether a new photograph is illegal is considerably less certain.

3 - Interpreting a conversation to determine whether one person is grooming another requires an automated system to judge language, context, relationships and intent.

There are several places where this inspection could happen.

The simplest is on a company’s servers. A user uploads a message, image or video, and the provider examines it after receiving it. This is already possible on services where the provider can access unencrypted content (and was allowed with, now expired, Chat Control 1).

End-to-end encryption prevents that kind of server-side inspection because the service provider does not possess the key needed to read the conversation. In a properly designed encrypted service, the message is converted into unreadable data on the sender’s device and can only be decrypted by the intended recipient.

That leaves the endpoints.

Application, or the operating system could perform the inspection when a file is selected, a photograph is taken or text is entered, before encrypting and sending it.

The scanning mechanism could also be integrated into keyboards, cloud-storage tools, image libraries or other system components that have access to content before it reaches an encrypted service.

The message would still be encrypted while travelling across the internet, but the system would inspect all the contents of your phones.

The encryption algorithm may remain intact (Signal, Telegram). The privacy of your data does not.

The European Parliament recognised this problem in its negotiating position and excluded end-to-end encrypted communications from detection orders.

The European data-protection authorities have also warned that the Commission proposal could lead to general and indiscriminate monitoring of everybody everywhere.

This creates a direct tension with the EU’s own ePrivacy Directive and GDPR principles, which were designed to prevent exactly this kind of general and indiscriminate monitoring of communications.

Child Porn and Grooming - intended purposes

The difficulty becomes even greater when the system is expected to detect material that is not already known.

Known illegal images can be identified using digital fingerprints, although even these systems can make mistakes or be manipulated.

Detecting previously unknown material requires AI to decide what an image depicts. Detecting grooming is more complicated still because conversations do not arrive with criminal intent conveniently marked in the metadata.

Adults and children discuss sex education, medical problems, abuse, relationships and personal experiences for legitimate reasons. Teenagers speak to one another in ways that an automated system may misunderstand. Parents send photographs of their children. Doctors, therapists, teachers, lawyers and support organisations handle extremely sensitive material as part of their work.

An automated system must somehow distinguish those situations from criminal conduct.

The European Data Protection Board and European Data Protection Supervisor warned that technologies intended to detect new material and grooming are likely to produce errors. They also noted that analysing all language and behaviour can be particularly intrusive.

False positives are not a minor inconvenience in such a system.

A mistaken Chat Control report may send private photographs, intimate conversations, account details and personal information to a company, an EU body or law-enforcement authorities for further investigation.

The consequences could include an account being suspended, private material being viewed by human moderators, police questioning, devices being seized or innocent people becoming associated with serious crimes.

Even when an investigation eventually clears the person, the intrusion has already happened.

Unintended purposes - worst case scenario

The scanning infrastructure would not necessarily need to remain limited to its original purpose.

Once applications or operating systems contain a mechanism examining private communications and comparing them against centrally supplied rules, changing what the system searches for is technically easy and so, becomes a political decision.

Those categories could later include terrorism, extremism, prohibited political organisations, copyright infringement, disinformation, hate speech, threats to public order or material deemed harmful.

The same applies to voice communication.

While Chat Control is currently debated mainly in relation to messages, images, videos, and grooming conversations, there is no technical boundary between analysing written words and analysing speech.

Smart TVs, voice assistants (Siri, Alexa, Google Assistant), and call transcription services already perform similar real-time analysis for “quality,” “safety,” or ad targeting purposes.

The classic example - casually mentioning socks in front of your TV or while scrolling Facebook, only to see sock ads minutes later - demonstrates that versions of this listening and content analysis are already widespread in consumer devices and platforms.

Lowering the barrier for government-mandated expansion is therefore straightforward from a technical standpoint.

The device would continuously inspect private material, make probabilistic judgements about its meaning and decide whether the user’s communications should be reported.

The inspection could happen without the user knowing what triggered it, which database was consulted, how accurate the system was or whether the material was retained.

Such a mechanism would also create a valuable target for criminals, hostile governments and intelligence services. A weakness deliberately introduced for authorised scanning does not magically become invisible to everyone else and the European Officials don't have a history of their data being secure.

Chat Control promises to find criminals by examining everyone.

Its worst-case outcome is a system that watches everyone while the criminals learn how to avoid it.