The following are the outputs of the captioning taken during an IGF intervention. Although it is largely accurate, in some cases it may be incomplete or inaccurate due to inaudible passages or transcription errors. It is posted as an aid, but should not be treated as an authoritative record.
***
>> Hi, everyone, we will start the encryption and AI session in just a minute. It's the session with the half good and half bad pun, depending on how you look at it. The "Perils of Forcing Encryption to Say, aiai Captain."
Could I request the organizers to help us see the speakers on the screen? The two speakers joining online?
Hi, just to check, Riana and Sarah, can you hear us?
>> RANIA PFEFFERKORN: Hi, hear me?
>> NAMRATA MAHESHWARI: Yes, we can.
>> SARAH MYERS WEST: Can you hear me?
>> NAMRATA MAHESHWARI: Yes, we. Thank you.
All right.
So my name is Namrata, MaheshwariW. Access Now. I lead our global work on encryption and our work on South Asia. I have the relatively easy task of moderating this really great panel. I'm excited about it and I hope we're able to make it as interactive as possible which is why this is a roundtable. We will open it up, hopefully halfway through, but definitely for the last to minutes. If you have any questions, please do note them down.
Quick introduction. I will start with Elisha capacity building Pirkova. She's senior policy analyst and global expression lead and as a member of the European team, she leads our work on freedom of expression and content governance and platform accountability. Thank you so much for being here.
I will introduce above, while we wait him. He's the head of product policy at Mozilla, where focused on A I will and connectivity. He was previously at the public policy team at Google and with Carnegie endowment. And online we have Riana Pfefferkorn. She's with the Stanford Internet observatory. Sarah Myers West is the managing director after AI Now Institute and recently served a a term as a senior advisor on AI at the Federal Trade Commission. She holds a decade of experience in the field of the political economy of technology and her forthcoming book, "Tracing Code" examines the issues of commercial surveillance.
Thank you so much. These are people who have played a very important role in shaping the discourse around encryption and AI in recent times. Thank you so much for lending your insights and expertise and thank you all for sharing your time with us here today.
Well, we're seeing a lot of proposals across the world, in different regions on AI and encryption. So this session really is an effort to shed some light on the intersections between the two, which we think lie within the content scanning proposals that we're seeing in different countries, US, UK, EU, India and Australia, a lot of others.
These proposals mainly subject scanning content of messages on encrypted platforms proponents say there's a way to do this. And opponents say there's an overreliance on AI because the tools that would be needed to scan this content are AI tools, automated scanning tools which are prone to biases and prone to, well, false outputs, and also that it would undermind privacy and erode end‑to‑end encryption as we know it.
Just some house keeping. Online we have my colleague Reetz moderating. If you have any questions, drop them in chat and Reetz will make sure that we address Thome. Riana, proposals to moderate content on encrypted platforms are premised on the idea that it's possible to do this without undermining privacy. Could you tell us a little bit more about what the merits of this are, what the real impact is on encryption, and on the user groups that use these platforms, including the groups that these proposals seek to protect?
>> RANIA PFEFFERKORN: Sure. So there's a saying in English which is that you want to have your cake and eat it too, and that's what this idea boils down to, the idea that you can scan encrypted content to look for bad stuff, but without breaking or weakening end‑to‑end encryption or undermining the privacy and security guaranteed intended for the user. We just don't know how to do that yet. It's not for lack of trying. Community security researchers have been working on this problem but they haven't found a way to do this. The tools don't exist yet and it's doubtful that they will at least in a reasonable time frame.
You can't roll it back and look for one specific type of material. Such as child sex abuse, which they want them for. If you are trying to scan for one type of content, you have to undermine the encryption for all the content. And that defeats the purposing of using end‑to‑end encryption so that nobody but the sender or the intended recipient can make sense of the message.
The United Kingdom government has been pretending that advertise possible to square this particular circle, basically the UK has been one of the biggest enemies of encryption among democracies. It's trying to safely scan encrypted through government‑sponsored tech challenges and Fassed a law the Online Safety Bill that engages in this same magical thinking. There is not any known way to scan encrypted content without undermining privacy and security and nevertheless, this new law in the UK gives their regulator for the Internet and telecommunications the power to serve compulsory notices on encrypted app companies forcing them to try to do just that.
The regulator has now said, actually, okay, we won't use this power because they basically admitted there isn't a way to do it. We won't use that power until it becomes technically feasible to do so, which might effectively be a while, because we don't have a way to make this technically feasible.
And part of the danger of having this power in the law is that it's premised upon the need to scan for child sex abuse material, but there isn't really any reason that you couldn't expand that to whatever other type of prohibitive content the government might want to be able to find on a service which might be anything that's critical of the government. It might be content coming from a religious minority, et cetera. And so requiring companies to scan by undermining their own encryption for their content the government says they have to look for could put journalists at risk, dissidents, human rights workers, anybody who desperately needs their communications to stay confidential and impervious to outside lookers who might be your own government and somebody else who has it in for you, even in cases of domestic violence, for example, or, you know, child abuse situations within the home.
So we've seen some, at least positive moves in this area, in terms of a lot of publish pushback and outcry over this. Several of the major makers of encrypted apps including Signal, what's app which is owned by Meta and Imessage owned by Apple have threatened to walk away from the UK market entirely rather than having them tell them that they scan for child sex abuse.
They are saying we would rather just walk away rather than undermine what our users is come to expect from us which is the level of privacy and security that end‑to‑end encryption can guarantee.
>> NAMIBIA: Thank you, Riana. Sarah, if you could zoom out little bit. There's talking about flawed policies based on airiest representation of the kind of capabilities that the technology has. Do you think there is a better term for it, and if so, well, what would it be?
And the second limit of the question was, again, there have been a lot of studies and debates around the inherent biases and flaws of AI systems so if they were to be implemented within encrypted environments which one of these characteristics or if that's true, if that's something that would happen, would these be transferred to encrypted platforms in a way that would lead to, well, unique consequences?
>> SARAH MYERS WEST: Sure. That's a great question. I think it is worth taking a step back and really pinning down what it is that we mean by artificial intelligence, because that's a term that is meant many different things over an almost 70‑year history and one that's been particularly value laden in recent policy conversations.
You know, if the current ‑‑ in the current state of affairs ‑‑ (No audio).
>> NAMRATA MAHESHWARI: Maybe we can come back to Sarah when she's online.
>> SARAH MYERS WEST: Sorry about that. What I was about to say was, you know, what we sort of mean by artificial intelligence and the present day moment is the application of statistical methods to very large data sets, data set that are often produced through commercial surveillance or through massive amounts of web scraping, and sort of mining for patterns within that massive amount of data.
So it's essentially, you know, a foundationally computational process. But really what Riana was talking about here is sort of surveillance by another means. And I think a lot of value ‑‑ you know, ideals get imbued on to what AI is capable of which don't necessarily bear out in practice, you know the FTC has recently described artificial intelligence as, you know, largely a marketing term.
And there's a frequent tendency in the field to see claims about AI being able to, you know, serve certain purposes that lack any underlying validdization or testing. Within the field, you know, benchmarking standards may vary widely and, very often companies are able to make claims about the capabilities of systems that don't end out bearing out in practice and we sort of discover them through auditing and other methods after the fact.
And to that point, you know, given that AI is essentially grounded in pattern matching, there is a ‑‑ you know, very well‑documented phenomenon in which artificial intelligence is going to mimic patterns of societal inequality and amplifying them at scale. We see wide‑spread patterns of, you know, discrimination within artificial intelligence systems in which, you know, the harms accrue to populations that have historically been discriminated against and the benefits accrue to those who have experienced privilege. And these and AI is claiming to be a robust magical system, but it's not as robust as claimed.
>> NAMRATA MAHESHWARI: Thank you, Eliska given the recent paper you led on content governance in times of crisis, could you tell us a lit about the impact of introducing AI tools or content scanning in encrypted environments in regions that are going through crisis?
>> Eliska Pirkova: Yes, maybe I would like to start from start from the content governance perspective and what we mean by the danger when it comes to client side scanning and weakening encryption, which is the main precursor for security on the Internet. And it becomes more relevant when we speak regions impacted by crisis.
But unfortunately, these technologies are spreading also in democracy, across the world and legislators and regulators increasingly sell the idea that they will provide these magical solutions to ongoing serious crimes, such as online sexual abuse. There's other types of terrorist content and misinformation, and disinformation that is spreading online, on encrypted spaces such as WhatsApp or other private messaging apps.
So of course, there are a number of questions that must be raised when we discuss content moderation, and content moderation has several phases.
It startswith the detection of the content, of evaluation, assessment of the content, and then consequently, ideally, there should be also provided some effective access to remedy once there is the outcome of this process. When we speak about end‑to‑end encryption violation and client side scanning, the most worrisome state is precisely the detection, where these technologies are being used.
And one very important ‑‑ and this is usually done through different ‑‑ using hash technologies, different types of these technologies photo DNA is quite known, and, of course, these technologies I ‑‑ and I very much like what Sarah mentioned, it's quite questionable whether we can even label them as artificial intelligence. I would rather go for machine learning systems in that regard.
What is very essential to recognize here, they scan the content and it's usually used for identifying the content that was already previously identified as illegal content, depending on the category they are supposed to identify, and so then they trace either identical or similar enough content to that one that was already captured.
And the machine learning system is such cannot particularly distinguish whether this is harmful content whether this is illegal content or whether this is a piece of disinformation because this content doesn't have any technical features per se that would precisely provide this sort of information, which ultimately results into a number of false positive and false negatives and errors of these technologies that impose serious consequences and fundamental rights protection.
So what I find particularly worrisome in this debate, increasingly, and that's also very much relevant regarding the ‑‑ the regions impacted by crisis, the impact on significant risk and justifications that these type of technologies can be deployed if there is a significant risk to safety, or to other, you know, significant risks that are usually very vaguely defined in these legislative proposals that are popping across the world. And if they have these ‑‑ if we have these risk‑driven trend, then what is decreased is precisely the requirements for the view of law and the accountability such as that, for instance, these detection orders should be fully packed up by the independent judicial bodies and they should be the ones who should actually decide whether something like that is necessary and conduct an issue assessment.
And when we finally put it in the context of crisis being, of course in terms when the rule of law is weakened, like an authoritarian rule of law where they crack down on human rights and Access Now, being a global organisation, we see this all over again that this is a primary goal of these types of regulations are also these regimes being inspired by the democratic western world where they are probably profiling more and more, and then consequences can be fatal. The number of extremely sensitive information can be obtained about human rights activists as a part of the broader surveillance campaign, and it means under such context where the State is failing and the State is the main perpetrator of violence in times of crisis, it's the digital platform and the private companies that act as a last resort to protection and any sort of remedy.
Under those circumstances not only do the importance increase, but so do their obligations and their responsibility to actually get it right and that, of course, contains the due diligence obligations so understanding what kind of environment they operate in and what is technically feasible and what is actually consequence if they, for instance, comply with the pressure and wishes of the government in power, which with often see when it comes to informal cooperation between the government and the platform.
That was a lot. So I will stop here. Thank you.
>> NAMRATA MAHESHWARI: Thank you. Our fourth speaker, Dr. Tiwari is having some problem with his badge at the entrance. I don't know if the organizers can help that. He's having trouble getting a copy. So just a request if you are able to help, no worries if not, he will be here shortly, but in the meantime, we can keep the session going.
Riana, I would like to come back to you. A lot of conversations and debates on this subject, revolve around what are the alternatives? There are challenges in terms of online safety, harmful material online and very real concerns around privacy and security. If not content scanning, then what?
In that context, could you tell us more about your research on content oblivious trust and safety techniques and whether you think there are any existing or potential privacy preserving alternatives?
>> RANIA PFEFFERKORN: Sure, I published research in Stanford's online journal of "Trust and Safety" in early 2022. There's a categorization that I did in this research which is content dependent versus content oblivious for detecting harmful content online. Content dependent means the technique is ‑‑ requires at‑will access by the platform to the contents of the user data. So some examples would be I made scanning for the DNA, for example, or human moderators who go to look for content that violates the platform's policies against abusive uses.
I would also include content ‑‑ client side scanning as Eliska was describing as a content dependent technique because it's looking at the content of messages before it gets encrypted and transmitted to the recipient.
Content oblivious, by contrast, means that the trust and safety technique doesn't need at‑will access to message contents or file contents in order to work. So, examples would be analyzing data about a message rather than the contents of a message, so metadata and how is this user behaving, even if you can't see the content of their messages.
Another example would be user reporting of abusive content, because they're the reason that the platform gets access to the content of something is not because they had the ability to go and look for it, but the user chose to report it to the platform itself.
So I conducted a survey in 2021 of online providers, which have non‑E‑to‑E type and I asked them what type of trust and safety, across 12 different from child sex abuse and Spam, and so on, and skied them which of three techniques, content dependent and metadata analysis and user reporting which is content oblivious did they find most harmful and what I found for almost every cat Gore, a content oblivious technique. Specifically user reports in particular, prevailed across many categories of abuse I asked about. The only exception was child sex abuse material where automated scanning was deemed to be the most useful, things like DNA.
Necessity should investing in making robust user knows ideally ones that expose as little information about the conversation, apart from the abusive. I find user complaint is the most useful. Plus once you have a database of user reports you could apply machine learning techniques to user or groups across your service if you want to look for some trends without necessarily searching across the entire database of all content on the platform.
Another option metadata analysis in my survey that doesn't fare as well as user reporting in terms usefulness as perceived by the providers but that was the a couple of years as, the use of AI and ML were being used. I want to mention, it's important to recognize that there are tradeoffs to any of the proposals that we might come up with, metadata analysis has major privacy tradeoffs compared to user reporting because the service has to collect and analyze enough data about its users to be able to do that kind approach. There are some services like signal that choose to collect minimal information about their users as part of their commitment to user privacy.
So when we talk about tradeoffs, tradeoffs might be inaccuracy, there might be false positive rates or false negative rates associated with a particular option, privacy intrusiveness, what have you. There's no abuse mechanism that is all up side and all down side. We can't let governors or vendors pretend otherwise and especially when it comes to pretending that you are going to have all of the upside without any kind of tradeoffs whatsoever. Which is what I see commonly used, like, oh, yeah, it's worth these privacy tradeoffs or security tradeoffs because we will realize this upside. Well, that's not necessarily guaranteed.
At the same time, I think as advocates for civil liberties for human rights for strong encryption, it's important for us not to pretend that the things we advocate as alternatives don't also have their own tradeoffs. There's a great report for that CDT looked at a different approaches calls from the "Outside Looking In" that's also a great resource to look at the different sorts of options in the end‑to‑end encrypted versus ‑‑ the tension between doing trust and safety and how to continue respecting strong encryption.
>> NAMRATA MAHESHWARI: A lot of proposals on content scanning is the well‑intentioned goal to eliminate harmful material online. From a product development perspective, do you think it is possible to develop tools that are limited to scanning certain types of content? And looking at the cross border implications as well as electric a platform that services in various regions what do you think the impact of implementing this in one region, versus at region that has different aspects.
>> UDBHAV TIWARI: Thank you. I think whether it's technically feasible or not, and the second is whether it's feasible in law and policy. I think both of them are two different answers. Purely on the technical feasibility perspective, it depends on how one decides to define client side scanning and what constitutes client side scanning or not, but there are different ways in which platforms already do certain kinds of scanning for unencrypted content, that some of them claim can be done for tempted content for a way ‑‑ interpreted content for a way that's reliable.
It's difficult to take any of those claims on face value. Almost none of these systems when they do claim to depict a piece of content have ongone the level of independent testing and rigorous analysis for those claims ting actively be verified, by the rest of either the security community or the community that generally works in trust and safety, like Riana was talking about.
And the second aspect, which is the law and policy aspect is the more worrying concern, because it's very difficult to imagine a world this which we deploy these technologies for a particular kind of content, presuming in meets the really high bar of being reliable and trustworthy, and also somehow privacy preserving. The legal challenges that it creates don't end with it existing, but of how other governments may be inspired by these technological capabilities exist in the first place. Once these exist, various governments would want to utilize it for whatever content they deem worth detecting at a given point in time.
That means that what may be CSAM material in one country and may be CSAM and terrorist content and maybe it's critical, say, of the political ruling class in that particular country as well. And as ‑‑ I think if there's one thing that we have seen with the way that the Internet has developed over the last 20 to 25 years, it's like the ability of companies and especially large companies to be able to resist requests or directives from governments has only reduced over time.
The incentives against them standing up against governments are, like, very, very aligned towards them just complying with governments because it's much easier for from you a business perspective to be able to ‑‑ if a government places pressure on you over an extended period of time to give into certain requests and we have seen examples of that happen with other services that are ostensibly parts of which are end‑to‑end encrypted such as iCloud where this certain jurisdictions they have separate technical infrastructures set up because of requests from governments as well. So if it has started happening, there it's difficult to see a world in which we won't see it for client side scanning and these perspectives.
And Mozilla has had some experience with this and the challenges that come with deploying end‑to‑end encrypted services and deciding to do them in a privacy‑preserving manner but not having ‑‑ and not collecting metadata which was this service called Firefox send which Firefox and Mozilla had created a couple of years ago to allow users to share files easily and anonymously. You went to a portal and it had a low limit, and you got a link and then the individual to click on the link and then you could download it.
And this ‑‑ the service worked reasonably well for a couple of years but what we realized, I think towards the end of its lifespan, was that there were also some use cases if which it was being used by malicious actors to actively deploy harmful content. In some cases malware and in some cases, like, materials that otherwise would be implicated in investigations and once we evaluated whether we could deploy mechanisms that would scan for content on devices which in our case was the browser which has a less of a possibility of doing such actions, we decided that it was better for that piece of software to not exist rather than for it to create the risks it did are to the trust and safety options because it was end‑to‑end encrypted.
That was a nod to the fact that there are different streams and levels of use cases to which end‑to‑end encryption and trust and safety measures that could being for different threat vectors if you would like to call them that way, and the ones that we're specifically talking about which is client‑side scanning, most ‑‑ is most popular right now for messaging, but the way that it's actually been deployed in the past or almost been deployed in the past by a company like Apple, was scanning all information that would be present on a device before it would be backed up.
So there is ‑‑ and that's the final point that I'm making that there's this implication that we're presuming it's a technology that would won't scan your technology when you are hitting the send button, but it's been deployed in a way where it proactively would scan individuals' devices to detect content before it ‑‑ before it is backed up or uploaded on to a server in some form. And that's a very, very thin line to walk between doing that and just keeping it ‑‑ and just like scanning content all the time in order to detect whether there's something that shouldn't exist on the device and that's a very scary possibility.
>> NAMRATA MAHESHWARI: Thank you, Udbhav. Is Sarah online? I think she's had an emergency. We will come back to her if she's able to join again. Eliska, in many ways, the EU has been at the center of this discourse or what is known as the Brussels effect. We see a lot of policy proposals, debates and discusses on Internet Governance and privacy and Free Expression, traveling from the E U. to other parts the world, also true ‑‑ it also happens horizontally across other countries but still in a disproportionate way from EU to elsewhere. And there's been proposals around looking at the question of content moderation on encrypted platforms. What would you say are the signals from the EU for the rest of the world from privacy perspective on what to do own what not to do?
>> Eliska Pirkova: The EU has been and in the route of platform governance and platform accountability. Specifically in the context of client side scanning, I'm sure many of you are aware of the still pending proposal on child sexual abuse material. It's the EU regulation which from the fundamental rights perspective is extremely problematic. As part of network, there was a paper that contains the main critical points around the legislation and a couple of them I summarized during my first intervention.
It's due to the disproportionate measure, it deploys from the detechion order and other measures that can only be implemented through technical solutions such as client side scanning. Very short sighted justification for the use of this technology, very much based on the risk approach I explained at the beginning and not recognizing and acknowledging that the use of such technology will violate the prohibition of general monitoring because it will have to scan the content indiscriminantly.
And I'm mentioning the man on the general monitoring because if you ask me about the impact of the EU regulation. It's the digital services act, even though the Digital Services Act, disseminated for public, if we speak about platforms but to some minimum extent we could say there are some minimum basic requirements for private messaging apps too, even though it's not a main scope of the digital services act.
But the DSA has a lot to say in terms of accountability, transference criteria and other due diligence measures that this regulation contains and we are really worried about the interplay between these horizontal legislative framework within the EU and the ongoing still negotiated proposal, the proposed regulation, the child sexual abuse material.
If it would stay in its current form and we are really not there yet, of course there would be a number of issues that would be in direct violation, with the existing Digital Services Act, especially those measures that stand on that intersection between these two regulations.
And, of course this sends a very dangerous signal to other governments outside the European Union, governments that will definitely abuse these kind of tools especially if a democratic government within the EU will legitimize the use of such a technology, which would ultimately happen and we hope it won't, and there is a significant effort to prevent these regulations from ideally not being adopted at all which is probably at this stage, way too late. But at least, do as much damage control as possible.
So we have to see how this goes but, of course, the emphasis on the regulation, within the European Union around digital platforms in general is very strong. There were a number of other laws adopted in recent years, and it will definitely trigger this Brussels effect that we saw in the case of the GDPR, but also in case of other Member States within the EU, especially in the context of content governance and in Germany where they are reporting a report every year where they clearly show how many different jurisdictions around the world follow this regulatory approach.
It's coming directly from the European Union, the situation will only get, you know, as much as I believe in some of those laws and regulations, what they try to achieve, everything in the realm of content governance and freedom of expression can be significantly abused if it ends up in the wrong hands that doesn't take the constitutional values and rule of law seriously.
>> NAMRATA MAHESHWARI: Thank you. My question for you is the flip side of my question for Eliska. Given that so much of this debate is still dominated by regions in the Global North, mostly the US, UK and EU, how can we ensure that the global majority plays an active role in shaping the policies and the context that are taken into account when these policies are framed?
And what do you think tech platforms can do better in that regard?
>> UDBHAV TIWARI: Thank you, Namrata. I think that generally speaking if we were to look at ‑‑ just for the first maybe minute, in the context of end‑to‑end encrypted messaging, I would say that probably the only country that already has a law on the books that nobody ‑‑ the government doesn't seem to make a direct connection between possibles of client side scanning and regulatory outcomes is the India government.
India currently has a law in place that gives the power to deman the traceability of content, in a manner that interferes with privacy and security. I don't think it's too much of a stretch for the government stakeholders in India to say, why don't we develop a system where there's a model running on every device or a system that scans for certain messages and then the government provides hashes and you need to scan a message before it gets encrypted and reported to us, and if it is a match, it means that that individual is spreading, you know, messages that are either leading to like publish shoes or misinformation that they want to clamp down on.
The reason I raise that even though traceability is not necessarily a client‑side scanning issue. I think the conversation is a lot less nascent but it has more potential to cause much more harm and that always because a lot of these proposals both float under the radar and don't get as much attention internationally and ultimately the only thing that holds or protects the individuals from these jurisdiction is the willingness of platforms to comply with those regulations or not.
Because so far, we ‑‑ apart from the notable exception of China, where in general, there have been systems where the amount of control that the state has had on the Internet has been quite different for long enough that there are alternative systems to the point at which, I think that the only known system that I've read of that actually has this capability is the Green Dam Filter which has does have the ‑‑ it was originally install ‑‑ I think it's almost mandatory for it to be presented on personal computers, which was originally a filter for pornographic website and adult content.
There's reports that it may have been reported to governments when people have searched for certain keywords or looked for content that may not necessarily be approved at that point as well. I think that show cases that some some places the idea that client‑side scanning may not be this hyponetticcal reality that will exist in the future. It may already exist in for some time. And given the fact that we are only rely for better or worse on the will of the platforms to resist such requests before they end up being deployed. I think the conversation we need to start having. How are these people actually holding platforms to account for when these measures to get passed. If you don't intend to comply with it, what is your plan for if the government escalates like, its enforcement actions against you.
As we have seen with many countries in the past, they can get pretty severe. This is something that will need to be dealt with at a country‑to‑country level, not necessarily platform to country level, because I think that ultimately, if ‑‑ depending on value of the mark for the business, or for the strength of the market as a geopolitical power, the ability of a platform to resist demands from a government, is ultimately limited and they can try and some of them do and many of them don't but ultimately, it's something that only international attention and international pressure could move the needle.
The final point I will make there, I do think even when it comes to the development of these technologies, these are still very much very like western‑centric technologies where a lot of the models they are trained in, a lot of information these things are designed on come from a very different, like, realm of information that may not really match up to pieces in the global majority.
I have numerous examples outside of the end‑to‑end encrypted context. Something that a lot of platforms do, they block certain keywords that are known to be secret keywords for CSAM. Which are not very well known. And they ‑‑ they vary radically in different jurisdictions. So in order to may seem like an innocuous word that means something completely different in a local language but if you search for that, you will find users and profiles where CSAM exists.
And just fining out what those keywords are in various local languages in individual jurisdictions is something that many platforms take years to do, be able to do well, and that's not even an end‑to‑end encrypted or a client side scanning problem. It's how much are you investing in understanding local context? How much are you investing in understanding local realities problem?
And if that happens there, I think that, like, it's because those measures fail, it's because when it comes to unencrypted content, the platforms don't act quick enough or don't account for local context enough. That governments also end up resorting to recommendations like client side scanning.
It's by no means the fault of the platforms that these ideas exist, there's more they could do in the global majority to deal with the problem on open systems where they actually have a much better record of enforcement in English and in countries outside the global majority than within the global majority.
>> NAMRATA MAHESHWARI: Thank you. I have one last question for Sarah, and then we'll open it up to everybody here and if anybody is attending online.
So feel free to jump in after that.
Sarah, as our AI expert on the panel, what would your response be to government proposals that treat AI as a sort of silver bullet that will solve problems of content moderation on encrypted platforms?
>> SARAH MYERS WEST: So I think one thing that's become particularly clear over the years is that content moderation is in many respects an almost intractable problem, and though AI may present as though a very attractive solution, it's in many ways, it's not a straightforward one, and, in fact, it's one that introduces new and likewise troubling problems.
A I., for, you know, all of its many benefits, it remains imperfect, and there's a need for considerably more scrutiny on claims that are being made by vendors, particularly given the current state of affairs where, you know, quite few models are going through any sort of very rigorous, independent verification or adversarial testing. I think there's concerns about harms to privacy. There are concerns about false positives that could sort of paint innocent as culprits lead to unjust consequences. And lastly, you know, there's been research that has shown that malicious actors can manipulate content in order to bypass these automated systems and this is an issue that's endemic across AI.
And underscoring, you know, even further the need for much more rigorous standards for independent evaluation and testing. So before, you know, we put all of our eggs in one basket, so to speak, I think it's really important to, one, evaluate whether AI broadly speaking is up for the task; and then, two, to really look under the hood and get a much better picture of what kinds of evaluation and testing are needed to, you know, verify that, in fact, these AI systems are working as intended because by and large, the evidence is indicating that they are very much not.
>> NAMRATA MAHESHWARI: Thank you, Sarah and thank you all so much on the panel. I will open it up to all the participants, because I'm sure you have great insights and questions to share as well.
Do we have anybody who wants to go first?
Great. Sure. Could we ‑‑ before you make your intervention, could you maybe just share who you are?
>> AUDIENCE MEMBER: Is it ‑‑ okay. It's better now.
Good morning, everyone or ‑‑ yeah, still good morning.
My name is Katarina and I represent the National Research Institute in Poland although my background is in law enforcement and criminology and also clinical sexology. So I really want the voices of children to be present in this debate, because there were already mentioned in the context of CSAM which is child sexual abuse material, scanned and ‑‑ yeah, on some other occasions. But I think there is a need to make a difference between general monitoring or general scanning, and scanning for these particular type of content.
It is such a big difference because it helps to reduce this horrendous crime, and there are already techniques that can be reliable, like hashes, and by hashes, I also mean experience of hotlines, hotlines present all over the world, and it's already experience of, I believe, more than 20 years of this sort of cooperation.
So hashes, there are gathers in a reliable way. There is verification, in the process of stating in a particular photo or video is CSAM, is not like a general scanning. It's scanning for something. What has been corroborated before by an expert.
And then on AI, I'm lucky enough because my institute is ‑‑ is actually working on AI project. And we train our algorithms to detect CSAM in a big bunch of photos and videos. I can tell you that this has been very successful so far. So we use also current project by In Hope for a specific anthology: And we train to pick up only these materials that are clearly defined in advance.
So ‑‑ and it's, again, it's an experience of years of cooperation, international cooperation and I can tell you that general monitoring is something very different than scanning for photo or video of 6‑month‑old baby that's being raped.
So please take it into consideration while we have take ‑‑ to take care of privacy and online safety, we first have an obligation to protect children from being harmed and this is also deemly rooted in all the EU conventions and U.N. Conventions and the EU law. So we have to make a ‑‑ we have to make a decision, because for some of these children, it will be too late. And I will leave with you this dilemma. Thank you.
>> NAMRATA MAHESHWARI: Thank you will. Thank you so much for that intervention and respect all the work you are doing. Thank you for sharing that experience.
I think one thing that I can say for everybody on the panel and in the room is that all of us are working towards online safety, and I know we're at a point where we're identifying similar issues, but looking at the solution from different lenses. So I do hope that conversations like this lead us to solutions that work for safety and privacy for everybody, including children. So thank you so much for sharing that. I really value it.
Anybody else?
Over there. And there.
>> AUDIENCE MEMBER: Thank you for the great presentation. I'm with the European Center for Non‑Profit Law. I would love to hear from Eliska, you mentioned the potential misuse of E U. laws and how do we fight human trafficking which are all laudable goals which as human rights a voluntary indicates we all fight for and thank for your mention about child protection.
Indeed, onvine safety applies to all, especially marginalized groups but I would love to hear from you how it's not as easy. It's not a black or white kind of picture, and how these narratives can often be abused and weaponized to actually prevent encryption.
>> ELISKA PIRKOVA: Thank you very much for your contribution. From a position of digital rights org, we, of course advocate for the online safety and protection of fundamental rights of all. And, of course, children also have the right to be safe and to ‑‑ they also have equally the right to privacy. We can go into nitty‑gritty details on general monitoring whether, you know, how these technologies work and whether there is any way how general monitoring would not occur. And I think maybe we would even disagree to some extent, but the point is that the goal is definitely the same for all of us.
And especially when it comes to marginalized groups, it's a major priority for us too. I definitely kind it difficult, mainly as an observer because we truly rely on the Adrian network. And I often see that precisely the question of children's rights is being to some extent ‑‑ I would say ‑‑ I'm trying to find the right term. But the emphasis on that, even though it's a number one priority for all of us, it can be used in the debate to maybe counter argue against opinions, that are slightly more critical towards some technical solutions while no one ever disputes the need to protect children and that they come first.
And that often complicates and maybe becomes to some extent almost counterproductive because I don't think that we have any differences in terms of goals that we are trying to achieve. We all are aiming at the same outcome and the process. But perhaps the means and the ways and the policy solutions and regulatory solutions that we are aiming at might differ and that's of course, subject to debate and to ongoing negotiations, what is that solution and none of them will be ever perfect, and there will have to be some compromises made in that regard.
But I do find this, you know, dichotomy and more kind of like, very straightforward, black and white differences in terms of when we are doing advocacy, almost occasionally trying to put it in a way that we should choose this side that's incredibly problematic, because there's no need for that. I think we all, as I have said have the same outcome in mind.
So I don't know I answered your question, but, indeed, this is a very complex and complicated project and we need to have this dialogue as we have today and inform each other's position and see each other's perspective to achieve that successful outcome that we are trying for and that's the highest level of protection.
>> NAMRATA MAHESHWARI: Thank you.
Bogisha.
>> AUDIENCE MEMBER: Thank you S. this working? Hi, I'm Bogisha, I'm a Ph.D. scholar at Georgia Tech. Could I ask you to condense all of that material into 40 minutes, I mean it's a vast thing that you have covered here. I think I have a comment that will lead to a question. So I will be quick.
Eliska, you mentioned about significant risk at the beginning. It's often masked by the concerns of national security when it comes to the governments, the governments and the national security risk could be often subjective and could be related to the context of which government and how they interact with it.
So ‑‑ and I think we have also mentioned about how harmful content is a big problem in this space. I mean, all of us agree about that. I think my question would be largely eluding to one, when you were talking about ‑‑ and this is to everybody, when you are talking about scraping of the content to be used further on, how much of the apps that are available online, actually score the data in an encrypted format? How big is that problem of scraping that data and encrypted and being in encrypted format.
Two, how do we think about it from a user's perspective? So what can a user do directly to either ‑‑ not solve this problem but intervene in this problem and present perspectives ahead?
>> NAMRATA MAHESHWARI: Actually, do you want Udbhav and could I ask everyone to keep their questions brief so that more people can participate. We have only a few more mint unites to go. Thank you.
>> UDBHAV TIWARI: Sure. So on the platform front, I think how big is the problem and how pervasive is the problem is an interesting one because on one angle it depends on whose perspective you are looking at it from. If you are looking at this from the perspective of an actor that either produces or consumes child sexual abuse materials, then it's orgably a lot of them, because this is how one would argue they communicate with each other and share information with each other, which is measures that aren't online at all or are ways in which they are encrypted.
I think that's definitely a space that needs a lot more study especially ‑‑ even specially going into like, what are the vectors in which these, like, pieces of information are shared and communicated with each other, because there has been some research in how much is online, how much of it is offline and how much of is national discovery and how much you have to seek out the fact that it exists kind of discovery, but a lot of that information is both very jurisdiction‑specific and I think overall has not been answered to the degree that it should be.
On one of the things that users themselves can do, I mean broadly in three categories. One there's reporting itself because even on other systems the ability for a user to say I have received or seen content that is like this and I want to tell the platform that this is happening is one route. And the second route and this applies to more limited systems the content exist Ms. In this form and I would like to directly take it to the police or to law enforcement agencies saying, I have it from this user in this way and this is the problem that it's creating, and ultimately, the third is for the user and this is like, something that a lot ‑‑ there has been a lot of research on is like intervening at the social level, where you talk to your, like ‑‑ if it's somebody you know, for why this is kind of problematic and you get professional psychiatric help. And platforms can or cannot play a role. Some of them can help you to seek help, and courts have mandated that these warnings be protectively surfaced and laws too like in India.
Ultimately, it's an area that needs more study.
>> NAMRATA MAHESHWARI: You will all of this is to by no means platforms keep people safer in a way.
>> Governments don't. By all means we need measures to make platforms more accountable, including the ones end‑to‑end encrypted, absolutely but the question is how to do it that most respects fundamental rights. I will pass it to you and then to the lady.
Online, Rihanna, if there's anything that you want to add to any question, please just raise your hand and we'll make sure you can comment.
>> AUDIENCE MEMBER: My name is rob Horner from the Electronic frontier Finland. I don't have a question but I would like to respond to the law enforcement representative here. I have lost a lot of credibility in law enforcement to use their tools for what they actually say they use them for: For example, in Finland, they tried to introduce a censorship tool. They did in 2008 and in the end, it took a hacker to scan the secret list of the police censorships ‑‑ the censors, the websites and we found out like the rationale for the tool went that it has to be used like that because it's hosted in countries that our law enforcement doesn't have access to or any cooperation. And it was the secret list. He was able to compile about 60% of the list and we scanned the material and had a list. Less than 1% was actually child sexual abuse material and the second point which I think is even stronger is that guess the biggest country that hosted the material.
It was US. After that, Netherlands. After that, UK. In fact, the first ten countries were western countries, where all you need to do is pick up the phone, call them to take it down. That's it. Why do they need a censorship tool?
The same goes for these kind of client side scanning. I feel it's going to be abused. It's going to be used for different purposes after it goes to gambling and so on. So it's a real slippery slope and it's been proven before that that's ‑‑ that's how it goes.
>> AUDIENCE MEMBER: Thank you. Thank you very much for your comment. So I work for ARPAT international, we are at the forefront of advocating for the use of targeted technology to protect children in online environments.
And I absolutely, I mean, what's interesting just even about the people in this room, is I think certainly what we're seeing is an example and we speak for both sides of the conversations being divided.
I'm very happy I'm here and I'm enjoy this conversation because I absolutely believe in critically challenging our own perspectives and views on different issues particularly about Global South and different jurisdictions. I think we have a system that's working. It's not perfect and there are examples where that ‑‑ that have been problems but in general, the sim is working very well and we can ‑‑ the system is working very well and we can give more examples of why that is and we need to build on the existing system to build out. One of the other things that has been a key issue for IGF and it's trust in techle and it's difficult to achieve. They are hard to gain, and it's trust in gender Ale.
I think on this issue, it's at the forefront of the problem.
I think one of the things I always regret is that there wasn't more discussion of why we do agree because there are areas we agree and one thing that comes up when we deal with issues of trust, it's algorithmic transparency, oversight, reporting, they're not perfect but as civil society, we can call for accountability. So I think that those areas are where we do agree and I wish we were speaking a little bit more about that.
In terms of legislation and general monitoring, you are right we will not go about the details of the processes in the EU but I think there's sometimes a convenient conflation of technology in general and specific technologies used for things and I think if we talk about targeted CSAM detection tools and spyware they are not the same thing and I think sometimes there's a conflation of different techs used for different things.
The other thing, which is data sets upon which these tools are trained, it ‑‑ it's true that we need to be doing much better at understanding and having data that will avoid any kind of bias in the identification of children.
But just to this final point, one of the reasons for differentiating between hosting of content and which is very much related to Internet infrastructure, but it is shifting, is also that we need to talk about victim identification. One of the reasons to take down and refer child sexual abuse material is so that it gets to processes where children can be identified and we have decades now of experience of very successful processes whereby law enforcement are actually identifying children and disclosing on their behalf because we have to remember that child sexual abuse material is often the only way a child will disclose because children do not disclose.
And one of the fallacies, I'm sorry I will finish here one of the fallacies about the child rights argument is that often we are calling for technical solutions as a silver bullet. Absolutely not. This is a very complex puzzle and prevention means technology, prevention means education, prevention means safety measures and prevention means working with perpetrators. It's everything that we need to be doing. And we're absolutely calling for that.
So I suppose it's not a question, but I wanted to sort of make that point, and maybe it's a question or a call to action is that we really need to be around the table to go. Because I think there are areas where we absolutely are agreeing.
>> NAMRATA MAHESHWARI: I absolutely agree with that and I do hope we will have more opportunities to kind of talk on the issues that we all care about. Unfortunately, we are over time already, but I no he that Riana has had her hand raised for a bit. Do you want to just close us up in one minute.
>> RANIA PFEFFERKORN: Sure. So to close, I guess I will just note that ‑‑ I will emphasize something that Eliska sold, we know that all fundamental rights are hasn't to be coequal with no one right taking precedent over another and how to implement that in practice is extremely difficult, but it applies to things like child safety, which is are these issues that we can get stuck on and that's the topic of a report that I helped to author including with an emphasis on child rights as part of the DFR Labs recent scaling trust on the web report that goes into more depth that all the different ways that we need to be forward looking, with regard to finding equitable solutions for the various problems of online harms.
I also just want to make sure to mention that when it comes to trustworthiness of institutions, we do need everybody to be holding governments accountable as well. There was recent reporting that Europol had some some of the closed door negotiations over the child sex abuse regulation in the EU demanded unlimited access to all data that would be collected and have that pass on to law enforcement so they could look for evidence of other crimes, not just child safety crimes. We also need everybody, child safety organisations to be holding governments account and ensure that if they are demanning these powers that they cannot be going beyond those and using that as the tip of the spear with one particular topic to be demanding unfettered access. It goes beyond the hallmark of the human rights respecting framework. Thanks.
>> NAMRATA MAHESHWARI: Thank you. A big thank you to all the panelists. Thank you for moderating online and thank you all so very much for being here for sharing your thoughts, and we hope all of us are able to push our boundaries a little bit and arrive at like a common ground that works for the best of all users online. Thank you so much. Have a great IGF.
[ Applause ]