PassW0rd – 9 September 2020

Speaker A: This program is brought to you by Resonance FM. If you like what you hear, please support our work by making a donation at resonancefm.com/donate.

Speaker B: Hello, I’m Peter Warren, and this is Password on Resonance FM. If the teachers at my school had been asked to predict what would become of me, it’s most unlikely they would have thought I’d end up here. I’ve got a degree in English and History and national awards for technology journalism. But I also organised a school strike and a rebellious school council sit-in on the tennis courts. I was told by teachers that I was a ringleading troublemaker and I was being watched. But I went to a respected grammar school, so if an algorithm had been tasked to predict my grades, I probably would have done quite well, if the recent row about the algorithmic awarding of GCSE and A-level grades in England and Scotland during the pandemic is anything to go by. And that’s not fair, is it? Well, not for young people like Joshua Peat. It’s not only unfair but soul-destroying and potentially life-changing. Joshua goes to the Sixth Form Centre at Thomas Mills School in Suffolk. He was assigned his A-level grades by algorithm.

Speaker A: So my algorithm results were ABD with a B in my EPQ, which was really fortunate, and I also had a Grade 8 distinction in singing, so that pushed me over the 128 UCAS point limit. So I therefore received my offer on the first results day. Well, how did you feel getting a grade from an algorithm instead of by sitting your exams in the normal way? That was quite strange because I wasn’t able to prove to myself. That’s the main thing. That was what taken away from me. And I did feel at the time that a D in maths was too low, as I’d been working at between a B and a C throughout the year, so I was very confused as to why I had received a D in maths. Did you talk to any of your teachers about this algorithmic prediction? I did briefly, saying that I would like to appeal my maths result. And my politics result on the basis of them being low. But because I’d received my offer, it wasn’t really the school’s priority because they were inundated with people who hadn’t got through the first time round. Really? Do you know how many, or what kind of percentage? I don’t know the percentage, unfortunately, but our head of sixth form held meetings of between 15 and 20 minutes in length per student, and he got booked for 2 days. So there were lots of people who were severely unhappy and didn’t get in, which was really gutting. Did it happen to any of your friends, anyone you know personally? It did. It happened to 2 students who have now got into Cambridge University, but First time round, they had not. And it happened to someone else who I know very well who didn’t receive a C in maths and got a D in maths, just like me, and his offer was conditional to getting a C. Luckily, most of them, when the teacher predictions came out, got through to their first universities. But there are a few that, because of the delay in time of nearly— I think it was 5 days— the university courses had filled up. So now that there are quite a few people I know who have had to take a gap year, which isn’t ideal because of the pandemic because there’s no traveling, despite receiving grades that would theoretically allow them to go on to their course that they’d applied to. Now, you’re studying maths, and I’m sure maths will play a big part in your architecture degree and your future career in architecture. What do you know about algorithms, and what do you make of the, the way that they’ve been used here? Well, I’m not that into computing, but I do understand some basics, and I can’t quite understand why, if the government didn’t want people to go down 2 grades, why they didn’t just put that— embed it within the algorithm. And I don’t think that seems that difficult. But then, on the other hand, something else I don’t quite understand, and this is on the other side and has happened to much, much fewer people, but I don’t quite understand how grades could be inflated from the teacher-assessed grades. Because teachers were asked, “Be optimistic but realistic, but how can you do any better than on a good day? How— what would you get?” There seems to have been a class size of 5. This seems to be some kind of key number where classes of size 5 or less were left with teacher predictions. So private schools were, in a roundabout way, treated differently to state schools because private schools in general have smaller class sizes than state schools. So that also means that if the private schools are getting A*s and As and they’re filling up their percentage, that means that there are more Bs through to use, to give to state schools, which therefore explains the big decrease in state schools and not private schools. And how does that make you feel, Joshua? I just think that the class of 5 is completely unnecessary. You either fully trust teachers or you don’t. You can’t have a halfway house where if a class has 5 or less students, the teacher is therefore more trustworthy? That’s just ridiculous.

Speaker B: We’ve also been talking to another young man, Charles Mayo from Singapore, who lost his Oxford University place because the algorithm underestimated his grades. Thousands of young people like Josh and Charles were victims of what British Prime Minister Boris Johnson called a mutant algorithm which assessed the probable performance of exam candidates who are unable to sit their national examinations because of the lockdown. When the Prime Minister called it mutant, did he really mean that a piece of software had somehow transformed itself and broken free of its bounds like a rogue robot or a Frankenstein’s monster? It does beg the question, does Mr Johnson, a man his Labour shadow Keir Starmer accuses of never doing his homework, actually know what an algorithm is? I’ve been talking to a number of top experts from all over the world in the making of this this radio program. Even though they all know exactly what an algorithm is, oddly, their answers are all different.

Speaker A: Algorithms are a set of rules that is set to computers to follow, so it’s a computer-implementable instruction. These algorithms are built by people, so the outcomes are of course also shaped by the experiences and beliefs of the people that create these algorithms. It’s not the math that we’re going to hold responsible or accountable. It’s the people who created that math, who implemented that math. You know, you can imagine an algorithm like the A-levels algorithm suggesting certain outcomes, but ultimately those decisions of how it’s used are human decisions. Instagram favors posts that show naked skin. It has to be women in bikinis, they can’t be really naked, and men, they would have bare chests, and of course, they can also not be entirely naked. Then you have posts that show this, and they are more successful than others. When we asked Facebook, Instagram’s parent company, about this and how they can explain that, their answer was, well, you don’t really understand how Instagram and its algorithm works, but we won’t give you any kind of proof that it’s not the way you say it is. So now what I’m going to do, actually I want to give the algorithm name here. So I’m going to select, I click the right button. So I am going to merge all these cells and going to create one single cell. I am going to write Algorithm 1: Bedtime Computation. Okay, I am saying this is number Algorithm 1. So if you are writing one more algorithm, next algorithm you can write algorithm 2, okay?

Speaker B: The algorithm’s grades were so demonstrably unfair, the Ofqual Qualifications Board came under huge pressure and the chief regulator Sally Collier resigned. The remaining staff are so chastened that they refused to be interviewed. Instead sending us a prepared statement read by password Simon Skitch.

Speaker A: We tested 12 different models and looked at overall accuracy, accuracy for different types of centre, and accuracy for centres with different proportions of candidates with protected characteristics. We also extensively tested possible variations of the chosen standardisation model to ensure we selected the model which gave students the fairest results possible and did not systematically disadvantage centers with differing levels of students with particular protected characteristics and socioeconomic backgrounds.

Speaker B: Those words are echoing around the world, leading some researchers to fear that a new AI winter is on its way. That’s the equivalent of a technology ice age when it falls out of favor with computing companies, something that’s happened over 5 times already with AI. Only this time, it might happen because the public refused to trust AI systems. You’re listening to Password on Resonance FM with me, Peter Warren, and after this you can hear DJ Ritu with A World in London, a celebration of our city’s musical diversity. Diversity, of course, is hard to classify, and that’s why some algorithmic decision-making is accused of racial bias. Whether the mathematicians who designed the algorithms meant to discriminate against certain groups in society is not clear. Probably not. But the result of training algorithms on specific datasets that are, for example, predominantly composed of white people, can clearly lead to discrimination and injustice. As with the British A-levels, the automated decision-making in the United States has a big impact. It can mean, for example, that a black person or a person from a certain neighborhood or zip code gets a longer jail sentence, whereas a similar offender from a different part of town is released on bail. Dr. Joy Buolamwini founded the Algorithmic Justice League to challenge these biases. Her research at the Massachusetts Institute of Technology Media Lab revealed that algorithms used in facial recognition consistently failed to correctly identify Black women and people of color generally. So when people talk about algorithmic bias, I’ve been hearing this theme of let’s eradicate bias from AI, right? Well, or our system is bias-free. And I always like, did you get rid of the humans? Like, how are you bias-free? Like, even if the tech is deemed neutral by some metrics, how we use it in and of itself has its own assumptions and biases. So instead of thinking, let’s eradicate bias— and I’m not saying we shouldn’t try to make systems as robust as possible— we need to also come in with the mindset that there will always be some kind of bias, some kind of harm that we need to be checking for. And so instead of thinking about bias eradication, let’s think about bias mitigation.. And in some ways, it’s more like having algorithmic hygiene.

Speaker A: And with hygiene, you wouldn’t just shower once in 2019 and say, we’re done, we checked.

Speaker B: We checked for bias 2019 on our model, we’re good for here on out. No, that would sound silly. So instead, when you’re thinking about something like algorithmic hygiene, it’s ongoing because you realize you have to stay vigilant, right?

Speaker A: The situation changes.

Speaker B: And so when it comes to how do we look at improving these sorts of technologies, we need to look at how the technologies work and also how they’re being used and make sure that there are appropriate safeguards and also mechanisms for transparency and accountability in the first place. Joy Buolamwini believes we should regularly audit algorithms. To prove this, she fed a facial recognition system pictures of famous Black women. Including Michelle Obama, Oprah Winfrey, and Ida B. Wells. The results were bad, to say the least. She wove those results into “AI, Ain’t I a Woman,” a spoken-word artwork based on the 1851 speech by freed slave and women’s rights campaigner Sojourner Truth, who bared her breast at a public meeting, declaring, “I can work as hard as any man.” And ain’t I a woman?

Speaker A: I have as much muscle as any man, and can do as much work as any man. I have plowed and reaped and husked and chopped and mowed, and can any man do more than that? I have heard much about the sexes being equal. I can carry as much as any man, and can eat as much, too, if I can get it. I am as strong as any man that is now. A 19th-century question comes into view in a time when Sojourner Truth asked, ain’t I a woman? Today we pose this question to new powers making bets on artificial intelligence, hope towers.

Speaker B: The Amazonians peek through windows blocking deep blues as faces Increment scars, old burns, new urns collecting data chronicling our past, often forgetting to deal with gender, race, and class.

Speaker A: Again, I ask, ain’t I a woman? Face by face, the answers seem uncertain. Young and old, proud icons are dismissed. Can machines ever see my queens as I view them?

Speaker B: Can machines ever see our grandmothers as we knew them? Ida B.

Speaker A: Wells, data science pioneer, hanging facts, stacking stats on the lynching of humanity, teaching truths hidden in data, each entry and omission a person worthy of respect. Shirley Chisholm, unbought and unbossed, the first Black congresswoman, but not the first to be misunderstood. Understood by machines well-versed in data-driven mistakes.

Speaker B: Michelle Obama, unabashed and unafraid to wear her crown of history, yet her crown seems a mystery to systems unsure of her hair. A wig? A bouffant?

Speaker A: A toupee? Maybe not.

Speaker B: Are there no words for our braids and our locks? The sunny skin and relaxed hair make Oprah the First Lady. Even for her face, well known, some algorithms falter, echoing sentiments that strong women are men. We laugh, celebrating the successes of our sisters with Serena smiles.

Speaker A: No label is worthy of our beauty.

Speaker B: If the traveling preacher Sojourner Truth were alive today, She might well join the Algorithmic Justice League or AlgorithmWatch, the think tank in Berlin where Matthias Spielkamp scrutinizes automated decision-making. He believes, like many other scientific observers, that algorithm decision-making has been shown to be flawed.

Speaker A: There are a lot of algorithmic processes that are already tested, that are already audited, and there’s already regulation in place. For example, high-frequency trading, that is regulated, and the regulation for that even contains the words algorithms, right? So it actually targets that, and it has been for a long time, because there it has been done for a long time. But the other two examples I would like to choose is medicine. There is a lot of regulation there, then there’s a lot of testing. If you are talking about diagnostic systems, it’s heavily regulated, and you can’t just build something, you know, like an MRT or something, you can’t just build it and start selling it. But it has to go through several rounds of testing and very high qualified experts look at it and look at the models that are used there. And the last one is cars, right? I mean, we have cars now that are at least partly automated. And there’s also a very high level of scrutiny. First of all, because there are the regulatory basis for this, law, and secondly, there are also the institutions for that to do this. There is this option that on some very fundamental level there can be a, let’s say, AI conformity testing in the sense that when we are talking about machine learning, there can be procedures to find out how the data set was collected and whether it was done correctly and how then the model was built on this.

Speaker B: Okay. So I mean, basically the regulation does exist and it’s just that in these various different areas, the regulators of these areas will just have to absorb the word algorithm and they’ll have to start looking into the use of that in their particular areas.

Speaker A: If they’re not already doing it. Yes, I agree. And in some cases, we need to come up with stuff like that. You know, for example, with the social media platforms, everyone is really trying hard to come up with ideas of what the standards and criteria should be. You know, take disinformation as an example. Now, the belief is, and I think there’s a lot of plausibility to that, that, you know, platforms like Facebook and YouTube, They amplify polarizing content, right? Now, is that a problem? Well, yes, I do think it’s a problem if they do that, you know, for their business purposes. Should they be prohibited to do this? That’s a different question because there’s no such thing in Western liberal democracies with free speech that says the government needs to control for disinformation. You know, you are allowed to say the world is flat, right?

Speaker B: You are allowed to lie. Yeah, you can’t, you can’t stop people from lying.

Speaker A: Exactly. You are allowed to say that, that corona doesn’t exist. And is that a good thing? Well, it’s not a good thing to say that corona doesn’t exist, but it’s a good thing that you’re allowed to say it, right?

Speaker B: How can we trust system to deliver fair results. Listen to this from the website of the Joint Council for the Welfare of Immigrants: Since 2015, the Home Office algorithm has used a traffic light system to grade every entry visa application to the UK. It assigns a red, amber, or green risk rating to applicants. Once assigned by the algorithm, this rating plays a major role in determining the outcome of the visa application. The algorithm suffered from a feedback loop. Applicants from suspected nationalities were more likely to have their visa applications rejected. As with the A-level algorithm, a sort of self-fulfilling prophecy was built into the criteria. But the Joint Council mounted a legal challenge, as their Director of Legal Policy, Chai Patel, explains.

Speaker A: I mean, I think our issue isn’t necessarily with algorithms, you know, in the abstract, but the use that the Home Office put them to in sorting and prioritizing visa applications was, you know, a textbook example of how not to do it. They took data that was itself tainted by racial bias, and then they put an algorithm on top of that that basically sorted people’s visa applications based on what country they were from. And that introduced direct nationality discrimination into the whole process. And they did it in a very secretive way. You know, we had to take them to court to find out details of what was going on, and they’ve still refused to even tell us which countries they were discriminating against before, you know, agreeing to withdraw the algorithm as it currently is and have a look at it and try again.

Speaker B: Isn’t this always going to be a difficult area for governments, though. For example, if you were discriminating against the Irish, the Irish have more of a record for terrorism than, say, other ethnic groups. Are you not always going to use data that you think will have some relevance when you’re feeding one of these algorithms?

Speaker A: Well, I mean, first of all, like, nationality is, is not an appropriate data point, and it’s unlawful. And the Home Office have accepted that the way that they were doing things— you know, they haven’t accepted it was unlawful, but they’ve withdrawn it based on a legal challenge that we brought on that basis. So I think you have to be very careful about that. On the broader sense of, are they entitled to take into account certain risk factors that might or might not end up disproportionately affecting some groups more than others? You know, sometimes that is lawful and that’s valid. The most important thing that we have to remember when we’re talking about how governments make decisions, particularly decisions, you know, in the immigration system that can change the course of someone’s life, is that the reasons for that decision, the data that goes into it, needs to be transparent and it needs to be accessible to the person against whom that decision was made so that they and their lawyers can properly assess whether the decision was fair they can just decide whether other evidence needs to be brought before the government to change that decision. You can’t do that if you have a black box system that has lots of different kinds of dodgy data going into it, has feedback loops which reinforce racism, as this one did. You had a situation where the way that nationality’s risk was assessed was by how many previous applications had been refused. That was a factor. Now, if you’ve got a system that disproportionately affects some nationalities, and then you use the results of that system to decide which nationalities are high risk in future, that is not a good way of doing things, because that creates a reinforcing loop whereby some nationalities will always be higher and higher risk as time goes on. So what’s important, and what’s essential, is when the government puts in place any kind of decision-making system, It needs to conform to the rule of law, and the rule of law says that decisions have to be made in a way that the person against whom they are made can understand how they were made, what information was taken into account, and can challenge that on that basis.

Speaker B: It seems unfair that a computer can categorise anyone as undesirable or inadequate based on a secret set of rules being applied to some personal data points. Yet the oddest fact in what we think of as cutting-edge technology is that this is not a new concept. In 17th century Geneva, the revolutionary Christian priest John Calvin preached a doctrine called the predestination of souls. Basically, it means that God has already decided before you were born whether you’ll go to heaven or hell when you die.. It categorizes people into the elect, the chosen few, and the rest. Dr. John Balzerak of Bristol University is a Calvin scholar and sees some parallels with algorithmic decision-making and its potential for harm. So Calvin, and for that matter lots of people before Calvin, taught that God not only created every single human being but predetermined where that individual would eventually be, either in heaven or in hell. And it was a predetermination that God made without consideration of the individual person’s qualities.

Speaker A: It’s not as if God looked at you and said, oh, you’re going to be a lovely person, so I will predestine you for heaven.

Speaker B: Quite the contrary. The whole idea is that God made these decisions which are mysterious to us, but he made them without any foreknowledge of what we would be like. So everyone’s eventual endpoint is already predetermined. It cannot be changed.

Speaker A: And that’s—

Speaker B: Calvin thought that was absolutely basic to understanding Christianity.

Speaker A: And yet the The term Calvinist used to describe followers of Calvin has rather different connotations.

Speaker B: Yeah, I mean, you’re absolutely right. I think it’s come about, it’s come about because followers of Calvin just did do horrendously bad things, and it’s very easy to think of obvious examples like apartheid in South Africa, which scholars have shown, I think pretty persuasively, arose out of the Calvinistic doctrine of the covenant, and what I think it’s fair to say Calvin would regard as a horrible misreading of that doctrine.

Speaker A: But nonetheless, it arose out of the doctrine of the covenant, which is just in, in the most simple sense, the idea that God has made a covenant, an agreement with God’s people. That’s all the covenant doctrine really has to say. But that doctrine then ended up being used by some to try to argue that they were— those people were favored, and that those outside of the covenant community were fundamentally different and unworthy of God’s love.

Speaker B: So my point is there’s some truth to the ascription of nastiness to Calvinism, and some, I would say, plainly polemical untruth.

Speaker A: One of the things that Martin Luther tried to do, of course, was to reveal to everyone what had been kept a mysterious secret by the priests and the Pope and the fact that the Bibles and worship were only in Latin. Can you imagine or identify a new Luther today who would reveal what’s in the black box algorithms that predestine us in so many ways?

Speaker B: You’re absolutely right. There’s this sense in which Luther and others, Calvin too, were keen to try to say that, for example, the Bible ought to be available to everybody. And in that regard, I mean, one thinks of in sort of extreme examples, somebody like Julian Assange. I mean, these people who try to expose government secrets, who try to expose wrong, what they regard as wrongdoing by using a computer to reveal the fact that decisions were made that were made out of wrong motives.

Speaker A: Out of greed or whatever.

Speaker B: Those are the people that come to my mind. Journalists, effectively. Here in the 21st century, we have a choice to accept or not to accept the results when computer says no. Computer is not God. We can send for a journalist or a publisher like The Guardian or a lawyer like Chai Patel. We can start a crowdfunding campaign in social media. But wait, who will see this crowdfunding appeal? Well, that depends on the algorithm. What’s delivered to our news feeds is not a complete set of all available campaigns, opinions, and information. Even if you could amass enough computational power to deliver it all, it would take many lifetimes to read. It’s a problem that Julia Haas at the Organisation for Security and Cooperation in Europe’s headquarters in Vienna, Austria, is working on to ensure national governments apply fair and transparent regulation to online content.

Speaker A: So we have seen this trend to regulate online content in many states. For example, I mean, the most well-known is the one in Germany, the NetzDG. But there, what we normally do is that we do a legal review. And that means that already at a very early stage, we get involved with the states, and ideally when the legislation is still in the draft state, so that we can then have a discussion. And then we normally look at it in very detail to see where we see potential problems for freedom of expression. And we would then make very specific recommendations to the state on how this law can be improved. And this is something where we have all across the OSCE region on different legislation, but for example, also in the context of online content regulations, been in a dialogue with the state where legislations have also been adopted and changed and been amended in a way that would improve the the situation for freedom of expression or media freedom in the respective country, that there is no adverse impact on freedom of expression. What sort of cases are you talking about here? I mean, here in the situation of algorithms, it’s of course a little bit more complex. What we have been speaking out for example is on hate speech cases. So if there are attacks, for example, on female journalists, This is something that we have been working on and focusing on for quite a while now because we have seen that over the last years by now there has been an increase of online attacks, particularly on female journalists. Now you mentioned earlier that algorithms also decide which information is served to us and when and in what order of priority. Why is that a bad thing? I would certainly not say that it’s necessarily negative or a bad thing, especially when we talk about how much content is out there. I mean, if we, if we take one of the big social media platforms, it would be absolutely not user-friendly if there was not some way of ranking. So of course it makes sense from the company’s perspective, but also from a communication point of view that there is some kind of of ranking mechanism. And it’s also a possibility to say you want to promote or demote certain content. And the challenge, however, or the problem to a certain degree, is that the algorithms used for ranking content online are driven by commercial interest. So the purpose of the algorithms to rank content on platforms is of course also to make the platform user-friendly and more attractive, but the main purpose, or at least the second purpose, is to increase the profit of the company. So we really have to look again at the context. The incentive of the social media platforms, for example, or others that use ranking mechanisms, their incentive is to increase the users’ time time on their services. So they will present the users with content that they think is so attractive to them that they will stay on the platform, that it will increase their time, and that they will look at more ads, that they will click maybe on more ads so that the company can make more money. And there have been studies, and it’s often referred to the term like echo chamber, that say that the information available out there, even though there’s so much information out there, it’s not really available to the individual user because the information that one sees when accessing these services is very much curtailed to themselves based on the algorithms which base their decisions on what to show the people on the data. So we are kind of creating a system of surveillance or like of data collection and retention. And this data is then used by algorithms to determine which information might be attractive to you, but attractive not in a way of this is particularly newsworthy, but attractive in a way of keeping you longer on the service. And this is the challenge in this context. Context. So the moderation per se does make sense to a certain degree or might be necessary because of the amount of data out there. But the way it is implemented is in a way to increase the profit for the platforms themselves. And this has an impact on the information seeing, it has an impact on polarization, It has an impact on potentially even radicalizing. When we speak, for example, of a video platform, and once you watch a certain video, you get a recommendation to watch the next one and the next one and the next one, then maybe each of these will be a little bit more extreme or a little bit more radicalizing, because otherwise you wouldn’t watch it anymore. You wouldn’t stay on that platform. So there’s an incentive kind of by the company to do whatever or to show you whatever content might keep you longer on the platform. And this is something that is not necessarily good for pluralism, that’s not necessarily good for democratic discourse, and that’s certainly not good for freedom of expression as such.

Speaker B: As well as the OSCE, The European Union is concerned about AI decision-making and particularly about the curse of the black box, and has asked in its consultation on the new Digital Services Act for opinions on it. What makes the full disclosure of algorithmic decision-making difficult is because most of the applications are commercial. So commercial confidentiality, patents, and intellectual property laws protect them. From scrutiny. Thus, the so-called black box prevents us from knowing what parameters the programmers gave as the required results in designing each algorithm and which datasets they used to train it. Getting inside the black box is not easy, but at Northwestern University in Chicago, Professor Nicholas Diakopoulos is teaching his students how to audit algorithms by literally starting with the output and working backwards to find the input.

Speaker A: Given that algorithms are sort of impacting us in so many different ways on a, on a daily basis, I, I do think it’s helpful to think about how, as individuals, we can poke and prod these systems that are around us. Us. And so a few years ago when Uber was rolling out its dynamic pricing algorithm, which, you know, inflates the price of a ride depending on the volume of demand and the number of people who are using the app, I would do an exercise with my classes where I would have everyone in the classroom open their Uber app at the same time. So you can imagine like 100 or maybe even 200 students all opening the app at the same time. And we know from reading the Uber patents that that is one signal that their algorithm uses to feed into this dynamic pricing system. So what I would do is I would have everyone open the app at the same time and then about 3 3 minutes later, I would ask them to check the price for a ride in the local vicinity. And you could very clearly see that the algorithm had spiked up the price. So it was reacting to that signal that we were feeding it. And so I think this idea of sort of finding ways to poke and prod those algorithms around us, whether it’s apps or online online sites to see how they respond back to us, I think, can be very empowering. It can maybe give us stories. It can maybe give us insight. It can maybe be used to inform experiences where we, where we share how these algorithms have reacted to us online. And maybe that can help create more awareness and visibility for these systems. Online. And as an academic researcher, I’m sure you have to present from time to time your research plans and grant applications to a board of ethics to scrutinize the way that you’re doing your research and make sure that it’s ethical and it’s not harming or disadvantaging anybody. How are algorithms governed by ethics? And how do you think they, they could be perhaps better governed? Yeah, that’s a fascinating question. I think that obviously the context matters a lot, you know, so in the context of academic research, we have institutional review boards which evaluate the ethics of our research at the university to the extent that they’re looking at the methods that we use and, and making sure that, you know, when we’re including human beings as subjects in our research, that we’re being thoughtful about things like privacy and whether or not we’re causing any harm, even inadvertently. I think the broader question of ethical approaches to AI is a fascinating one. I think that in general, my attitude towards this is that we should be developing ethical standards around transparency related to algorithms in society. And what I mean by transparency is really just making information available about the performance of some algorithm in society. And, you know, by making that information available, we can help people evaluate and monitor and assess the workings or the performance of those algorithms. So transparency in and of itself doesn’t mean that we would necessarily be able to hold algorithms accountable, but it’s sort of like the starting point. It’s the informational substrate that would allow society to have the kinds of conversations that we need to have to debate, you know, Well, should the algorithm be more like this, or should it really be looking at this piece of data over here because it’s not reliable? Or, you know, maybe we should talk about how this is impacting certain minorities in different ways. And so I think if we can establish strong norms around transparency and maybe even regulate some forms of transparency, I think that should encourage the ethical, sort of conscientious use of algorithms more broadly in society. You know, I don’t want to say that, you know, transparency is the one and only answer or that’s going to solve all of the problems with algorithms. But, you know, to come back to the use of algorithms in the education education system, imagine that we had a law that said, you know, whenever a government organization wants to implement an algorithm that impacts the admissions of students into universities, imagine that there was a law that said you had to publish all the information about how that algorithm was created and that there had to be a 3-month public comment period for people to weigh in and and sort of have a conversation about, is this good? You know, how is this going to work? Is this hurting anyone? I think that we can put these systems in place so that we can actually reap some of the rewards of these algorithms, but again, do so in sort of conscientious, careful, ethically informed ways.

Speaker B: That’s the result that Professor Dr. Katharina Zweig is also working towards. But instead of reverse engineering algorithmic output, she’s concerned that ethics, law, psychology, and a whole range of human factors should be built into the input, the creation of the algorithms. At the Technical University of Kaiserslautern in Germany, it’s evolved into a new academic discipline.

Speaker A: State or an institution decides to use an algorithm to make their own decisions. And we are at the moment researching about the question how you could regulate that and when do you need to regulate that. Because I’m sure that not only in Germany we have the discussion of too much regulation that might hinder innovation and too little regulation that might harm us. So what I proposed is that we need a differentiated approach to assessing the risks of an algorithm decision-making system, and those that bear a higher risk should be regulated more heavily and also evaluated more heavily. Who or what sort of public bodies or independent experts would be suitable according to your research to do this evaluation? You definitely need experts that have a computer science background because you really need to dive into the algorithms to understand where they make the decision depending on, on what kind of information. On the other hand, it is not enough if you only have the technical expertise. So this is why we started a new field of science at the TU Kaiserslautern, which we call socioinformatics. Informatics. In this field of science, we give the students a 50% software engineering education, or 50% of the time is dedicated to software engineering, and the other 50% are dedicated to ethics, philosophy, psychology, economy, and sociology to make sure that these students know how the society and individuals interact with software. And at the moment, I would think that in Germany there are maybe 2 to 3 dozen experts that could do this. So I think that will be a job opportunity in the next 10 years. Given that these decisions are so important and so far-reaching, not only in the United States but everywhere, doesn’t it seem strange that only now socioinformatics is emerging as an academic field of study and research when we have had AI since the 1950s. Why has it taken so long to put together ethics and algorithms? Yeah, that is a very good question. But the thing is that the methods in part stem from the 1960s or ’80s. But the data wasn’t there. So for the first time in history, we have a situation in which the data is actually there to look at the behavior of humans. And so coming to the question of should grades be predicted or not, we wouldn’t have had the data on the students in the earlier days to really do that. In a large scale. So the big difference is that now we have the data, now we have the tools and the computational power to actually compute all of this stuff. And now comes the question: is that actually what we should be doing or not? And what do you think is the answer to that question? Yeah, so my answer is it depends. I think that’s the most favorite answer of any professor. So Of course, you can use AI and algorithmic decision-making systems in production to decide whether a product that is on the line is perfect or not. That is also a decision made by a computer, and I’m fine with that one because it doesn’t bear these very heavy consequences on a human life. However, if it is used to make a prediction about human behavior. I think we should know better. Humans can be predicted, of course, in the sense of statistical behavior. Of course, you can predict what percentage of people will buy a certain product on a given day. So if it’s raining, it’s not so hard to anticipate that more people will buy a raincoat than on a nice However, do we really want to have the same kind of thinking if it’s about the future of a person, either in court or in school? I don’t think so. There are also in these commercial uses of algorithms that you were talking about just then difficult areas known as a black box, a lack of transparency, which is important for companies to keep secrets. It’s a trade secret, and if everybody knew, then everybody would do it, and no company would then have an advantage over another by the algorithm it’s using. How can we get past that need for a lack of transparency or a commercial confidentiality? So again, do I really need to know why a certain online shop predicts me to buy another I’m not so sure we need transparency there. So if I don’t like the skirt being predicted for me or recommended to me, then I can just change the online shop. And there’s so many of them that I don’t have a big problem with that. So let’s look at an algorithmic decision-making system that might be a little bit more consequential, which is search engines.. And you might want to have even more transparency on how the algorithms work. Here, the idea of transparency or not getting insight in the algorithm is not only the intellectual property argument. It is also that if everybody knew exactly how the search engine works, we would get a lot of manipulation of search engine ranks. And this is actually not in the interest of any one of us. So there’s good reasons to keep some of the algorithms more opaque, at least with respect to the wider society.

Speaker B: It’s all very well for academics to poke and prod and preach about ethics in algorithms. In the real world, they’re so widespread they can be bought as off-the-shelf packages. Just buy one, point it at some data, and tell it what you want to achieve. Whoever originally constructed the algorithm might have had irreproachable ethical standards, but do they fit your new applications? I talked to Matti Aksela, who modestly calls himself a technology guy. He’s actually the Vice President Artificial Intelligence at F-Secure, one of the world’s leading cybersecurity companies. I think in today’s world, I think really it is about those people who develop and design. And also in a way you could say steer towards. So there’s more and more tendencies of having different frameworks for building AI solutions, some are becoming more and more automated. So actually it’s a little bit the line between building an AI solution, how much of that is actually somebody who really deeply understands the machine learning and is thinking about that underlying algorithms, how much of it is that and how much of it is basically that you take a framework and then you just give it some data and give it an objective to optimize and then it becomes even more important. What I’m saying is basically if you were to say the stereotypical, some years ago, machine learning expert is a PhD in computer science and really knows fundamentals. Is it enough for those people to understand the ethics? Actually, probably not, because there’s a lot of people applying machine learning in these existing packages that don’t really think about even the underlying models. And that’s kind of, in a way, the democratization of data science and machine learning is both a huge opportunity, but then there’s also kind of a slightly a risk, because when people, like we were talking earlier, in a way, common understanding sometimes is that actually it’s more than it really is. I think, I forget whose quote it was, but modern AI is basically complex curve fitting. So that’s actually pretty true. So most of them are really complicated, simple curve fitting solutions that optimize a certain boundary between decision boundaries and try to make the most out of it. So when you don’t also think about that stuff, I think that’s one of these challenges potentially. If you don’t really have the right expectations on what your algorithm can do, and then you point it towards even a good task, a positive thing, like for example these recruiting examples. If you don’t understand in a way that your data will be biased or is biased in that situation, if you don’t see that coming in a way, if you don’t think about it in advance, then you might reach totally different outcomes than you intended. And that can be arguably, well, again, we’re a little bit kind of, ethics isn’t my strongest point, but kind of even if your intentions are good, you can also reach outcomes that are nothing like you hoped. So therefore then, what you need to do is, because for AI to work properly, as you’ve already said, it needs good data.

Speaker A: Yep.

Speaker B: Now, what protects the good data is the cybersecurity. Yes. So, you know, the data that the AI system is drawing on for the purposes of whatever it is doing. Hmm. Now, taking into account what we’ve just said about people having different imperatives and and each considering that what they’re doing may be moral. How are you going to say, should there be a blanket rule that says interfering with somebody’s data should be proscribed, that should be an illegal act? I know there’s different viewpoints in this, and I’m not saying mine is correct by any means necessarily. I think that, but usually I’m a technology guy, so how I see these things, it’s not actually really, it’s not the technology, and it’s not necessarily even a very low-level action that is the one that you should control. So, when we’re talking about AI especially, if you try to control the technology, it doesn’t make sense because you can use it for many purposes. Your example of basically interfering with some data, well, if that data is the count down timer to basically world extinction in a nuclear holocaust, I think it’ll be a fairly good thing to change that timer even if it’s touching upon somebody’s data that’s not yours, if you can stop really bad things happening. So, from my perspective, what I think is that it’s really the intent of what you’re trying to achieve. Most of this is prescribed in the current framework. So, we don’t really, to my knowledge at least, most of the legal frameworks So they deal with the kind of actions and what you’re aiming at. So if you, like, God forbid, kill a person, so it’s like the killing is bad. It doesn’t matter if you do that killing via kind of driving, I don’t know, hitting with a hammer or a knife or anything. That’s kind of, in a way, besides the point. If the intent, if your intent is kind of what you want, the outcome of your desired outcome of your action is negative, or if you want to cause harm to people, then it’s a bad thing. Forbid that for most parts. So for like AI, when we think about even in this kind of a, where you could easily argue that interfering with another person’s data is bad, it’s also possible to kind of think about the counterexamples of where they actually might be good. Matti Aksela agrees with me that algorithms are motive-neutral, but they are shaped and steered and stoked by humans who provide them with their fuel. Data and make them pump out their results. It’s the human motives we need to know and regulate. Just as a judge looks for the mens rea, the intention behind the crime, so those who would regulate algorithms must scrutinize what they are for. We can all do that. And as Joy Buolamwini says, we need to do that well and often. We don’t take a shower once a year and imagine we will stay clean. In both cases, if we stop paying attention, something will soon start to stink because our human bias is corrupting and trying to remove it is an ongoing process. That’s it for this month’s password. The researcher was Simon Skitch, Jane Wyatt wrote the script, and the producer is Blue Buffery. The recording of the original “Ain’t I a Woman” speech was made by Arlene Maguire for the Sojourner Truth Project, and Katerina Zweig’s new book “Ein Algorithmus hat kein Tagenfuhl”—”An Algorithm Has No Tact”—is published by Penguin Random House. The European Commission’s consultation on digital services has just closed, but the OSCE’s is still open for your views until the end of September. Next month I’ll be back with more of my musings on the role of technology in all of our lives. I’m Peter Waddell.

Speaker A: Thanks for listening. Goodbye. This program has been brought to you by Resonance FM. If you like what you heard, please support our work by making a donation at resonancefm.com/donate.