Home
Trending
New Release
Speaker A: This program is brought to you by Resonance 104.4 FM. If you like what you hear and want to support our work, please make a donation at fundraiser.resonance.fm.
Speaker B: Hello, I’m Peter Warren. I’m a middle-aged white man with brown eyes. If I stand in front of a facial recognition camera, the AI behind it will recognize that I’m a middle-aged white man with brown eyes. But what if I look different? A report just published on the recent Metropolitan Police trials of live facial recognition algorithms warns, and I quote, the technology may behave differently depending upon an individual’s sex, race or colour, thereby giving rise to discrimination. So, in this edition of Password, we confront the uncomfortable questions: Is AI racist? And if so, what can we do about it? Earlier this year, I interviewed the pop star Will.i.am at a conference we had organised on AI because Will.i.am has big plans for AI. And is investing heavily in the technology. So I asked him why.
Speaker C: I come from a very poor neighborhood. Lack of education is getting worse and worse, and I’m optimistic about it because in a society that I come from that is underdeveloped, I believe that the next jobs of tomorrow are going to come from communities like mine building around this new technology. A new way for us to communicate with machines naturally. That’s the way we speak. No more swiping, no more typing with QWERTY, even though your phone isn’t limited to the typewriter, but yet we still type QWERTY. I believe we’re going to have, you know, a bright future with AI coming from the underserved.
Speaker B: Okay, and so how do you see that happening then? What are the ways that you would like to see that happen?
Speaker C: First off, there are these data monarchies in society today, and no one is educated on the power of data, especially inner cities. And the way that I see that happen is when data is empowering my life, and the only way to have that is I needed my own data scientist, and a data scientist in the form of an AI to make sense of my data for me and communities that and communities as a whole. Because right now, the only thing that’s being benefited— the benefactors of data are companies, the 5 data monarchies on Earth today.
Speaker B: Okay, so do you want to see much more education then in those sorts of areas you’re talking about, and probably hopefully across society as a whole, about how people should use that data and what they should do with that data?
Speaker C: Well, then that brings up a whole new— a whole different conversation. I don’t think AI is the problem of humanity. I think greed, which is a human thing that we still haven’t fixed. How do you balance the power that we have and the resources that we have so that everyone has an equal opportunity to an awesome education to where data is empowering their lives? How are we going to educate? Because right now we’re not, right? And 2040 is around the corner, and a 7-year-old is not in the inner city or developing countries are not learning these things. I believe education is the most important thing, right? As machines are getting smarter and smarter, we’re having this conversation on AI. Humanity is not getting smarter and smarter because of the limitation that we have aimed at children to develop, right? So if you look at the, the investment for the development of AI versus the development for human intelligence, it’s kind of lopsided. And that is the reason why AI is an issue.
Speaker B: Okay, now you talk about poor areas where you were brought up, but you’ve also just mentioned lots of other countries. I mean, there are obviously, you know, places in Africa which don’t have a very good internet at the moment. So do you think that AI should be rolled out worldwide in, in that sense, and, and the opportunity should be given to all of those places?
Speaker C: As a society as a whole, we haven’t specified the, the areas that we’re concerned about. The AI that I’m speaking on is natural language understanding, natural language processing, having a conversation with the computer to keep the disabled capable of the power of computation and the ease of what you compute. And that’s the way we speak. So I’m specifically talking about natural language understanding, natural language processing, and conversational computing.
Speaker B: Okay. Here, for large parts of the day, we’ve been talking about bias and bias in AI. Are you concerned about AI becoming biased?
Speaker C: I was up until we started developing our artificial intelligence operating system to make sure that that bias doesn’t apply in systems like ours.
Speaker D: So if you—
Speaker C: if I would be concerned if we weren’t developing it.
Speaker B: Okay, so you think that that’s the way to avoid it, to actually use the AI to root out its own bias?
Speaker C: It’s biased if only one demographic is building it. It’s always going to be biased based on who’s building the platform.
Speaker B: Will.i.am on why the development of AI can’t be left in the hands of good, well-intentioned chaps. Dr. Joy Buolamwini of the Massachusetts Institute of Technology Media Lab has founded a movement called the Algorithmic Justice League. She realized the limitations of facial recognition tools when she was making a video art project. Joy found that she had to put on a white mask before the AI-powered camera would recognize her facial features. You can see it at the Barbican in London or find it on YouTube. Not surprisingly, Dr. Bulu Lambwini welcomes the recent decision by some United States cities to ban facial recognition technology?
Speaker E: I think it’s a great move to put context limitations on technology, and it also shows that we have a choice and a voice. Oftentimes the argument is the technology already exists, there’s nothing you can do. What we see with a city like San Francisco or a town like Somerville in Massachusetts having a ban is that you can actually say we can set the parameters for how technology is used or not used. And so in so much as it demonstrates a counter narrative to the technological determinism that we often hear, I think it’s a win in that direction. And also given what we know about the technology, that it has racial bias, that it has issues when it comes to gender, when it comes to age, Using it in a high-stakes situation when you already know there are these technical shortcomings in and of itself would be irresponsible. But not just do we have the technical shortcomings, we also have to think about the implications for privacy. Or in the US, right, when we think about First Amendment, Fourth Amendment, Fourteenth Amendment rights, what does it mean if we have technologies that enable mass surveillance for having a democratic society where you can’t think of freedom of association, or you might not want to go out and protest because you’re going to be tracked, or you get searched without warrant because you somehow look suspicious to a technology, right? Or that you have no idea AI has been used. So maybe apply for a job and because of algorithmic bias, you don’t get that opportunity. But how would you ever know, right? When I mentioned the 14th Amendment really talked about due process, that I ought to know the systems and mechanisms that are in play so that if something happens that has a negative consequence, I have some sort of recourse. And so until we reach a place where the AI systems around us really are adopted in a way where we have affirmative consent, I get to choose whether or not I’m using the system, there’s meaningful transparency, What system is it? Why is it being used? What do we know about the limitations of the system? And then also continuous oversight. What safeguards have we put in together as a society to reduce these harms or address these harms? We should be taking the approach that we’ve seen in places like San Francisco, which is to say there are certain areas that we do not want to use this technology.
Speaker F: Right. And of course, there are some AIs, for instance, chatbots, which as soon as they’re released, they immediately pick up on human interactions, some of which might be racist, Islamophobic, antisemitic. Microsoft’s bot called Tay only lasted for 16 hours before it had to be taken down because of all the hate speech it was propagating. To what extent do you think that artificial intelligence is has almost got a mind of its own in this sense because it harvests so many different human attitudes.
Speaker E: In this case, what was good about Tay is you could actually see what was wrong because as a chatbot, it’s interacting with people, right? And so you see what it’s saying and it was vile. And because it was so public, and vile, it was taken down quickly. It had high visibility. What’s more pernicious are the AI systems you don’t see, the AI systems that aren’t necessarily being quote unquote explicitly racist, right? This is more akin to unconscious bias. So let’s say you have an AI system that’s taking in zip codes and it’s using zip codes to determine if you’re creditworthy. Well, there is a history in many parts of the world, right, where depending on where you lived, you were not going to have access to certain opportunities because of either your race or religion or whatever have you. And so now you can end up using zip code as a proxy for something like somebody’s race. And so now you don’t necessarily have to be explicitly racist, right? Instead, you can use this proxy that has been shaped by racist practices and you can nonetheless come out with a negative result, even if that’s not your intention. But here you don’t necessarily see some kind of vile tweet going out. You don’t actually see what’s going on in the first place. And that’s why I’m saying we need to have meaningful transparency for these systems. Here’s another example where you can have AI systems reflecting society in ways that aren’t obvious at the beginning. You have a whole set of tools for natural language processing. We’re going to read your social media posts and try to give you something like a sentiment. Is this positive or negative? Or we’re going to, in some cases, what we’re seeing are governments wanting to use social media to see if somebody is a threat or not. A threat to public safety or to national security to make a decision about immigration. What we know about the processes around natural language processing right now is many of the models that are used are mainly trained on English. And they’re trained on standard English. And so if you have people who are is speaking in a different manner than what is expected, or even speaking a different language, the interpretation of what’s happening is already biased and skewed by the language models in the first place. So here you don’t have to have somebody who explicitly doesn’t want an immigrant population to come to a country. Instead, you could have a situation where natural language processing tools that were trained on a very specific subset of English are now being deployed on people who don’t fit those assumptions, and it seems somehow neutral, right? And I mean, there— I do believe there was even a case where a UK man was denied visa entry, I think coming to the US, because he had written “destroy America” on social media. But the context for “destroy America” was he wanted to go to party. Not to blow up the place.
Speaker B: Keeping society safe from terrorists is the main rationale behind using facial recognition software. Britain’s former head of anti-terrorism at the Met Police is Tony Porter, now the UK CCTV Commissioner. His mission is to win public support for these new technologies and the databases that inform them.
Speaker G: What are the standards that can provide assurance that the data is accurate? And the data emerges, it percolates across this whole debate. Is it the data that you use to inform the watch list that the police will cross-check against? Is that accurate? Is that fair? Are people on that database that perhaps haven’t been charged and there’s no intelligence case that they should be used in that arena. So that’s one area. The other area, of course, is do the police have sufficient guidance in terms of where cameras are sited, what the lighting should be, what the acceptable level of performance is? And I have not seen that. In fact, I know that it is not there. And I think they are the arguments. If the debate is to move along, it’s not simply data and it’s not simply technology, it’s legitimacy and the law underpins it. But you asked specifically about data. Those questions have to be addressed, and they have to be addressed through standards which provide reassurance to the public that they are there to protect them.
Speaker B: I mean, a lot of people have begun to say that AI is racially biased, and the reason that they give for that is large proportion of the prison population in America, for example, is black and that there are social reasons for that. That knife crime, if you read the London Evening Standard, would appear to be carried out by a lot of black people who are involved in drugs. Terrorism, that a lot of the terrorist atrocities that we’ve currently seen seem to come from people within the Islamic community. So if you feed those databases into your search, What’s being said is that’s instantly going to teach the AI to be racist. What are your thoughts on that?
Speaker G: I think it is extremely complex. From what little research I’ve seen or been exposed to, there does seem to be a bias in the use of facial recognition technology that prejudices against minority ethnic communities in this country. And indeed gender, and it prejudices them insofar as there is difficulty in accurately recognizing certain minority ethnic groupings or diversity. Women, I think, as I understand it, is more complex or less successful in its identification. So the question you ask is, well, so what if you have human intervention? Is that a problem? And I think the answer very simply and in a nutshell is, well, it is a problem if you’re a minority ethnic or you’re from a gender that is inappropriately or with a lack of balance identified, because it’s potentially you that will be subject to an intervention, whether that is a check or an identity check or a suspicion. That is a problem, and I think very much that before this kind of equipment is used Those concerns need to be completely eradicated. We live in a harmonious society, and one of the ways we keep that is by demonstrating absolute fairness.
Speaker B: Okay, but we need— what we seem to be saying is let’s rip up the databases that we currently use, set up the databases that we’re going to use for this new technology along rigorously fair Guidelines. Well, for AI to work, we need data. So that means de facto that really we can’t start using AI in these particular scenarios until we’ve developed a usable amount of data that people are prepared to cooperate with.
Speaker G: Again, an interesting point. I think what we need to say is that where we do use the data that we currently have, we need to absolutely make sure and evidence that it’s accurate and fairly held. Now, there has been a contentious debate about the custody photo image database that is held across the police PNC, their national computer, and the arguments are around there are some people on there that actually haven’t been charged where there’s no intelligence case there’s no automatic weeding of that data. So people could end up on that database and indeed on a watchlist purely through happenstance, and that actually is one of the key concerns. Now, I don’t agree with you. I don’t think that we can ask the police to get rid of 18 million custody images and more, but I do think we can ask the police to fast-track and ensure that database is accurate and fairly held and they can evidence that. I don’t think it’s acceptable that we ask the public to ensure that they are not on the database, which is the current position. And these are some of the debates that have percolated up from the facial recognition issues that have arisen.
Speaker B: Do you think then that there’s a case for a licensing technology for this? So you wish to do something with AI in CCTV, you approach the license in body and you say, this is what I want to do, and you stick clearly to that remit, you get the relevant data according to that remit, that there needs to be that level of transparency.
Speaker G: I do believe there needs to be that level of transparency. I know that in the, in the current work I do, we have third-party and independent certification of many police forces and actually most local authorities now, and I think that can a kind of governance that is perhaps a little more than light touch but can provide oversight and independent assessment is what is called for. Whether that is done by a government licensing body, I suspect probably given the invasive nature of the technology, or whether it’s done on a par with the sort of work we already do, that is a matter for policymakers. But yeah, I think, I think that could indicate the direction of travel.
Speaker B: Okay, and also that the police should make applications to a judge for particular things that they wish to do with data and AI at the moment. A lot of people have been suggesting that.
Speaker G: Well, again, I made a speech at the annual data conference at Taylor Wessing last year, and what I drew from that speech was look at the parallel between what the police have to do to acquire a covert surveillance, warrantry, and look at how invasive the type of AI facial recognition technology is. There is a very clear parallel. Yet in the former, the police need to go to judicial commissioners and chief constables to get authority. Yet at the moment, in relation to this technology we’re talking about, which is arguably more invasive, they don’t. And I think that’s a very clear lopsided approach to governance of this issue, and that needs to change.
Speaker B: But it’s unreliable. That report on the Met Police trial showed, and I quote, live facial recognition technology matched 45 individuals at the scene to individuals on the watch list. 16 of those matches, more than one-third, were deemed non-credible and rejected. And on the anniversary of the Macpherson Report that followed the murder in Southeast London of Stephen Lawrence, the Metropolitan Police Commissioner Cressida Dick is clearly aware that Macpherson labelled the Met as institutionally racist. She’s been answering MPs’ tough questions on what has changed in the past 20 years. Across the pond, latest statistics reveal the male prison population of the United States is 37% Black and 22% Hispanic.
Speaker H: And Princeton University’s Associate Professor Ruha Benjamin, the author of a new book, Race After Technology, says that can create a self-fulfilling prophecy through AI-driven predictive It’s not just a matter of making technology better at the level of technology, but asking what kind of society do we want and whether certain technologies may, as a starting point, something that we just don’t want to include. And I think more and more people are converging around the idea that in terms of facial recognition technology in particular, it does not sort of benefit the greater good. And so you have places like San Francisco that have banned it. From use among their law enforcement. Other cities are considering legislation, and more and more people who otherwise are supporters of other types of automated systems, when it comes to facial recognition, they understand the nefarious ways that it could definitely be used, for example, in dampening social protest. So if you can deploy these in terms of big crowds and look for people who are exercising their right to protest, it could make people more reluctant to get involved in the democratic process, in the process of holding politicians accountable if there are these technologies that are surveilling them at a distance, right? And so these are the next layer of questions I think that we have to wrestle with.
Speaker F: But Ruha, it’s not really fair to blame the technology, is it? Surely the datasets on which these artificial intelligence algorithms are being trained are the information on which they base all their decisions. And if you could fix the input to the AI, then surely the output would also follow in a more racially balanced way.
Speaker H: It’s a great question. And I think you’re right that blame is not necessarily the right framework, but I do think that there’s plenty of responsibility to go around. So there’s many, at many different points, there are places in which we could be making better decisions. Yes, the existing data is one point, right? But in terms of how you actually code a system, what you weigh, let’s say algorithm that is meant to decide whether someone should be released on parole in prison or have a longer or shorter sentence. In the process, yes, the input data is biased because of widespread racial profiling practices that make it so that if you’re Black or Latinx, you’re more likely to have a criminal record. Therefore, you’re training software to associate criminality with these racial groups. But another point of responsibility or a place we have to look is exactly at the level of code where you’re weighting certain types of factors more than others. And so, for example, if you live in a community in which there’s a high unemployment rate. Many algorithms in the criminal justice system would say that that makes you more at risk for being a repeat offender if you’re entering a community that has a high unemployment rate. So the unemployment rate of your community is then made associated with you being a higher risk. Now that’s a problem because that unemployment rate, first of all, you have no control over, but more importantly, It’s a result of widespread forms of discrimination, oppression, historical legacies that have to do with the government divesting from your community. And so the fact that we’re taking this larger form of risk production, that’s the responsibility of municipalities, of social policy, and then we’re saying that that is actually your risk as an individual, and we’re not going to release you on parole, or you’re going to be labeled as higher risk because of this collective factor, right? And so that’s not just about the input data. That’s also about how the theory behind the algorithm, right, the ideas about where risk is, what constitutes risk. And so one of the things I really want to advocate for is moving away from this individual-level risk production and associating individuals as higher and lower risk and look at the institutions that actually create risk for people? What if we trained our algorithms to look at how hospitals produce risk or how various policies are, you know, produce risk rather than saying this individual is high risk? And that’s a different way of orienting the use of technology and also just thinking about where the danger in society lies. It’s not about individuals, simply individual-level danger, but it’s about how our social policies and our social order produces danger for different communities at very different rates also.
Speaker F: And what has been your own personal experience of this kind of automated bias?
Speaker H: The way that I sort of entered this was thinking about, you know, some old-school forms of policing, things that I grew up experiencing in my community in Los Angeles, and just seeing the way in which the targeting of people that I loved and my community in general was the justification for that over-policing was often a larger framework of security. And we have to make Los Angeles safe, therefore we need to deploy more police, more helicopters to guard and to surveil my particular neighborhood. And so it’s coming out of that experience experience where I realized that these two things go hand in hand. Like, surveillance of some people is often done under the pretense of a greater good. And that’s given me a kind of disposition to look at, okay, anytime the greater good, anytime these big platitudes are deployed to justify various kinds of surveillance, whether low-tech or high-tech, then I’m interested in looking more closely at what’s the actual effect on those who are being surveilled. And so I’m taking that experience growing up of policing and thinking about it in terms of this context of surveillance in which the human actors are often hidden. And so my goal in this book is to kind of pull back the curtain or pull back the screen as it were, and look at who are the humans making this made this decision? With what assumptions? Why are they only focusing on these forms of risk or these racialized populations? And who is it supposed to help in the end? And kind of trying to demystify that process.
Speaker B: In her new book, Ruha Benjamin is critical of the rapper Jay-Z, who’s created an AI startup called Promise. It aims to keep Black, and Hispanic suspects out of jail by providing funds for bail on condition that the suspect wears an electronic tag linked to a smartphone app. The issue here is that while it may be better than being locked in a cell, the promised tag puts a suspect under 24-hour automated surveillance. That same issue is exercising Britain’s Open Rights Group. Matthew Rice of Open Rights Group Scotland chaired a panel on bias in facial recognition tech at the AUGCON in London on the 13th of July. He told us about his concerns.
Speaker D: You know, one of the driving questions here is whether or not AI is racist, right? This idea that, are we using technology that is discriminating based on protected characteristics. Really, the question isn’t actually whether AI is racist, it’s whether we are racist and whether our policing systems and the data collection that we’ve done for not just policing but say for shoplifting or receiving benefits or anything like that, whether those have biases in them. These systems are not coming from the perfect training environment. They are based on databases that have already been made, right? And these databases are often from existing data collection initiatives, such as CCTV records or crime records. And those have been shown to have bias attached to them. So all of a sudden, you’re training a system based on a biased dataset. You then risk outputting bias. So even if we’re trying to, say, give some certainty or improve decision-making for policing, In fact, what we might be doing is just pushing decision-making into an opaque black box and then trusting it implicitly because it’s technology and we should trust technology, which is an intuition that we want to challenge.
Speaker B: Well, it strikes me that the two issues in this are one, data, the data that you’re using, because obviously the historical data, as you say, has bias in it, but it’s It’s also the search terms that you’re going to use or the way you’re going to program the systems to look for information. Because for example, a CCTV picture of an entire street, that’s not going to be biased in terms of the data collection. But if you say, look for all of the black faces in that street or look for all of the Muslim faces in that street, then to an extent you’ve biased your collection, haven’t you?
Speaker D: Yeah, definitely. Or the command would be, look for persons of interest or look for people that fit a particular profile of risk that we’ve added into this dataset. And so then it goes back to the dataset and it loops and goes, okay, so what are the people that have been classified as being at risk or dangerous? And if the algorithm comes out with saying, well, all these people with this color skin or all these people with this kind of outfit on or all these people walking around at this time of night, they have been deemed a risk. And so output risk for those factors. Unless you can begin to interrogate that decision-making process, you are at risk of perpetuating and compounding all the bias that we’ve seen already.
Speaker B: But a lot of people will say, hang on a minute, Matthew, a lot of the people who’ve gone around running people over, a lot of the people who’ve gone on stabbing sprees, they have all been Muslims. They’ve all got Muslim facial characteristics. So surely we’re right to search for those people.
Speaker D: Oh, they happen to be Muslim, right? That’s not because they’re Muslim that they’re at risk. That’s merely a factor at play. There’s a lot of other factors that are going on there as well, and it’s not their physical appearance that makes them criminal. It’s a whole host of other things, and that’s the real problem. Of course, we’re also seeing a rise in white supremacist acts of violence as well. So how we build those pieces in, and we go point, counterpoint, back and forth on all of this. And all of a sudden we introduce technology into the mix and we’re told we either have to accept it or reject it. It’s a bit of a heady mix and it seems to kind of polarize when what we should be doing is trying to be a bit more nuanced in our debates.
Speaker B: Well, but again, I mean, I was talking the other day about this just down the pub, people were saying, well, hang on, all of the people who are involved in knife crime, the majority of those people are black. And if you put in information about white people saying, look at white people too, then you’re distorting the database because you’re puzzling it.
Speaker D: Yeah, I can see that. And I guess it depends on the purpose that you’re using it for. So is it knife crime that we want to use it for? Is it counter-terror? Is it blue-collar crime? Is it for protests that will have a lot, you know, a very different kind of profile attached to it? How you kind of define the success vectors or how you define the particular purpose is going to be key to avoiding any of that bias. And yeah, I mean, I think we’ve What we get mixed up in is sometimes alongside the reporting of knife crime or the reporting of violence, you could quite easily move away from, say, skin tone or ethnicity and instead say that it’s coming from particular areas and particular locations and particular neighbourhoods, right? So all of a sudden Tottenham becomes this deep hotspot, and so then people in Tottenham will be seen as a greater risk even though there may not be any greater risk and it purely happens to be that these people live in Tottenham or purely happens to be that they are of this ethnicity. So it’s a risky business. I think it’s more important that we focus on, if you want to introduce this technology, how do you make sure that it doesn’t fall back on the discrimination or the bias that we’ve perhaps established? How do you make sure that you correct for those? How do you make sure that you can interrogate them rather than just rely on it, blinded by the notion that it’s producing an output and so we just have to go with that output?
Speaker B: You’re listening to Password on Resonance FM, and after this you can hear DJ Ritu with A World in London. One of the world’s top AI designers is Jana Eggers, unusual in the AI and high-tech world because She’s a woman, though now 25% of those involved at the highest level in AI are women. So why then do AI systems not like women?
Speaker I: You have to be very thoughtful about what you ask for, and you have to be very thoughtful about the data that you give it. And honestly, the algorithm is the most simple part, and what most people think of as AI is the algorithm, and they don’t think of it as the data and the objective that’s been set. Been set. So that’s where the bigger challenge is. Now, you do have to know about your algorithm and what control you have over it. And is it a black box or not? And there’s lots of different AI algorithms out there. Most— what most people are thinking of, and honestly, many of the mistakes and the things that cause you to say AI hates women are really because they’re because of the data and not the algorithm. And there’s lots of different algorithms out there, and deep learning is the one in biggest use. That’s the one that eats the data in this way that it’s not transparent and you can’t see some of the biases. Some algorithms talk back, I guess I would say, more than others.
Speaker B: That must be a problem because we could be— we could get into a position where we could get to an algorithmic spiral where one algorithm talks to another and feeds it wrong information and they both start behaving badly. If we don’t like what AI is learning, how can we stop it?
Speaker I: Well, I’d go back, you know, data is one, and how is the algorithm trained and how do we make sure that we have datasets that are representative of something fair and not biased? Especially when we live in a biased world. Human beings are extremely biased. The only reason we’re finding out about this now is that AIs can scale it at a different scale than humans ever could. So you had, you had a few biased loan agents and now you have somebody that can make a decision much faster and at greater scale.
Speaker B: So what you seem to be saying is that currently AI is making us a hostage of our data. Should we use AI then to clean up humanity’s data before we let it loose on humanity?
Speaker I: I love the fact that we’re being held hostage to our data. I don’t— you know, there’s a good part of me that looks at it and says, I’m very grateful that AI has brought this to light. You know, we— I don’t think we realized how much of our data was biased. I really don’t think so. And I, and I believe that, that AI has brought that to light. And now, yeah, we can, we can focus on it. I’m not sure about cleaning data, though.
Speaker E: I am—
Speaker I: the cleaning data point is a tough one because I talk to our customers about that quite a bit and they always want to clean their data. And I’m like, yeah, but you can’t guarantee that it’s coming in clean. So I think there’s more that we can do with different types of algorithms to really play it against itself to see if this is a good idea. And so I’m not sure we’re going to have to clean it. I think we’re going to have to come up with different algorithms. And I think that we’re going to have to have different datasets, some that are, you know, what we would consider gold sets to play against those that are more normal sets that are collected in our biased world and have it to find things that aren’t biased within it. And there’s really some— I keep saying I think, and it’s actually not a think. I know that we have those kinds of opportunities because there’s leading indicators that we do.
Speaker B: If we do have those gold datasets, we’re going to have to make sure that cybersecurity protects them absolutely, because any interference with a gold dataset would be a very bad thing to do. But if AI needs historical data to learn, our historical data is flawed. We’re in the position that you say. Shouldn’t we just abandon the idea for the time being? Why do we need AI now?
Speaker I: Because of the amount of data that’s out there now. So much is— what’s changed is the availability of that data. You know, before, none of this was being recorded, and now our lives are recorded all over. And that’s, you know, you can say, oh, that’s a terrible thing and we should stop recording, but you can’t do that. We all love the fact that our lives are recorded now. We’re You know, we enjoy that. We share more pictures with friends. We get on the phone, you know, across the pond and record conversations between each of us. And all of that is out there now. What’s really exciting is leveraging that, and we just have to learn to leverage it. I mean, you know, we’re— one of the ways I describe it is, uh, you know, you’ve got a bunch of computer scientists and mathematicians like myself who are drunk on data and compute power. And, you know, now we have to learn to— we’re the, you know, I’m going to use the US example, of course, you know, we’re the 18-year-olds who just learned to drink and we’re drinking too much. And we have to straighten up and learn to drink responsibly.
Speaker B: So, as Jana says, we have been drunk on data and there have been a couple of gaffes. Microsoft Tay being one of them. Remember Tay? Tay was a chatbot. It took her just 16 hours to become a fanatical race hater after her feeds were interfered with by social media trolls and she took the temperature of the internet. In the original idea behind computer pioneer Alan Turing’s idea of artificial intelligence was that it could perfectly imitate a human conversation. Turing called it the Imitation Game. You may have seen the film where Keira Knightley implausibly plays the role of Turing’s lover. It’s implausible not because she’s a bad actress, but because Turing was gay. There you go, another one of those bad data points that we bedevil the truth with. Imitating our bad data real life was how Tay became a racist, picking up on anti-Semitic and pro-Nazi tweets and posts that went viral and making them even more extreme. So it’s possible to train AI in bad ways by using bad data. Dr. James Zhu is an AI researcher at Stanford, the Silicon Valley university. He co-wrote an influential article in the scientific journal Nature calling for better training of artificial intelligence to prevent racism and other biases.
Speaker A: Right, so that’s exactly the major challenge, right? So all of these algorithms, AI algorithms in particular, are trained on large quantities of data, especially historical data. So they’re really quite hungry little things that require a lot of data to train and to to feed on. And when the training data is biased, which is typically the case with historical datasets, the algorithms just learn that these biases are real patterns, right? And they would latch onto these biased patterns and then deploy them in applications. So there’s an active set of research ongoing right now to develop methods to first to reduce biases in the dataset and also in parallel to reduce biases in the learning algorithms, even when the training set, training data is biased.
Speaker B: Okay, but that is still going to be a tremendously difficult task, isn’t it? Because what you’re basically saying is that the Any bias that you get into the model at the beginning from the data will start to skew the program, make the program biased, and in the process of the program training itself, it will then become even more biased because it will be finding data that in a sense feeds its prejudices.
Speaker A: That is true, yes. So, but there are methods that we and other researchers are working on to mitigate some of these effects. So for example, one thing you can do is to create new data, right? That tends to be more balanced, right? So an example of this would be, let’s say, you know, from training algorithm on sort of like a voice recognition system, right? It’s natural language processing. So typically the data that’s used to train these algorithms are historical text data, maybe data from Wikipedia, from BBC, New York Times. And those data might have certain types of gender or even ethnic racial biases built into the text. So there are ways to take those data and create synthetic datasets, right, that are very similar. The only difference is that it tends to be more gender or racially balanced, right? It doesn’t really address the root of this challenge. It doesn’t solve all of the problems, but it does go towards some ways towards mitigating the biases in the algorithms.
Speaker B: Okay, so in a sense, what you’re beginning to do when you’re using a process like that That’s going to be the beginning of the training of the, of how to clean a database of bias, because presumably you use the AI to clean its own data.
Speaker A: That’s right.
Speaker B: Given that we’re in this situation, isn’t it a little dangerous to start deploying these systems until we’re sure that they’re actually pretty fair, unbiased?
Speaker A: Well, ideally, you know, we would do a lot more experiments and to really make sure that these systems are fair and unbiased. But the reality is that these AI systems are already being deployed, many of them, by both companies large and small, right, all the way from Facebook, Google, to startups. And they’re deployed right now in many applications applications, both from cars to medical applications to looking at financial investments. So they are already being deployed. So what we can do right now is to think about first, how can we carefully audit these existing algorithms for potential biases, right? Similar to how someone might audit Donald Trump’s tax return. So we also want to audit these machine learning algorithms to detect what are potentially biases and issues in these learning algorithms. And once we identify these biases, then come up with some of these techniques like I described to try to mitigate the issues.
Speaker B: Okay. But in the meantime, I mean, so you think it’s safe to deploy AI at the moment? Or do you think it is safe to deploy AI in particular areas at the moment?
Speaker A: I think there are many areas where AI is being deployed and it’s providing a lot of benefits and value to society.
Speaker B: Right.
Speaker A: So for example, you know, when you are talking on your, you know, talking to your smartphone, asking for directions, right?. So there are already AI systems working under the hood to first parse the speech to recognize it and then to do more intelligent question and answering, right? And there are also many other settings where the AI systems are actually already being used to help, and they are helpful to clinicians, to physicians, to augment the human capacities. So there are certainly many, many benefits of smart applications of AI systems. So I would certainly would not want us to stop using them. I think we just have to be careful and cautious and cognizant about where are the potential biases in these systems and try to be very transparent about where are the algorithms being used in every step of this decision-making process as a way to to start to understand the biases in the systems.
Speaker B: I mean, that’s the key, isn’t it? DeepMind developed a system that allowed it to find eye disease in people by looking at patterns in people’s eyes. There, the data is already clean, isn’t it? Because you’re looking at existing patterns. And also, in that case, you’re looking for historical data. And data in a particular form. So that’s a very good way that you can use it, isn’t it?
Speaker A: I think that is a good way. I would caution that even in examples such as that one, there could still be issues in the datasets that could bias the algorithms, right? So for example, in many medical diagnosis settings where people are trying to train machine learning algorithms, the data used to train algorithms actually come from not from the general population, population, but from, let’s say, people who are more likely to get checkups, right, which tends to underrepresent sort of the, let’s say, the minorities or the less, you know, maybe the lower class or people who don’t have access to the medical care, right? So there are often different types of subtle selection biases in the data even when we try to be very careful to collect good datasets.
Speaker B: Skin color’s not much of an issue in computer games since it can be easily swapped and customized, just like the costumes and accessories, all collectively known as skins. But the artificial intelligence that powers the games is definitely biased, at least according to Kate Edwards of Geographi. On PlayStation, she’s Valkyrie Kate, but by profession she’s a cultural consultant and mapmaker. Obviously, the Valkyries and Norse gods in popular computer games are white. Does Valkyrie Kate think they should be more ethnically diverse?
Speaker J: Well, it’s interesting because in the different art forms, you know, the Norse gods as an example that you raised have been represented in a lot of different ways. I mean, I mean, if you look at something like the Marvel Cinematic Universe with the Thor movies and whatnot, I mean, they have diversified the Asgardians. They’re not all white and blonde-haired. I mean, the character Heimdall, who even in the Marvel comic book was white, is represented by Idris Elba in the movies as a Black character. Same with the Valkyrie character, who in the comic book was white and blonde-haired, and in the movie she’s played by Tessa Thompson, who’s also of a you know, has a Black background. So it’s just like, I didn’t really hear anyone complain about that. Nobody really blinked an eye necessarily. I mean, some people may have noticed, of course, but the portrayals were so well done that really any criticism of the portrayal was pretty much, you know, squashed in a certain degree. So I think in this age where we see diversity inclusion as a positive value, I don’t think a lot of people are too angry about that, but it really depends on the context. I mean, the Marvel Cinematic Universe is quite different compared to if you’re going to actually make a movie that’s based on, like, you know, the Elder Edda and the older poems and trying to be super accurate and true to the original context in which that information was created. Then yeah, in that case, you’re probably going to go for more accuracy from a racial standpoint, but it just depends on the context and the demand for the narrative. So I think that’s always really important to consider, that it’s not necessarily about being politically correct for politically correct’s sake, or, you know, having underrepresented people just for the sake of showing underrepresentation. You know, at the same time, we can look at other pantheons of gods like the Hindu gods, which are obviously South Asian in nature, and you’re not going to see, you know, white, blonde, blue-eyed people in the Hindu pantheon either. And I don’t think anyone really complains about that necessarily, because that’s essentially something that is, you know, pretty much set in cultural history.
Speaker B: If you eliminate bias, then you’re theoretically going for some sort of truth, aren’t you?
Speaker J: That’s true. You, you are trying to adhere to some level of what is true. I mean, you know, in my background as a cartographer, as just as an example, I mean, cartography is dedicated to the notion of ground truth. So for those of us who make maps, when we’re creating a map, what we’re really trying to do is represent the quote real world as accurately as possible, obviously in a very generalized form that’s easier to consume in a, you know, in a scaled-down form. But the reality in the process of cartography geography is that many countries have what I call the geopolitical imagination of what they own. So for example, in India, you must show Jammu and Kashmir, the disputed region in the north, as Indian territory, you know, unequivocally. It cannot be shown as disputed territory. And that’s required by law. And conversely, in China, you have to show parts of Kashmir as well as the state of Arunachal Pradesh which is occupied by India. All of that has to be shown as Chinese territory, including Taiwan and including the entire South China Sea. So just those two examples alone, those realities are, they’re fictions on maps. And yet the Chinese official map and the Indian official map, which are required by law, those are fictions from a ground truth point of view. And so sometimes we have to make those adjustments adjustments in order to basically accommodate what would be the local expectation, you know. And sometimes that local expectation is reinforced by government decree or policy, and sometimes it’s just a cultural mores that more or less dictates what would be considered appropriate.
Speaker B: So in a sense, you’re saying that that’s biased, because— and you’re saying that that’s a bias that we have to live with. So we have to— we have a sort of thorny issue here, don’t we? Because, for example, obviously we’ve been suffering a lot of terrorist attacks here, um, and a lot of those seem to be coming from people who are of an Islamic background. So if we are to unbiased the AI racial facial recognition technology theoretically we should put into the database more information about people who aren’t Islamic, and then theoretically that wouldn’t be biased. But other people would say, hang on, that’s not true though, because lots of the people who seem to be attacking you seem to be coming from the Islamic community, right?
Speaker J: Exactly. And I think that’s one of the things that we have to account for. I mean, number one is we have to absolutely absolutely embrace the bias. And by embrace, I mean, we have to be keenly aware of it. I think sometimes we spend so much effort on trying to pretend we’re not biased, when we are. I mean, it’s— we’re human beings, we’re biased. And there’s not much we can do around that other than be aware of what our own bias might be and be, you know, as individuals, we have to be introspective. And as cultural systems, whether it is a political system or social system or whatever it might be, I think it demands us to be introspective and be very mindful of what bias we might be building into a system.
Speaker B: Still, not every games manufacturer follows Kate Edwards’ guidance when it comes to Black and minority ethnic characters and skins. Even deliberately offer players the chance to persecute other races. Take in the details of this from Brad Davies, FI’s games analyst. That’s a big pile of meat chunks.
Speaker F: I didn’t do nothing.
Speaker D: It’s on now.
Speaker C: Oh, oh, now you going to hide, huh?
Speaker I: Call your mother.
Speaker G: Anyway, shut the fuck up.
Speaker B: Watch out, get these big ass It’s an AK-47.
Speaker H: Awesome.
Speaker B: Litters are an inspiration for birth control.
Speaker E: I think I just made her into a martyr.
Speaker B: Oh well.
Speaker H: I love blowing minorities up.
Speaker B: I did not see you there. Too dark. This’ll come in handy for shooting niggers from afar. Oomph. That’s all from this edition of Password with me, Peter Warren. Blue Buffery’s the producer, Brad Davies the gaming consultant, and Jane Wyatt wrote the script. You can find out more about AI at our website Future Intelligence and by joining Matthew Rice at the London conference of the Open Rights Group this weekend. Thanks for listening.
Speaker A: Goodbye. This program has been brought to you by Resonance 104.4 FM. If you liked what you heard and want to support our work, please make a donation at fundraiser.resonance.fm.
