PassW0rd – 10 February 2021 (March of the Machines)

Speaker A: This program is brought to you by Resonance FM. If you like what you hear, please support our work by making a donation at resonancefm.com/donate.

Speaker B: Hello, I’m Peter Warren and this is Password on Resonance FM.

Speaker C: Advanced enemy air defense is a major problem.

Speaker B: It’s a promotional video for a Harpy military drone. Aerial vehicles like these played a crucial role in Azerbaijan’s 2020 victory over Armenia in the disputed territory of Nagorno-Karabakh. Oh, Artask, as the Armenians call it. More on that later as the Password team explores the latest thinking on killer robots and robot soldiers. Should they be banned? Could they be banned? I’m one of 4,000 people who are calling for a ban, but I want to find out, is it realistic? Is it right? We’ve been equipping fighting forces with new technology since the crossbow gave way to the blunderbuss. At the end of last year, the French Military Ethics Committee gave the green light to microchipping France’s soldiers, sailors, and fighter pilots. Chips inside their body will help to locate them in battle and enhance their resilience. It sounds like sci-fi. Indeed, in her press release, the Defense Secretary Florence Parley felt the need to reassure us that we’re not going to see Terminator marching down the Champs-Élysées. But AI systems are marching into conflict zones, according to a US ethics committee on the deployment of AI and military systems at the start of 2021. They say the US has a moral obligation to deploy AI to protect its citizens against potential enemies. Former British naval officer Commodore Patrick Tyrrell says that this march of the machines is inevitable.

Speaker C: I read today that there was a small robot that they’re looking to develop which has got a hawk moth’s antenna attached to it because the hawk moth is particularly good at finding different odours, finding scents. And you know, the feeling is if this little robot can then fly into an area after an earthquake, they should be able to detect carbon dioxide coming from a a human being who’s still breathing, or they can search for gas, or they can, you know, do all these other things. And these are things of science fiction 10 years ago, 15 years ago, but they are reality today. There’s going to be an issue in this, isn’t there? Because yeah, you could take that to that hawk moth technology, you could implant it into a person, you could probably fine-tune their nose. You’re using these implants of the, the the French are talking about putting into their soldiers. But that means that you’re going to make the soldiers different from normal people. You’re going to create an elite that is going to be actually distinct in capability from other people. They’re not going to be normal human beings by definition. Well, you are, but you know, it’s only an extension of equipping a group of elite soldiers with new technology weapons. You know, all we’re doing now is saying, well, okay, instead of using this particular rifle, we’re going to give you— enhance your sense. You know, when you give a soldier night vision goggles, it makes him see in the dark.. So it’s merely a question of degree rather than being something that’s absolutely brand new. Okay, but the French Military Ethics Committee say that banned modifications include cognitive implants that would affect the exercise of a soldier’s free will or changes that would affect their reintegration into civilian life. The Americans are talking about being able to put in an interface that allows huge data input. If you’re going to start doing these things, then some of those modifications, or those banned modifications, could be overwritten, surely? Oh yeah, absolutely. But that’s just like when you discover you can— in the First World War, you could fire phosgene gas shells at the enemy and it deterred them from attacking you. You know, and then gas was banned, and some of these things will get banned, but all the ability to deal with data, having the ability to take out a laptop, an iPad, and have instant understanding of the situation around you is going to be a valuable thing to do. And I, I think, yes, there will be some things that are considered almost not cricket, but you just hope that you have the international treaties that define what those are going to be. But as we saw in the Nagori— Nagori Karibak, yeah, there was a certain ruthlessness in that conflict where People were saying, well, we’re going to win at all costs, we don’t care. Yes, yes. And of course, that’s one of those most unpleasant things about warfare, is that there are times when human beings act in a way that we would not see as being, you know, positive behavior for human beings. Right, so the first response of most people to this is going to be what happens if a hacker takes over a regiment of soldiers? Does that mean that one, cybersecurity is going to become more important, two, that cybersecurity or that hacking in particular circumstances will be proscribed? Well, it may get prescribed, but you know, if someone gets control, let’s say, of a group of soldiers, is that any different from intercepting messages being sent to part of an army and changing the order so they do something different that benefits their aggressor, of which history is laden with examples. And it just makes— means it will be more effective, because what will happen is, presumably, within the helmet that the soldier is wearing, an order will come up and says, I want you to withdraw, whereas Actually, his boss wants him to advance. So it’s a question of how you get in and ensure that the messages that are getting to your forces are the messages you want them to get rather than the enemy wants them to get.

Speaker B: As far as we know, robot soldiers didn’t play a part in the latest brief bloody fighting over Nagorno-Karabakh. But it was a conflict that blended remotely controlled drones with conventional weapons. I’m calling it a conflict because history is written by the victors. Wes Rumbough, at the Center for Strategic and International Studies in Washington, D.C., has been explaining what happened to Password’s Jane Wyatt.

Speaker D: I think that technology played some role in that. I think that In particular, so one thing that missile proliferation people that had paid attention to Armenia and Azerbaijan before the conflict erupted in September there had paid more attention to their ballistic missile arsenals. In particular, Armenia had purchased the advanced Iskander missiles from Russia. And one thing that sort of prevented their usage of these sort of larger ballistic missile platforms was just the limited numbers. That were able to be used. And so they didn’t have enough to use them for larger strategic effect in the same way that maybe— so one example that we point to in our research that’s a sort of a parallel is that the Houthi insurgents in Yemen have been able to carry out a multiple-year ballistic missile series of ballistic missile attacks in Saudi Arabia. And the way that they’ve been able to do that is they’ve been able to refresh their stocks through proliferation from in collaboration with Iran. There wasn’t sort of that same prospect here. And so that had sort of limited some of the, some of the aims. I mean, so that’s a question of how good the intelligence and the ability of the either the remote pilot or the platform to be able to sort of suss out the pictures that it is taking determines some of their precision. Tanks in particular were a significant target. A lot of Armenian tanks were destroyed by both rocket artillery fire that was directed by drones in terms of giving them better targeting information, and then air defenses. So the systems that the Armenians would deploy to defend themselves against drone attacks or attacks from aircraft, those systems were also sort of a priority target for a lot of the Azerbaijani drones throughout the conflict.

Speaker A: And how good were the drones at recognizing appropriate targets?

Speaker D: I can’t make a commentary, you know, without being on the ground, and there’s obviously some amount that we can’t know about misses. Obviously, there was lots of video and lots of photographs taken of the successes, and it speaks to a decent amount of precision. These are very capable weapons when operated in combination with other modern weaponry.

Speaker A: Would you say that this conflict showed us the state of the art, or is there more advanced technology in terms of military hardware that we don’t know about?

Speaker D: It’s always difficult to comment on what we don’t know. You know, there’s sometimes the unknown unknowns in terms of what’s going on in the classified sense. It’s certainly the usage of many of the Turkish drones that Azerbaijan used, as well as the loitering munitions, that were, that it purchased from Israel are towards the top of the line in terms of modern weaponry. And I think that the most valuable piece of information that came out of it or came out of the Nagorno-Karabakh conflict was sort of the understanding of how these weapons platforms can be used, not just independently, but in concert and in combination to create new tactical problems on the modern battlefield.

Speaker B: The death toll for Armenia was 3,439 troops killed. Azerbaijan lost 2,854 with 50 missing in action. At least 150 civilians were killed on each side. We watched as people were blown to pieces. A cathedral was shelled. Villages burned. Drones have cameras. They can shoot propaganda videos too. Toby Walsh, professor of artificial intelligence at the University of New South Wales in Australia, believes war is starting to look altogether too much like a computer game.

Speaker E: I think one of the misconceptions people have is that, well, maybe we’re just going to have robots fighting robots and that’s going to be a good thing, we’ll get humans out of the battlefield. Battlefield, and we could just let the robots fight it out. And surely, surely there are some people, Ron Atkin, who would say we’re morally obliged therefore to develop, I mean, robotic soldiers because we can, we, because we can actually then reduce the collateral damage, the humans that get killed on the sidelines of all these battles. The problem with such arguments is that it’s not going to be robots against robots. There’s no such part of the world anymore called the battlefield, you know. There’s a little sign saying, “Battles over here, please.” We fight battles, sadly, in towns and cities. We see this going on today in Syria and elsewhere, in and amongst civilians, in and against civilians. And certainly to begin with, it’s going to be innocent civilians that are going to be the victims of these robots. And we’ve seen this with semi-autonomous warfare. We see this with the drones that were being used in Afghanistan, in Iraq, and we see lots of innocent people being killed. I mean, it was the The Pentagon Papers that were released out of the UN that demonstrated a few years back that 9 out of 10 of the people that they were trying to target were not the intended victims. And that’s still why you’ve got a human in the loop with all their situational awareness. And so to begin with, certainly it’s going to be far worse when we hand over those decisions to purely autonomous systems. So the idea that we can make fighting a clean thing with robots against robots, And of course, if it was that case, it would be simple. We wouldn’t even have to have battles anymore. We could decide it by a game of chess or a game of tiddlywinks. But war tends to be very asymmetric. The sorts of people we end up in conflicts with these days, ISIS and the people like this, they’re not gonna sign up to these nice rules of war. My robot against your robot, may the best robot win. It’s not gonna be anything like that. It’s just going to be warfare that’s much quicker, much deadlier. And it’s also worth pointing out warfare that happens by mistake. Because we’ve already actually seen a very good example of this. We already see what happens when we pit computer programs against computer programs. It’s called the stock market. And we see frequent flash crashes where we get in unfortunate feedback loops and everything goes wrong. And they put the put the circuit breakers in and they get— they stop everything and they unwind all of the deals that went wrong and say, I’m very sorry and we didn’t intend that to happen. That works with the stock market, but can you imagine what happens if that was, say, the North Korean DMZ, the demilitarized zone between North Korea and South Korea, and North Korea accidentally went to war with South Korea and a couple hundred thousand people got killed? Well, we could put the circuit breakers in, but we can’t unwind killing that couple hundred thousand people. And that’s— I certainly worry about, for example, that escalating into, for example, a nuclear conflict.

Speaker B: So we’re not going to see those scenes that did happen apparently during the Crimean War where people would turn up and picnic by the side of battlefields?

Speaker E: Hopefully not. Well, hopefully we’re not going to see any more wars like that again, but warfare is going to be, if we’re not careful, much deadlier, much quicker, much bloodier affair that’s over in seconds. And humans are not going to have the reflexes to be active participants. It’s going to be an arms race as to who’s got the fastest, best algorithms.

Speaker B: But what’s it like to actually fly an armed intelligent drone? They all come wrapped in euphemisms. There’s the MALE, Medium Altitude Long Endurance. There are UAVs, unmanned aerial vehicles, and loitering munitions. Loitering with intent to kill for up to 36 hours in the air at a stretch. It should be stressed that currently most drones are remotely controlled by people, but the fear is that soon they will develop into fully autonomous systems. Already manufacturers have developed systems like the Northrop Grumman SG-2 stealth drone drones where a number of units are controlled by one human, which the company is proposing to deploy to replace its Reaper units. Britain’s killer drones are US Northrop Grumman Reapers. Former Army chaplain, the Reverend Peter Lee, professor of applied ethics at Portsmouth University, told me how they work in action.

Speaker G: I have a very deep and personal connection to an incident where a human did not override an air defense system with tragic consequences. Many years ago, in the first days of the Iraq War, I was a military chaplain and I was called to conduct what’s called a kin-forming, to tell the next of kin that a British Tornado had been shot down. And it had been shot down as it returned to base by an American Patriot missile battery. And whilst none— I’m not privy— and none of the details were of the secret kind were ever published, the Patriot batteries were semi-autonomous or automatic, and human beings could override it if it saw, for example, that the system was going to make an attack on what they recognized as a friendly allied aircraft, there is a possibility of intervening in that system back then, and it didn’t happen. So I think that there is an issue about dependence. We can become dependent on satnavs. There’s an easy one that I initially resisted satellite navigation in my car because I loved maps and I still love maps, but I’ve become increasingly dependent on it. And as someone who didn’t intend to. So, yes, I think I think humans can become dependent on technology. I think what humans often like is predictability. They like the predictability of patterns, they like consistency, especially if you’re at war and your life might depend on it or success in a military operation might depend on it. So yes, I think there is a potential difficulty in how humans work with technology, and I think that would have to be an area of important training actually help humans, help military personnel to critique, if you like, the information that they’re being given, and also to have the confidence to override it and say, no, I’m not going to make that decision. And that’s not an easy thing.

Speaker B: Well, you know, there’s a celebrated case of the Russian soldier who overrode a nuclear strike because he didn’t think that a nuclear strike was coming in. That’s going to be a tough decision for anybody to make.

Speaker G: One example of overriding a decision that I wrote about in my book about the Reaper Force actually happened in Afghanistan where a Taliban, a senior Taliban target, had been watched for days and weeks and on this particular day was seen riding his motorbike into the local market and he was being watched and the crew concerned when the when this person drove out of the market and he was a few miles from his home, the crew was, if you like, cleared hot. They were authorized and cleared hot to fire a missile and kill him on his motorbike. But there were other vehicles coming and going, so there was a slight delay there. And they’d been trying to shoot this person for days. And then when they were cleared hot again, Someone in the next room, in the operations room, radioed into the crew and said, “Negative, there’s a kid on the back of that bike,” and the crew looked at the screen in front of them and said, “Nope, there’s no child on the back of that bike. We want to proceed with the shot.” From next door came the voice, “Negative, there’s a kid on the back of that bike,” so the controller at the headquarters got another aircraft who was a reconnaissance type aircraft that was overhead, to confirm, is there a second person on that motorbike? No, there’s only one. And this went on for several minutes and got to the point where the target person arrived back at his compound and pulled in, and so his family surrounded him, and that stopped the shot because of all the civilians. He then turned round, and from what looked like a small box or parcel on the back of his bike he lifted a tiny toddler and put that perhaps 18-month to 2-year-old child on the ground. And the crew who were quite willing to take that shot and kill this target, the pilot described to me that moment of horror when he realized that if he had fired the shot, it would have killed this child. So the hero of this particular incident, if there is a hero, was the senior mission intelligence coordinator in the room next door on her first day in that job. Outranked by every person in the crew and by everyone else in the system, but who thought that when this parcel was put on the back of the motorbike that there was something about it that just didn’t look like a normal parcel. And even when the tapes were reviewed and reviewed and reviewed, there was no clear-cut indication that this was anything other than a box or a parcel rather than a child. So there was some kind of intuitive insight there that a human made that many, many, many other humans missed in that incident. And so I think how that is captured in any kind of AI or autonomous system will be deeply, deeply complex. And I’m a bit skeptical that that level of, if I call it intuition, will be replicated in any easy way. Humans have evolved over millennia to have what we might call perception, or for the brain or the subconscious to see things that the conscious overlooks. How that could even possibly be built into a machine, I have no idea at this stage. So there’s a big gap between what might be possible and what people are aiming for, and somewhere along the line there might be AI might take an unexpected direction that we’ve not yet anticipated and it be used for something completely different that we’ve not even thought about yet. So there’s always the unpredictable and that which cannot be anticipated.

Speaker B: In that near miss, human intuition saved the life of one small child. If you’re old enough, You might recall a similar reaction which probably saved the whole world on September 26th, 1983. Russian Stanislav Petrov was on computerized sentry duty when his screen showed him 5 United States ballistic nuclear missiles heading for Moscow. His orders were to report it and trigger the Russian counterattack. It was completely unexpected, as such things usually are. The siren sounded very loudly, and I just sat there for a few seconds staring at the screen with the word “launch” displayed in bold red letters. A minute later, the siren went off again.

Speaker C: The second missile was launched, then the third, and the fourth, and the fifth.

Speaker B: The computers changed their alerts from “launch” to “missile strike.” There were no rules about how long we were allowed to think before we reported a strike.

Speaker C: But we knew that every second of delay took away valuable time that the Soviet Union’s military and political leadership needed. And then I made my decision. I would not trust the computer.

Speaker B: I picked up the telephone handset, spoke to my superiors, and reported that the alarm was false. But I myself was not sure until the very last moment. I knew perfectly well that nobody would be able to correct my mistake if I had made one. Yes, no, 50/50. Stanislav Petrov, who distrusted the machine, disobeyed orders and saved the world. That’s an archive BBC interview. Petrov was moved sideways to a less sensitive job, took early retirement, and later had a nervous breakdown. He subsequently died in 1997. Taking life-or-death split-second decisions comes with the territory in any military endeavor. Having machines taking them for us can actually help. According to most experts, it’s actually safer to use an automatic intelligence killing machine. That’s because of something called neuromuscular lag. It’s the time it takes for a human brain to communicate to the finger on the trigger. In that moment, a machine can adapt to change circumstances, like a 2-year-old toddling into the line of fire. A soldier can’t, and the child is dead. You’re listening to Password on Resonance FM. The radio station relies on your donations to stay free from advertising, independent, and open-minded. Visit fundraiser.resonance.fm and support their initiatives, including a lovely live stream gig by Art Terry from Cafe OTO, two outstanding outer globe DJ relays on Mixcloud Live, a crazy curtain-up show quiz, an astonishing art rocker compilation, and the Hello Goodbye Show’s epic pay-as-you-go. After this on Resonance FM, you can hear DJ Ritu with A World in London. To find out more about the ramifications of technology on society, visit our newly relaunched website at www.futureintelligence.co.uk. And if you are interested in our back catalogue, you can find us on Mixcloud. Meanwhile, from the London headquarters of the human rights group Reprieve, I’ve been finding out more about military drones, talking to Jennifer Gibson, Reprieve’s officer in charge of investigating extrajudicial killings.

Speaker A: I think it’s not an automated decision in the sense that we think of killer robots, where the robot makes the decision. But you have to understand how sophisticated the computer programs are, how sophisticated the systems are that are analyzing the intelligence, and how difficult it is for the person then getting the readout from that computer to understand the workings that have gone on behind that. And so we know, for example, General Michael Hayden, who used to be the head of the CIA in the US, has come out and basically said, we kill people based on metadata. And that almost says it all, which is you’re hoovering up a lot of data about people and having computers, very sophisticated programs, no doubt, help run that data. But in the end, really, you’re killing based on data. You’re not killing based on any known individual identity.

Speaker B: So your issue then is the use of AI systems to identify people in the first place. And that because there’s not necessarily any transparency over the way that those artificial intelligence systems have come to their conclusion, that in a sense that you, you could quite easily make a mistake with this. You don’t know how the AI system came to the conclusion that this person was somebody who needed to be shot.

Speaker A: Exactly. I think it’s the lack of transparency around the system. It’s the lack of transparency on the other end as well. So you have a program right now, for example, the US drone program that is targeting and killing people based often on this metadata that’s hoovered up and analyzed. And then no post-strike investigations, whether it’s because there’s not boots on the ground, whether it’s because there’s not capacity, whether because quite honestly, it’s not a priority for the US. What that then means is you also don’t have any refining of the algorithms. You don’t know if the algorithm is right, wrong, sort of right, because you’re not doing the work on the other end to figure out who you actually hit. And what then winds up happening is we see cases like one where we’re working on right now, where a family’s been targeted 7 times and they don’t know how to stop being targeted. An element of that is almost certainly the kind of interconnections, relations they have, and the fact that somewhere along the line they’ve been picked up by a network and deemed to be suspicious.

Speaker B: Now, I suppose there’s another issue here, isn’t there, which is, you know, these are— they’re not judicial in any way, sense, or form. I mean, are these states at war with America? Are these people just being picked off because there is this perception that there is a political opposition?

Speaker A: Yeah, I think that’s a very good point and a very good question, which is You have one of the biggest controversies about the US drone program. And then, you know, I say the US drone program, but let’s not kid ourselves, Europe is heavily involved in this program in terms of helping with the intelligence, in terms of providing bases here that enable the program in places like Yemen. But that program doesn’t follow traditional battlefields. If I told you where strikes were occurring, most people would be shocked to find out we were even engaged there, much less that we were firing missiles. It tends to be in many cases that these strikes are happening in communities where there is no active fighting going on. And so you can imagine someone going to a market and suddenly there’s a Hellfire missile launched at the market. That sounds absurd, but it’s common day practice in some of these places to look up and see a drone hovering overhead and be afraid that the drone might strike, not because you’re just afraid, but because it has struck and killed people you know in the past. There are people who are being hit by weapons systems who are not in war zones. Absolutely. Far from war zones and in places that are going about normal life without the specter of conflict, who are suddenly then put into a conflict zone by the Hellfire missile that launches from the drone rather than from anything going on on the ground. And what the US has done is basically formulated a counter-terrorism strategy that identifies suspicious behavior and then kills people for it, doesn’t try to arrest them, doesn’t try to capture them, but instead takes a shot and uses lethal force purely based on someone in the White House sitting there and acting as judge, jury, and executioner.

Speaker B: So you’re saying no due process, that the internet is It’s almost on the basis of an internet search that someone is executed.

Speaker A: Yeah, it’s absolutely no due process. And in fact, we’re in the courts in DC right now on behalf of an American journalist in Syria who narrowly missed being killed 5 times, believes it’s because his metadata got picked up by US surveillance systems and accidentally identified him as a terrorist because he was meeting with and interviewing a number of people the US was interested in. He’s gone to US court, and remember, this is an American citizen I’m talking about, Bilal Abdul-Karim. He’s gone to US court and just simply asked for the US government to tell them whether they’re trying to kill him, and if they are, to give him a right to defend himself in court. The US has gone to court to stop him from getting due process, has argued in court that they have the right to kill him without any due process or transparency at all.

Speaker B: Is that because they don’t want people to know how they’re coming to the decisions that they’re coming to and that they would regard that as classified information?

Speaker A: Exactly. They’ve declared it a state secret, and it is such a state secret that to even admit whether they are trying to kill him or not would risk national security, is their argument. And if you can imagine, this is an American who has constitutional rights before the courts. If you’re in a Yemeni family like the Al-Ameri family, who we’ve just filed a petition on behalf of before the Inter-American Commission, you have no right to go to U.S. court. The U.S. courts have basically closed their doors. So you’re then left in this bizarre conundrum where there’s a drone overhead, it’s already targeted your family 7 times, And you don’t know who to wave the white flag at. You can’t wave it at the drone overhead. You’ve written to the US and tried to clarify for the US that they’re making mistakes. The US isn’t conducting any post-strike investigations to find out whether who they’re hitting is accurate, and they’re not engaging with those groups on the ground who are investigating and trying to bring evidence to them. And so the result is that you’re running an execution program outside any sort of judicial process in which there is no ability for someone to stop you if you’re making a mistake.

Speaker B: Reprieve’s office in the City of London is just about 12 miles away from Belmarsh High Security Jail, where the founder of WikiLeaks, Julian Assange, sits shivering in his cell awaiting extradition to the United States. It was WikiLeaks who first showed the world how it looks to shoot civilians by mistake from a drone. You may find this distressing.

Speaker C: Hey, can I open the courtyard? Roger. I just estimated 20 of them. Oh yeah, that’s a weapon. Yeah. Hotel 26, Crazy Horse 18, have individuals with weapons. Yep, he’s got a weapon too. Hotel 26, Crazy Horse 18, have 5 to 6 individuals with AK-47s. Request permission to engage.

Speaker I: Roger that.

Speaker C: Uh, we have no personnel east of our position, so, uh, you are free to engage.

Speaker I: Over.

Speaker C: All right, we’ll be engaging.

Speaker B: Roger, go ahead.

Speaker F: I can’t get them now because they’re behind that building.

Speaker B: What you can hear in this clip from the WikiLeaks documentary Collateral Murder is the dialogue as a United States aircrew fires remotely on a Reuters TV news film crew. They’re carrying cameras, not guns. Even with humans in the loop, They got the wrong target. Machines are no more reliable than people in recognizing faces, the way people walk, the way they’re feeling. Yet China’s deploying emotional recognition technology so it can judge at all times the mood of a population under surveillance. But will it work? Across the United States, there’s been pushback If you’re a regular listener to Password, you’ll remember the Algorithmic Justice League founded by Joy Buolamwini after she discovered that facial recognition tech is racially biased. It often fails to recognize dark skin, women, and older people of color. Joy points out that this flawed technology coupled to a weapon could have terrible consequences.

Speaker I: When it comes to addressing issues of algorithmic bias and more broadly issues of algorithmic harms, we have to think of both cost of inclusion and cost of exclusion. So what you’re speaking about, right, are there ways to in some way make the systems perform better, to have better technical performance? The answer is yes, but even if you have better technical performance, let’s say for something like facial recognition, right, you have a cost of inclusion, and that cost of inclusion is being readily available to a mass surveillance state apparatus. So you can have more accurate systems, but these accurate systems can be abused. So you can have more accurate facial recognition that you say works well on a variety of faces, and you now you put that kind of technology on a drone with a gun with facial recognition technology, and the question is no longer how accurate is it, but is this the kind of society we want to be living in? So it’s really important that as we’re talking about ways of mitigating issues to do with algorithmic bias, we come at it with the perspective that these are socio-technical systems. So technical solutions by themselves are not enough. We have to address not just how the technology works, but how is the technology used, and also what kind of processes do we have in place to check what’s going on when they’re deployed in the real world. So when people talk about algorithmic bias, I’ve been hearing this theme of let’s eradicate bias from AI, right? Well, or our system is bias-free. And I always like, did you get rid of the humans? Like, how are you bias-free? Like, even if the tech is deemed neutral by some metrics, how we use it in and of itself has its own assumptions and biases. So instead of thinking, let’s eradicate bias— and I’m not saying we shouldn’t try to make systems as robust as possible— we need to also come in with the mindset that there will always be some kind of bias, some kind of harm that we need to be checking for. And so instead of thinking about bias eradication, let’s think about bias mitigation.. And in some ways, it’s more like having algorithmic hygiene. And with hygiene, you wouldn’t just shower once in 2019 and say, we’re done, we checked. We checked for bias 2019 on our model, we’re good for here on out. No, that would sound silly. So instead, when you’re thinking about something like algorithmic hygiene, it’s ongoing because you realize you have to stay vigilant. Right? The situation changes. And so when it comes to how do we look at improving these sorts of technologies, we need to look at how the technologies work and also how they’re being used and make sure that there are appropriate safeguards and also mechanisms for transparency and accountability in the first place.

Speaker E: Is there a way that we can turn back the clock and fix it?

Speaker I: Well, it depends on what we’re saying we’re fixing. If we think fixing is just looking at technology alone, that will not be enough. If fixing means looking at social technical processes, then we have some hope. And so when you have tools like what you see with AI Now with algorithmic impact assessments, this gives me hope because this says, let’s bring together the stakeholders, right, whose lives could be impacted by these tools we want to adopt. And let’s see if these systems are actually fit for purpose. Let’s think about the social and ethical implications at the beginning before we deploy. And so when we change the process by which we think about these technologies, that’s what gives me hope. If we stay in a situation where we just wait for something to break and then react instead of being proactive, then we’re going to be in a negative place. But we have so much opportunity right now and so much interest. I see it with the people who send us requests and stories to the Algorithmic Justice League to say, How do we do these things better? What are steps we can take? Where do we need to say no to these technologies?

Speaker B: Where can we improve it? This week, the Algorithmic Justice League announced its facial recognition campaign is coming to an end. It’s not exactly job done, but enough states and cities have taken it on board to ensure it’s on the agenda of the new administration in the White House because Of course, military hardware quickly finds its way into civilian life. As long ago as 2018, police in Dallas sent in a killer robot to end a siege, and data-driven predictive policing is the new normal in most European countries now. But does this reliance on data make us safe? I’ve been writing about cybersecurity for over 30 years and published 2 books on the subject. So I know the answer is no, because we need to protect that data to make sure no one’s interfering with it. That’s the point I put to John Brunei, founder of the SAGE think tank. He thinks there’s general support for military AI.

Speaker H: We may start seeing states and non-state actors rolling out these swarms of drones. You know, they may be the size of planes, they may be the size of gnats. Wherever they can be experimented on, they will be, and we will see what happens as a consequence of that. It won’t be until we get to see the actual horror of that sort of conflict, and it has to come to us on our iPhones, sort of like, you know, the Vietnam War was a television war. We need to see this in real time for us to see the absolute extent and horror that these things, these systems can let out. Once we’ve done that, then we probably can pressure politicians to do something about prohibiting them. But you see, arms companies, they will always make the case, well, you know, a drone, it doesn’t matter how big it is, it’s not a nuclear weapon, right? It’s a conventional weapon. And the beauty of a drone is that the smaller it is, and if it has a small munition on it, and if we can target an individual, say, for instance, the head of al-Qaeda in the Arabian Peninsula, as an example, and we take him out by the use of a penny packet explosive on that person, and we assassinate him for the common good, isn’t that better than marshaling the military forces of the United States and going to war against a country like Afghanistan or Iraq? So again, the argument that the government will put forward is that we need these weapons because these weapons will prevent us from going state on state like we did in World War I, World War II, Vietnam, Korea, and so on and so forth. The list goes on because, of course, we don’t want to see our young men and women coming home maimed or killed. We— and I suppose therein lies the problem. It gives everyone an out. We can fight our war and it can be clean or clean-ish. I mean, one or two people will be killed. But 1 or 2 people is much better than 1 or 2,000 people or 1 or 2 million people. And so when you make those kind of logical arguments and extensions, which then will be calibrated for media consumption, people will make a very strong case why we need to have AI.

Speaker B: AI is good. But John, I mean, that would, that would be the point because, and that works on that conventional model because one of the things about the Nagornyi Karabash conflict was it was over very quickly in real terms because absolutely basically one party had lost because it was as technologically advanced as the other. However, one of the problems with an AI war is that there will be a cyber component and that cyber component will actually drag in the population at large. One of the things that happened at the beginning of the first Iraq conflict, the first thing that happened was that the British turned off the Marconi telecommunications system that underpinned the country. They hacked it. They turned it off. They achieved psychological impact immediately. And those— that capability, that’s what people have been talking about as the backdrop to what’s going on with Huawei’s penetration of the UK system. That’s the fear. So how are you going to deal with that? Because that is a far more real problem than your drones.

Speaker H: Well, I mean, here in Australia, we deal with it by spending more money on cyber and hoping for the best. I mean, ultimately, that’s what all countries are going to be doing because we are— the democracy of modern warfare is that we’re all facing the same sorts of problems moving on, you know, moving forward. So the problem is, okay, we’ll always have enemies and we need to know how to get to them in the quickest possible time with the fewest casualties. So the technology allows us to do that. And behind the drones, as you rightly point out, are going to be all the cyber warriors who are going to be busily trying to hack the AI and make sure that they fly into hospitals and kindergartens and make the other side look really terrible. And show the world just how awful this whole conflict is. And of course, we’ve got the whole notion of the crowd and how cyber will influence the opinion of the people on the ground in terms of whether or not they’re going to support military activity or not. But, you know, we, we think we have a solution in terms— and it’s a very basic solution and it’s not really much of a solution. And that is We’re just going to sit back and say, government, spend some more money on here, invest some more in cyber warriors, and maybe we’ll have enough people of critical mass and skill that will be able to outfox or block the advances of the enemy. I mean, this is how it probably will progress.

Speaker B: There’s a clear answer to that question published just a few days ago. 61% of the international respondents to an Ipsos Mori poll by Human Rights Watch said they were against the deployment of autonomous lethal weapons. Across the 26 countries in a poll, India, Israel, and Brazil were the only ones where only a minority, 4 in 10 of those surveyed, wanted to see these weapons banned. For a robust defense of roboticized killing, I turned to Evan Ackerman at the Institution of Electrical and Electronics Engineers.

Speaker F: He wrote the blog post, “Why I Don’t Want to See Autonomous Weapons Banned.” I would separate drones and drone strikes and that kind of thing from what the next step is. Drones, as far as I’m concerned, well, robots in general are tools, and drones are remote controlled aircraft with a human in the loop. And I think the concern is that, is the next step going to be that a human is not in the loop for the drone and then the drone is somehow making algorithm-based decisions on whether to fire a missile at somebody or not? But I think that drone strikes right now and the use of drones right now is not something that really should factored too much into the discussion because they’re just remote-controlled aircraft. Their artificial intelligence, as far as I’m aware, is limited to, you know, keeping themselves from crashing to the extent that they need to and some navigation and whatnot.

Speaker B: But they’re not— Well, some people are arguing that that’s not the case, that their targets are selected using AI and that the drone, the only bit of the human that’s in the loop is the person saying to kill that person or not. The person the human doesn’t necessarily know that that person is the person who is a guilty person. In fact, there’s been a U.S. journalist who’s of Syrian extraction who’s been targeted 5 times by drones, and he’s tried to go to U.S.

Speaker F: courts to find out whether the U.S. government is trying to kill him. Well, again, I would say that what is the actual issue there? And is the issue that there are are humans making decisions that are incorrect, or they’re making those decisions based on bad information. And if the information is bad, where does that bad information—

Speaker B: where does it come from? And if it— algorithms chasing around the internet, identifying the person in the first place. So in a sense, it is an AI impact.

Speaker F: Well, right, but those algorithms— I’m not trying to kind of pass the buck from artificial intelligence here, but we have to remember that these algorithms are designed and deployed by humans, right? And in the same way that all robots are designed and deployed by humans. They’re tools. And I think that it is certainly a risk when you have these algorithms running around and collecting lots of information and analyzing it in ways that you don’t really understand. But I don’t think that that necessarily means that like AI is fundamentally a bad thing because I think that there are ways to do this intelligently and with accountability and in potentially a much safer way than we are now. And I think that as such, it would be a very powerful tool.

Speaker B: Yeah, accountability is the key word there though, isn’t it? Because you want to be able to say, ‘Can you stop firing these Hellfire missiles at me because it’s more than irritating?’ So you want to be able to say, ‘I want to find whoever wrote this program or whatever information source they’re getting this stuff from that is targeting me, and I want to be able to refute this and much more importantly, stop the drones from firing at me.’ There’s been another family who’ve apparently been shot at 7 times. By a health permit, because I think they’re remarkably lucky given how effective those missile systems can be.

Speaker F: I think human accountability should be a crucial part of AI, yes.

Speaker G: Good.

Speaker B: So what, in a sense, you seem to be saying is that you have to have a human in the loop, that we need this— the reason that I suppose we’ve been talking about AI in the way we have when really what we’re talking about is machine learning, is that the absolute nightmare of this entire scenario would be if we achieve the singularity and that some system starts to make its own decisions about who should die and who shouldn’t die, which I think we would probably find uncomfortable. I mean, that’s an interesting point, though, isn’t it? Because we’re only a few days away from the 100th anniversary of the first performance of Rossum’s Universal Robots by Karel Čapek. And in that book, the robots or the androids are deployed as soldiers, and that becomes a very, very bad thing to do. And when they are given a little bit of a soul, a little bit of empathy, they rebel. And take over and kill the people.

Speaker F: Very, very chilling thought, eh? Yeah, I’m not all that worried about it, and I think the reason that I’m not worried about it is because I spend so much time writing about robotics that it’s given me a very, very tangible appreciation of just how hapless robots are so much of the time and how ineffective they are at even the simplest tasks. And it’s difficult sometimes to square that with the very sophisticated things that they can also do. But the idea that they do anything intelligently, I think, is a stretch right now and it will remain a stretch very far into the future. So I’m not worried about that.

Speaker B: Yet. Brilliant. We can leave it there. Is there anything you want me to ask you about that I haven’t covered?

Speaker F: I think, I think we’ve covered most things that I would— I mean, the issue around whether or not autonomous weapons should be developed, there’s just so much to talk about there. I mean, we could go on and on. I think we’ve hit a lot of the high points. I mean, one thing to make clear, I guess, is a big part of this issue is just whether or not an autonomous system should be able to make a decision on its own whether to use lethal force. And in that, the answer might be no. And I know a lot of people think that the answer is definitely no. And I just think that there is nuance here and it’s worth discussing and it’s worth doing some research on. Because it, it could be a good thing rather than a bad thing, and there’s potential there. And I don’t know if that’s how it’s going to shake out, but there are a lot of very smart researchers who say we have an idea of a way that we could make this work, and we should see if we can do it.

Speaker B: Now, these nuances are giving me pause for thought. If a machine can decide to shoot or not with no muscular drag, can react far more swiftly than a human. If soldiers can go into battle feeling strong and well supported thanks to the microchips inside them. If proxy wars like that between Azerbaijan and Armenia can be over in a few weeks with minimal casualties. And computerized killing takes less of a toll on the mental health of the armed forces who protect actors? Should I change my mind about calling for a ban on lethal autonomous weapons? No, I won’t. My gut feeling tells me it’s wrong. It tells me that we are losing our humanity and that we are passing over control to the machines. My gut feeling tells me that what we have to do is work on our humanity and become human. As Commodore Pat Tyrrell puts it, it’s just not cricket. It’s not because I’m a human and I believe in human primacy. I believe in fairness. I think these systems need to be fair, and it’s only when we know that they are fair that they should be deployed. I think that for all of the technology we’re using at the moment, because as things go, what’s happening is a technology is being used to give one group an advantage over another group, and there is no level playing field, whether it be in war or society. If you’d like to find out more about these nuances, visit our new website, www.futureintelligence.co.uk. Follow me on Twitter and LinkedIn, and join us next month for another edition of Password on Resonance FM. This one was produced by Blue Buffery and written by Jane Wyatt. I’m Peter Warren.

Speaker D: Goodbye.

Speaker H: Weather conditions and does not require precise intelligence data.

Speaker A: This program has been brought to you by Resonance FM. If you like what you heard, please support our work by making a donation at resonancefm.com/donate.