The government has developed new technology which can “automatically detect terrorist content” online. Propaganda by Daesh (also known as ISIS) could apparently be blocked as soon as it is uploaded.
The software was created by private company ASI Data Science, with £600,000 of government money.
The claims are bold, but will it really make any difference? Or, could it lead to accidental censorship and the loss of important evidence against terrorists?
What counts as a Daesh video?
According to the Home Office the algorithm was developed by analysing “over 1,000 Daesh videos”.
The exact details are a closely guarded secret, to stop terrorists decoding the software. But John Gibson, from ASI Data Science, suggested that homemade videos by individual supporters may not be included.
“We focused on a very specific set of videos – but very, very deliberately,” he told FactCheck.
“There’s a kind of canon of videos that are created by Daesh that are sort of official, or quasi-official. They’re released by Daesh media agencies and, quite specifically, these are the videos the Home Office prioritises most highly.”
Videos used to train the software were “surprisingly highly produced and actually just very good,” Gibson told us. “Their job is explicitly to motivate people to commit acts of violence, and so they’re emotive, they have incredibly graphic images of brutal acts; tragic acts.”
However, labeling videos specifically as Daesh propaganda is potentially tricky. After all, there are many militant groups which are technically distinct from Daesh, but could seem extremely similar from an outsider’s perspective. Some are even aligned with Daesh.
For instance, Kateeba al-Kawthar is an “armed terrorist group fighting to establish an Islamic state in Syria”. The UK government says it is “aligned to the most extreme groups operating in Syria” and “has released YouTube footage encouraging travel to Syria”.
It may sound a lot like Daesh, but it’s named as a distinct organisation on the government’s list of proscribed organisations.
Without the full details of exactly which videos the algorithm is based on – and what it’s designed to detect – it is impossible for us to be sure how targeted the technology is.
The founder of the Bellingcat website, Eliot Higgins, who is an expert on the conflict in Syria, said that top experts would be needed to distinguish between some videos.
“You need people who understand the situation on the ground in Syria to make judgments,” he told us.
“Every single faction in Syria – from the government to the rebel groups – they’re all terrorists, depending on who you talk to. And this is another problem: who’s a terrorist group this week? The situation on the ground changes quite swiftly.”
How accurate is it?
Announcing the new technology, the government claimed it could detect Daesh propaganda “with 99.995% accuracy”.
That means out of a random sample of a million videos, it would wrongly block 50 – incorrectly believing they were Daesh propaganda. John Gibson said this rate makes it “very operationally plausible”.
But Eliot Higgins is more sceptical. Last year, his own videos were retrospectively deleted from YouTube after the firm’s artificial intelligence wrongly labelled his journalistic research as extremist material.
“It was probably looking for the audio track to see if there’s anything in there that could indicate it’s a jihadi video, and picking up on things like black flags,” he explained.
ASI Data Science says the algorithm was tested against “examples that were most likely to trick it”. But the accuracy rating given by the government is based on a random sample of videos – rather than simply those that might easily be mistaken for Daesh propaganda.
So is the accuracy rating far low when looking only at certain types of videos – such as footage of the Syria conflict?
When asked by FactCheck, ASI Data Science wasn’t able to immediately answer. “I don’t know with confidence whether actually that stuff gets flagged more frequently,” Gibson told us. “It’s possible that that is true, but I don’t know that it is true, and we definitely trained the model very intensely to minimise the extent to which it is true.”
How much human review would be needed?
The government initially claimed: “If [the algorithm] analyses one million randomly selected videos, only 50 would require additional human review.”
However, the Home Office and ASI Data Science have now admitted this is not correct.
Their figure only refers to the videos that are wrongly labelled as Daesh propaganda. But in order to spot these, the human reviewers would need to look at every video that was flagged.
That means the total number of videos a human moderator would need to review depends on how many real Daesh videos are uploaded on a given day. For instance, if there are 100 Daesh videos out of a random sample of a million, we would expect human reviewers to assess 150 videos in total: 100 labelled correctly and 50 labelled incorrectly.
This might sound like nitpicking, but in practice it could greatly increase the number of videos that need reviewing by experts.
What are the implications?
Reports have said the Home Secretary “would not rule out forcing technology companies to use it by law”. But the government has stressed there are no current plans for this and that the technology is still being discussed and reviewed.
Supporters of the software have welcomed any progress towards stopping Daesh propaganda. But – aside from its accuracy – there may also be concerns about the possibility of deleting useful evidence.
Eliot Higgins explained: “There are groups in Syria who are active on YouTube and Facebook who you would definitely consider extremist. But they also post a lot of stuff that’s very useful for investigations.
“If one video gets removed, then you’re losing a lot of information that’s useful to investigate.”
If implemented, the technology may throw up other questions, like what happens to people once their videos have been flagged, and whether content is fully deleted or just blocked in the UK. For now, though, the government is still in the consultation stage.
