QR code abuse 2012-2023

QR code abuse is in the news again—see the list of headlines below—whch reminds me that I first wrote about this in 2012 (eleven years ago). Back then I made a short video to demonstrate one potential type of abuse, tricking people into visiting a malicious website:

As you can see from this video, there is plenty of potential for hijacking and misdirection via both QR and NFC technology, and that potential has existed for over a decade. In fact, this is a great example of how a known technology vulnerability can linger untapped for over a decade, before all the factors leading to active criminal exploitation align. 

In other words, just because a vulnerability has not yet been turned into a common crime, does not mean it never will be. For example, the potential for ransomware attacks was there for many years before criminals turned it into a profitable business. Back in 2016, I suggested that combining ransomware with the increasing automation of vehicles would eventually lead to a form of criminal exploitation that I dubbed jackware. As of now, jackware is not a thing, but by 2026 it well might be.

Here are some recent QR code scam headlines:

• The QR code scam leaving victims thousands out of pocket
• Woman targeted in £13k railway station QR code scam
• QR code warning: Cybersecurity experts report alarming rise in 'quishing' scam
• QR code scams on the rise during festive celebrations; here’s how to be safe

Artificial Intelligence is really just another vulnerable, hackable, information system

Recent hype around Artificial Intelligence (AI) and the amazingly good and bad things that it can and may do has prompted me to remind the world that: 

Every AI is an information system and every information system has fundamental vulnerabilities that make it susceptible to attack and abuse.

The fundamental information system vulnerabilities exist regardless of what the system is designed to do, whether that is processing payments, piloting a plane, or generating artificial intelligence.

Fundamental information system vulnerabilities put AI systems at risk of exploitation and abuse for selfish ends when the ‘right’ conditions arise. As a visual aid, I put together a checklist that shows the current status of the five essential ingredients of an AI:

Please let me know if you think I'm wrong about any of those checks and crosses (ticks and Xs if you prefer). 

Criminology and Computing and AI

According to routine activity theory in criminology, the right conditions for exploitation of an information system, such as an AI, are as follows: 

• a motivated offender, 
• a suitable target, and 
• the absence of a capable guardian. 

A motivated offender can be anyone who wants to enrich themselves at the expense of others. In terms of computer crime this could be a shoplifter who turned to online scamming (an example personally related to me by a senior law enforcement official in Scotland). 

In the world of computing, a suitable target can be any exploitable information system, such as the payment processing system at a retail store. (Ironically the Target retail chain was the target of one of the most widely reported computer crimes of the last ten years.) 

In the context of information systems, the absence of a capable guardian can be the lack of properly installed and managed anti-malware software, or an organization's failure to grasp the level of risk inherent in the use of digital technologies.

When it comes to information systems that perform artificial intelligence work, both the good and bad uses of AI will motivate targeting by offenders. The information systems at Target One were hit because they contained credit card details that could be sold to people who specialize in fraudulent card transactions. An AI trained on corporate financial data could be targeted to steal or exploit that data. An AI that enables unmanned vehicles could be targeted for extortion, just as hospital and local government IT systems are targeted.

Do AI fans even know this?

One has to wonder how many of the CEOs who are currently pushing their organizations to adopt AI understand all of this. Do they understand that all five ingredients of AI are vulnerable? 

Perhaps companies and governments should initiate executive level AI vulnerability awareness programs. If you need to talk to your execs, it will help if you can give them vulnerability examples. Here's a starter list:

1. Chips – Meltdown, Spectre, Rowhammer, Downfall
2. Code – Firmware, OS, apps, viruses, worms, Trojans, logic bombs
3. Data – Poisoning, micro and macro (e.g. LLMs and SEO poisoning)
4. Connections – Remote access compromise, AITM attacks
5. Electricity – Backhoe attack, malware e.g. BlackEnergy, Industroyer

Whether or not vulnerabilities in one or more of these five ingredients are maliciously exploited depends on complex risk/reward calculations. However, execs need to know that many motivated offenders are adept at such calculations. 

Execs also need to understand that there is an entire infrastructure already in place to monetize vulnerability exploitation. They are sophisticated markets in which to: sell stolen data, stolen access, stolen credentials; and buy or rent the tools to do the stealing, ransoming, etc. (see darkweb, malware as a service, botnets, ransomware, cryptocurrency, etc.).

As I see it, unless there is a sudden, global outbreak of moral rectitude, vulnerabilities in AI systems will—if they are not capably guarded—be exploited by motivated offenders. 

Internet crime losses reported to IC3/FBI

For a sense of how capable guardianship in the digital realm is going, take a look at the rate at which losses due to Internet crime have risen in the last 10 years despite of record levels of spending on cybersecurity.

Attacks will target AI systems used for both "good" and "bad" purposes. Some offenders will try to make money attacking AI systems relied upon by hospitals, schools, companies, governments, military, etc. Other offenders will try to stop AI systems that are doing things of which they don’t approve: driving cars, taking jobs, firing weapons, educating children, making movies, exterminating humans.

Therein lies one piece of good news: we can take some comfort in the likelihood that, based on what has happened to every new digital technology in the last 40 years, AI systems will prove vulnerable to exploitation and abuse, thus reducing the chances that AI will be able to wipe us all out. Of course, it also means AI is not likely to make human life dramatically better.

Note: This is a revised version of an article that first appeared in November of 2023.

What is ChatGPT and how can AI get things wrong: an annotated example using jackware

You can't trust what ChatGPT says

ChatGPT is, as you probably know, a computer system that uses artificial intelligence (AI) to answer questions. Sometimes the answers it gives are wrong, and that's the short version of this article. The long version explains more about what ChatGPT is, with a detailed look at an example of how wrong it can be. 

Here's how Mashable describes ChatGPT in Everything you need to know about ChatGPT: "in essence, a simple online artificial intelligence chatbot created by OpenAI in December 2022." Technically speaking, ChatGPT describes itself as "a language model developed by OpenAI, based on the GPT (Generative Pre-trained Transformer) architecture...designed to understand and respond to natural language queries and generate human-like text in response." 

If you check out ChatGPT online it can come across as a chatty version of a search engine, but it can do things that search engines don't, such as put together plausible explanations of phenomena, as opposed to simply listing search results relating to them. For example, suppose you encounter the word jackware and wonder what it means. You could put the word into a search box and get a bunch of responses, like this:

If you are lucky, one or more of the search results will give you an idea of what the search term means. In this case we can see that "The goal of a jackware attack is to lock up the hijacked device until you pay" (which happens to be correct). 

But what if you're writing an article or essay about jackware and want to know who came up with the term and when it was coined? I couldn't find that information in the search results for jackware, and looking for it took time. In my experience, search engines often fail to answer questions like this, as you can see from the following example. When asked who coined the term jackware, Google provides one link to a story about jackware, but all the rest of the results—many thousands of them—assume that jackware is just a typo for Jaguar, as in the car, not the animal:

Personally, I find the jackware/jaguar assumption very amusing—for reasons I will explain later—but first, let's see how ChatGPT gets on when asked the same question: Who coined the term jackware? Here's the answer ChatGPT provides:

The term "jackware" was coined by two researchers, Stefan Savage and Tadayoshi Kohno, in a 2016 paper titled "Experimental Security Analysis of a Modern Automobile". In the paper, Savage and Kohno described a proof-of-concept attack in which they were able to take control of a car's electronic systems using a laptop computer. They used the term "jackware" to describe a new class of malware that could be used to take control of vehicles in a similar way to how ransomware is used to take control of computer systems.

Frankly, at first glance this looks like is a very impressive answer: it names names and cites an academic source, while providing a helpful description of jackware, all articulated in grammatically correct English. It's way more helpful than the output of a plain old search. You can see why so many people get so excited about artificial intelligence. 

But sadly, several key facts in ChatGPT's answer are just plain wrong, so wrong that you could be in serious trouble with your teacher-editor-readers if you pasted that paragraph of ChatGPT output into a piece of work. For a start, anyone familiar with the subject matter would know that you hadn't read that paper.

The academic paper to which ChaGPT refers was published in 2010, not 2016. You can see that from this listing of the paper. This is not just a pedantic quibble; the named paper is legendary in the world of automotive cybersecurity, partly because it was published way back in 2010. It documents groundbreaking work done by Savage et al. in the 2000s, way before the flashy Jeep hack of 2015 by Miller and Valasek.

More blatantly erroneous is the identification of this 2010 paper and its authors as the source of the term jackware. Simply put, the paper does not contain the word jackware. In fact, the person who coined the term jackware to describe malicious code used to take over vehicles, was me, Stephen Cobb, and I did that in May of 2016, on this blog, in a post titled: Jackware: coming soon to a car or truck near you? 

In July of 2016, I penned Jackware: When connected cars meet ransomware for We Live Security, the award-winning global cybersecurity blog. As further evidence, I present exhibit A, which shows how you can iterative time-constrained searches to identify when something first appears. Constraining the search to the years 1998 to 2015, we see that no relevant mention of jackware was found prior to 2016:Apparently, jackware had been used as a collective noun for leather mugs, but there are no software-related search results before 2016. Next you can see that, when the search is expanded to include 2016, the We Live Security article tops the results:

So how did ChatGPT get things so wrong? The simple answer is that ChatGPT doesn't know what it's talking about. What it does know is how to string relevant words and numbers together in a plausible way. Stefan Savage is definitely relevant to car hacking. The year 2016 is relevant because that's when jackware was coined. And the research paper that ChatGPT referenced does contain numerous instances of the word jack. Why? Because the researchers wisely tested their automotive computer hacks on cars that were on jack stands.

To be clear, ChatGPT is not programmed to use a range of tools to make sure it is giving you the right answer. For example, it didn't perform an iterative time-constrained online search like the one I did in order to find the first use of a new term. 

Hopefully, this example will help people see what I think is a massive gap between the bold claims made for artificial intelligence and the plain fact that AI is not yet intelligent in a way that equates to human intelligence. That means you cannot rely on ChatGPT to give you the right answer to your questions. 

So what happens if we do get to a point where people rely—wisely or not—on AI? That's when AI will be maliciously targeted and abused by criminals, just like every other computer system, something I have written about here.

Ironically, the vulnerability of AI to abuse can be both a comfort to those who fear AI will exterminate humans, and a nightmare for those who dream of a blissful future powered by AI. In my opinion, the outlook for AI, at least for the next few decades, is likely to be a continuation of the enthusiasm-disillusionment cycle, with more AI winters to come.

--------------^------------- 

Note 1: For more on those AI dreams and fears, I should first point out that they are based on expectations that the capabilities of AI will evolve from their current level to a far more powerful technology referred to as Artificial General Intelligence or AGI. For perspective on this, I recommend listening to "Eugenics and the Promise of Utopia through Artificial General Intelligence" by two of my Twitter friends, @timnitGebru and @xriskology. This is a good introduction the relationship between AI development and a bundle of beliefs/ideals/ideas known as TESCREAL: Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, Longtermism.

Note 2: When I first saw Google assume jackware was a typo for Jaguar I laughed out loud because I was born and raised in Coventry, England, the birthplace of Jaguar cars. In 2019, when my mum, who lives in Coventry, turned 90, Chey and I moved here, and that's where I am writing this. Jaguars are a common sight in our neighbourhood, not because it's a posh part of the city, but because a lot of folks around here work at Jaguar and have company cars.

Internet crime surged in 2022: possibly causing as much as $160 billion in non-financial losses

Financial losses reported to the FBI's Internet Crime Complaint Center in 2022 rose almost 50% over the prior year, reaching $10.3 billion according to the recently released annual report (available here). 

This increase, which comes on top of a 64% surge from 2020 to 2021, has serious implications for companies and consumers who use the Internet, as well as for law enforcement and government.

Those implications are discussed in an article that I wrote over on LinkedIn in the hope that more people will pay attention to the increasingly dire state of Internet crime prevention and deterrence, and how that impacts people. In that article I also discuss the growing awareness that Internet crime creates even more harm than is reflected in the financial losses suffered by victims. There is mounting evidence—some of which I cite in the article—that the health and wellbeing of individuals hit by online fraud suffers considerably, even in cases of attempted fraud where no financial loss occurs. 

One UK study estimated the value of this damage at the equivalent of more than $4,000 per victim. Consider what happens if we round down the number of cases reported in the IC3/FBI annual summary for 2020 to 800,000, then assume that number reflects a fifth of the actual number of cases in which financial loss occurred. That's 4 million cases. Now assume those cases were one tenth of the attempted online crimes and multiply that 40 million by the $4,000 average hit to health and wellbeing estimated by researchers. The result is $160 billion, and that's just for one year; a huge amount of harm to individuals and society.