AIs are hackers, for better or for worse

Artificial intelligence will eventually find vulnerabilities in all manner of social, economic, and political systems and then exploit them with unprecedented speed, scale, and scope, writes Bruce Schneier in a probing 54-page report:

“After hacking into humanity, AI systems will then hack into other AI systems, and humans will be little more than collateral damage.

Most of these hacks don't even require major breakthroughs in AI research. They are already happening. However, as AI becomes more sophisticated, we often won't even know this is happening. »

The summary he published on Wired and his blog is only a substitute that does little to testify to the richness of the essay published by the Council on the Responsible Use of Artificial Intelligence at the Belfer Center for Science and International Affairs of the Harvard Kennedy School. So we synthesized it.

Schneier begins by recalling, in the preamble, that if hacking is generally considered as an act targeting computer systems, this conceptualization can be extended to "any system of rules", such as the tax code and the financial markets, for example.

The essay therefore envisions a world where AI could discover vulnerabilities in all of our social, economic and political systems, but also be able to exploit them "at the speed and scale of a computer". He postulates no “technological singularity,” the assumption that the AI ​​learning feedback loop becomes so fast that it overwhelms human comprehension and induces unpredictable changes.

Nor do his scenarios start from the premise that the AIs, nor their authors or operators, would be ill-intentioned. Some of the hacks he mentions don't even require major breakthroughs in research.

On the other hand, he believes that they will improve as AI techniques become more sophisticated and that this evolution will logically occur, as they progress in terms of learning, understanding and solving. of problems.

MacGyver was a hacker

Schneier recalls that the late hacker and author Jude Milhon (St. Jude), a pioneer of the cypherpunk movement, believed that "hacking is a clever circumvention of imposed limits, whether these limits are imposed by your government, your own personality or the laws of physics”, and therefore comes back to the definition of the term “hack”:

1. An intelligent and fortuitous exploitation of a system that: a) subverts the rules or norms of that system, b) at the expense of another part of that system.
2. Something a system allows, but not anticipated by its designers.

Hacking would therefore not fall within the purview of "cheating" without faith or law, but follows rules, and subverts their intention. It is an exploitation, consisting in “playing with the system”. Or, put another way, "MacGyver was a hacker." Systems are optimized for specific results, so hacking is never more than the pursuit of another result, “often to the detriment of the original optimization”.

But systems tend to be rigid, limit what we can do, and invariably some of us want to be able to do something else with them: “So we hack. Not everyone, of course. Not everyone is a hacker. But there are enough of us."

If the tax code is not software, and if it does not require a computer, nothing prevents it from being considered a "code" in the computing sense of the term, namely a series of algorithms which takes an input – financial information for the year – and produces an output: the amount of tax due. "It's deterministic, or at least supposed to be," notes Schneier.

However, all software contains defects, or bugs, whether they are errors in specification, programming, or even implementation. For once, "the numbers may surprise, but modern software applications usually have hundreds or even thousands of bugs", without this preventing them from working or necessarily being detected.

But they are there, introducing security flaws, or vulnerabilities, that an attacker could exploit to subvert the intended functionality.

Taxes have a code, so they have bugs