Hardening up

How does advanced AI impact cybersecurity?

We are at war. There is currently a constant struggle between countless opposing forces to overcome the other’s defences. But this fighting is not immediately noticeable in the world around us — it is going on in the abstract world of digital communications called cyberspace. As more and more of our activities are impacted by what goes on in this domain of warfare, it becomes more and more of a target. The recent developments in artificial intelligence (ai) — and more specifically large language model (llm) — capabilities have had large headline impacts on the field of cybersecurity. Regardless of if Anthriopic’s Mythos has the capability to autonomously gain access to complex digital systems, it is inevitable that some other model soon will. Frontier labs have all invested enormous resources into training these models for cybersecurity tasks. What will the consequences be of such automated systems, and how will they impact the fundamentals of cybersecurity?

Cyberspace is defined in nato’s joint doctrine for cyberspace operations as an entirely man-made domain mostly consisting of computerised systems. The domain consist of three principal layers - physical, logical and cyberpersona. The later mentioned domain is inherently dependent on the previous one, and have fundamentally different properties.

The physical layer consist of hardware, such as servers, network hardware, laptops, mobile phones and earbuds; and the cable or antennas connecting them. All of this hardware runs software that allows the phone to be something other than a paperweight. While an Ethernet cable is part of the physical layer, and can be attacked by being plugged out or cut, the electromagnetic potential in the cable orchestrated in “Manchester coding” is a part of the logical layer. An attack on the physical layer is executed in a domain other than cyberspace, and consist of physical destruction or other impedance on the hardware.

The logical layer consist of data executed as code - software.¹ This includes low-level hard-coded microcode (software controlling how the hardware functions), operating systems, and high-level abstract web services. It also includes other data components such as protocols. An attack on the logical layer consist of malicious abuse of the functionality of the software. This may be done by exploiting a well known quirk of the software, such as pass-the-hash in windows environments; a bug in the software, such as EternalBlue; or a bug in the protocol the software is orchestrating, such as in the poodle vulnerability. It can also be done through supply chain attacks like in the famous case of xz Utils.

The cyberpersona layer consist of virtual identities, such as user accounts, email addresses and aliases. Cyberpersonas have a limited correlation to the real world; while the account may appear to be a single person, in reality there may be a group or entire company behind it. An attack on the cyberpersona layer may be phishing or impersonation.

Attacks in cyberspace can be modelled using the Mitre Att&Ck framework, a framework used both to explain and reason about cyberattacks. The model divides an attack into 14 steps, each step containing multiple techniques. A technique can be executed on the physical layer, such as t1200 “hardware additions”; executed on the cyberpersona layer, such as t1598 “phishing for information”; although most are executed on the logical layer. Att&ck is particularly good framework when modeling an attack on a larger system containing multiple steps, but is limiting for discussing individual steps. The framework explains that each attack starts of with recognisance, where the attacker gathers information, before it gains a foothold in the system, moves through the system until it achieves the goal - such as usage of exfiltrated data or ransomware payment.

In order to discuss secure software, one must define the threat profile of data. A popular model for data security is the cia triangle: confidentiality - only authorized profiles shall be able to read the data, integrity - only authorized profiles shall be able to change the data, and availability - profiles shall be able to perform the actions they are authorized to perform. Those are the principles that needs to be unquestionably fulfilled in order to constitute complete data security. An example of software that has proven its compatibility with these principles is the sel4 micro kernel. It has done so through the process of “formal verification”, where the code is mathematically proven to cohere to the specification and does nothing more, and the specifications is proven to cohere to a set of requirements - including conformance to the cia triangle. The proofs have underlying assumptions and run on hardware that may or may not be formally proven.

The issue with formal verification is one of scale and expense, making it impractical for anything more than small exceedingly important software in the current state of software development. A middle ground is to formalise a proof only for the parts most exposed to security risks, such as core protocols, but not the software running the protocols. This is the case with, for example, tls v1.2 (the security protocol of the web and more) and wireguard (a vpn). The software implementation could (since it is not formally verified) still be vulnurable, as in the case of the Heartbleed vulnerability in Openssl in 2014, but fixing it merely requires a software update.

While the cia triangle defines the principles that needs to be protected in order to achieve data security, they are impossible to achieve in a system where humans need to interact with data. This is because man’s impersonation of a cyberpersona can always be replicated. This risk can be protected against through authentication,² but never mitigated. Ignoring the presence of man, the complete fulfillment of the cia triangle may be possible in an isolated system where the physical layer is assumed secure and correct, and the simpler ci (not including A) in a connected system. Writing software without bugs is difficult and near impossible in the scale of a complex system, making complete fulfillment of the cia principles unreasonable even without the influence of man on the final system.

Designing a system that completely fulfills the cia principles is not necessary. One should instead try to design a system that protects against most attackers, as according to you “threat profile” (the adversaries you expect to be defending against), and that detects a sophisticated attacker early enough and slow the attacker down enough to prevent them from doing significant harm before the incident response team has dealt with the threat. It is for this reason that the cyber defence doctrinal preview of the Swedish armed forces differentiates the roles of cyber protection and defensive cyber operations in the cyber defence. “Protective measures” are passive measures that should prevent even a seasoned hacker from doing any harm while defensive cyber operations are active measures to identify and/or respond against an adversary. This, together with offensive cyber operations, constitutes the Swedish model of cyber defence.

The Swedish cyber doctrine explains that a significant part of the protective measures are taken in the early design stage. The system should, of course, try to keep an attacker completely out of the system, but a sophisticated enough attacker with enough time will always be able to gain a foothold in the system. The system should therefore try to restrict the attacker all the way to the exfiltration step of mitre att&ck (the second to last step), in order to slow the attacker down and increase the chance of them making a mistake, increasing the risk of detection and time to execute a response. This should be done by limiting the attackers available options as far as possible by running a hardened operating system that is as restricted as possible; keeping the system simple; logging as much as possible into a central searchable database to allow for efficient incident response and early (any) detection; having effective means for the incident response team to execute the response; having active recognisance tools such as anti virus or endpoint/intrusion detection and response tools configured; and following a security framework, such as nist cyber security framework or the swedish armed forces krav på säkerhetsfunktioner (ksf),

A useful metric of the available options for the attacker, particularly the ones unknown to the defender, is the “attack surface” — the total size (in complexity, amount of code or similar metric) of an system exposed to a potential attacker. There is a roughly proportional trend of the number of software bugs per the size of the codebase. Reducing the size of the attack surface is therefore critical since a reduced amount of exposed code correlates with less, potentially harmful, bugs. The size of the attack surface increases as the attacker moves laterally through the system during the attack. The earliest, and therefor most critical, attack surface for most systems is the connection between the system and open internet. It is therefore inherent that one should limit the number of processes available from the internet, through the use of a firewall, and take measures to secure those few processes left exposed, in order to reduce the attack surface. Many organisations make use of a virtual private network (vpn) to minimise critical attack surfaces to one single, highly trusted, piece of software specifically made for this purpose. Only through the vpn can you access the organisation’s internal systems, limiting the internal systems to (1) insider attacks, (2) attackers with a compromised authentication method, or (3) attackers that have compromised the vpn software itself.

Option 3 would only be available to a competent Advanced Persistent Threat (apt),³ since any other actor would prefer disclosing the vulnerability responsibly and receive a bug-bounty (As long as one uses one of the three popular vpn technologies: wireguard, openvpn, ipsec). The usage of a compromised certificate is a more realistic, and commonly abused, method of access. This vector is difficult to protect against and can only be done so on the users device, in combination with the organisations culture and certificate rotation. Since there is a human operating the computer with the certificate, this threat vector can never be fully mitigated; an attacker can always steal the laptop or login while the owner is not looking, impersonating the trusted cyberpersona. The threat of an insider is often overlooked while being very difficult to protect against. This is not a breach against the cia principles, since the individual is supposed to have the vpn credential. The protective mitigations against this attack must be made at the systems inside the secure environment through the use of access control (only giving users and groups access to what they need to access) and logging (with corresponding analysis). The system should limit the attack surface even for insiders, for example by limiting access too systems administration behind a vpn, and making sure the administrator can’t change the logs.

It is obvious, with these techniques, abstractions and fundamentals; that the goal cyber security experts strive for have always been to make software logically secure. While it is impossible, as argued earlier, to design a formally proven, secure complex system connected to the internet; there are methods to get close enough for any organisation. The introduction of ai capable of orchestrating entire exploit chains for a complex system, while extraordinary for a machine, does not reinvent the methodology of cyber security. Auguste Kerckhoffs (1835) stated what would later be known as Kerckhoffs’ principle. Originally meant for military ciphers, the principle states: “[the cipher] should not require secrecy, and it should not be a problem if it falls into enemy hands”. This principle have followed the cyber security profession to this day, where security by obscurity (the opposite of Kerckoff’s principle) are frowned upon in favor of logical security. Since ai and man, in most part⁴, are governed by the same laws while in cyberspace, they will face the same logical limitations in the act of exploiting systems.

Anthropic’s recently announced model Mythos came with staggering claims of cybersecurity capabilities. While ai and man face the same laws in cyberspace, a sufficiently competent language model has the ability to quickly and at scale discover (and exploit) vulnerabilities in software. Since logical security has been the goal for 150 years, the discoveries Mythos have made and will make should be celebrated and lead to a more secure cyberspace. With the proposed ease Anthropic claims Mythos found these bugs, one would be naïve to think there is not a huge number of similar ones waiting to be found. While Mythos remain closed to a specific few in order to find these vulnerabilities in the most critical code (although access to it has already been leaked), it is only a matter of time until an open model with equivalent capabilities is released — as also hinted to by The Economist. At that time only select codebases will have been sufficiently audited, and even fewer had their software updated. This is especially true in industries like critical infrastructure or airlines, where programmable logic controllers (plcs) are rarely updated.

In order to defend a system against the future’s ai enhanced threat actor, the systems administrator and security operation center needs to identify the golden nuggets in the system, that is the specific part of the system that must not be compromised (such as confidential data or a particular service), and make sure it is sufficiently protected and that there are means to detect the adversary and actively defend the nugget. They must make sure the entire system is sufficiently secured through software updates and software inventories that keep track that the software is actively maintained and all potential vulnerabilities are accounted for; and make sure the software runs with restricted permissions. They should make sure to have regular training and drills, testing (and improving) the teams ability to identify and respond against an attacker, and have the system tested through a penetration test and/or red-team operation. They should secure the system the same way as before Mythos: through competent systems designers, that are sufficiently funded and have enough man power, aided by a security framework like nist or ksf, with sufficient accountability and outside vetting. Mythos does not change the equation, although it may increase the importance of these practices and further force institutions to dismantle inefficient processes. ai may make implementation of these tools, some of the daily maintenance, the threat hunting and incident response, easier. The entire creation and maintenance of the system may even be automated; but the requirements that keep them safe will not disappear.

The process of developing software has seen significant changes since the advent of llms, through the creation of “agentic”- or “vibe”-coding; where the developer prompt an llm how the software should be and the llm alone writes the code. This method of software development changes a part of the secure software development practice. Since the developer is significantly decoupled from the code, the practice of following sane coding principles becomes difficult to follow. Vibe coded software seems to generally result in larger, more complex, codebases, and generally seem to add more features; increasing the attack surface. Some protection against these problems may be obtained by giving the ai a guideline prompt, encouraging clean coding practices, and have strict coding guidelines in the project. Since this type of software development seem to write code very quickly, with the developer less adept to how the software works in large and small detail, compared to traditionally developed software; there may be a higher risk of bugs in the final product since less thought has been put into how it works . While llms today make some code cheaper to produce, they also make automated testing cheaper to produce. It can even act as a basic autonomous tester and be a great tool in a human orchestrated test. While vibe-coding may in general produce code that traditionally would contain significant amount of bugs, vulnerabilities, and increased attack surface; the shrinking cost of automated testing, particularly vibe- penetration testing, could result in more secure software.

A technique that works for both software and complete systems is that of bug bounties — the practice of which has changed dramatically as a consequence of llms. A bug bounty program is where hackers get paid to to ethically disclose a bug, most often security-related ones, to the developer - in the process incentivizing the ethical hackers to search for vulnerabilities. These programs make it possible for firms or individuals to sustain themselves entirety on the ethical testing and reporting of the world’s software.

Daniel Stenberg, a programmer from Huddinge and the lead developer of libcurl (curl) (a tool that underpins much of global internet communication) prominently announced the end curl’s bug bounty program back in February due to a flood of low-quality bug reports that were impossible for humans to filter through. But Stenberg has recently written that automated bug reports by ai models no longer merely produce “slop”, but has lead to an increase in the share of quality reports at the same time as the volume has increased. Even without a bug bounty as an incentive, it is in the world’s interest to make sure that the level of software security remains high, and llms are now one such tool that help make that possible.

In military history, some of the biggest advancements ever occurred during the first world war. An officer from before the war, in 1914, would have immense trouble understanding the strategic choices being done in 1918. Meanwhile, an officer from after the war would have no trouble at all understanding the dynamics of the contemporary war in Ukraine — including the consequences of aircraft, electronic warfare, drones, or the threat of nuclear weapons. This is an example of the enormous transformational changes that can take place in war. Mythos is not that. It is a significant incremental improvement, but it does not change the qualitative nature of cyber operations.

The different areas that llms are useful in provide yet another example of the “jagged” frontier of ai capabilities. It speeds up and automates certain tasks while in many other areas not being particularly impactful. In many ways llms form the reverse threat to cybersecurity as quantum computers do; instead of destroying existing paradigms and requiring global changes in infrastructure ai simply shines a light on the preëxisting flaws of our designs and allows them to be exploited at scale — creating a sort of “second wave” of cybersecurity exposure similar to the one that the initial expansion of the internet created. Thankfully we are more knowledgeable and prepared this time, although there is also vastly more at stake.