What are the security risks of AI?

What is AI?

Artificial Intelligence, or AI for short, refers to the intelligence of software and computers to resolve problems and make decisions by leveraging the advanced data processing prowess made possible by computing devices. AI functions are not unlike human intelligence, only at levels that go far beyond human capabilities.

How does AI benefit businesses?

The promise of AI to revolutionize business functions is vast and nearly infinite. While the extent of what’s capable is not known, some of the possibilities that are available at the onset are helping industries, like supply chain, healthcare and finance sectors – among many others – from developing processes for getting products from point to point seamlessly to processing volumes of health data to identify patterns and anomalies in diagnosis and provide better treatment to more intelligently detect fraud and block fraudulent transactions to keep financial assets safe…and that’s just the tip of the iceberg.

Some examples of AI

AI acts as an umbrella term that includes variations of the technology, each providing a benefit to business and society et al. Examples of the different types of AI technology are:

Machine Learning (ML): Machines are able to discover their own algorithms, or models by being fed data to ‘learn’ about the problem they are trying to solve. The more data points they learn from the greater the potential of the results. Initially, the learning process may require human labelers to identify correct results, as time goes on the human element will be less necessary as more accurate results are produced.

Large Language Model (LLM): Based on deep learning, which is a broader subset of ML, LLMs are pre-trained and rely on neural networks made up of tens of millions of parameters that process large volumes of data in parallel. Whether operating in self-supervised or semi-supervised learning modes, their aim is to not only obtain knowledge but embody contextual facets of knowledge, such as syntax, semantics, and ontology pertaining to humans, such as the way we think and communicate.

Generative AI: A technology that is capable of generating media, such as text and images, in response to prompts by learning the structures of input training data. By receiving input data from users and applying ML techniques by processing the data via neural networks, the resulting media is generated by AI and can be used in multiple applications, such as creating inspired works of art, developing code used in software design or writing documentation, likes articles and reports, complete with cited text – and so much more.

Security risks associated with AI

For all the talk of benefits to organizations around the globe, AI poses an equal and significant risk to each industry. And while cybersecurity risks are nothing new per se, the impact that AI currently has on risk and how that will evolve as AI continues to push into businesses in novel ways certainly is.

This is not just a belief held by a few or the plot of a blockbuster film detailing how AI’s rise will lead to the demise of humanity. In fact, the general consensus among the majority of cybersecurity professionals is that not only will AI be weaponized to a scale and speed that is far beyond what we understand and know today, but in a twist of irony, AI-enabled defenses will be necessary for organizations “to fight these advanced attacks with advanced tactics that detect, interpret, and respond to the threat before it has a chance to make an impact.”

And what exactly are the AI-based risks that organizations are facing to keep resources safe?

Thanks to OWASP and their Top 10 for Large Language Model Applications project for 2023, a comprehensive report dedicated “to educate developers, designers, architects, managers, and organizations about the potential security risks when deploying and managing Large Language Models (LLMs).” The listing includes:

• the most critical vulnerabilities impacting AI
  • highlighting their potential impact
  • ease of exploitation
  • prevalence in real-world applications

LLM01: Prompt injection

For those familiar with SQL Injection attacks, prompt injection vulnerabilities in AI are a similar attack type. Inputs are crafted in a way to manipulate the model to cause unintended actions. Direct injections are capable of overwriting system responses while indirect injection attacks seek to manipulate inputs received from external sources.

And just like SQL injection attacks, security strategies to mitigate this vulnerability involve the implementation of both input validation and data sanitization practices for user-provided data. Additionally, formatting output encoding helps to filter responses while further reducing the vulnerability of prompt manipulation.

LLM02: Insecure output handling

Attackers often employ fuzzing tactics to determine how to best attack software. By examining the output responses to specially crafted input, critical information may be exposed that provides threat actors a clue as to vulnerabilities that can be exploited to compromise systems. When LLM output is not scrutinized, exposure to the underlying system can occur through Server-side Request Forgery (SSRF) vulnerabilities. To minimize this and further exploits that could allow effective bypassing of access controls and unauthorized access to sensitive data, a combination of input validation and sanitization is necessary to mitigate threats initiated by malicious requests. Additionally, frequent review of auditing data is recommended to ensure that resources remain protected from AI.

LLM03: Training data poisoning

If training data is the lifeblood of AI’s deep learning process, then it stands to reason that AI-generated output is only as good as its input. This precept is especially significant when considering that vulnerabilities may be introduced that could easily compromise data security, integrity and efficacy. This is why it’s so important for organizations to ensure that training data is obtained from trusted sources and that its integrity is verified to ensure that training data has not been poisoned or tampered with, nor that bias has been introduced that could impact the ethical behaviors of AI systems.

LLM04: Model denial of service (DoS)

Not unlike DoS attacks on networks, LLMs represent a valuable target for threat actors. The resource-heavy operations, when attacked, can lead to service interruptions and increased costs which are only further complicated by the reliance on AI-based tools for everything from business operations to cybersecurity. When coupled with the level of variance that comes from user inputs, the number of variables only grows exponentially. Despite having their work cut out for them, security pros should implement resource caps in order to limit excessive requests that would otherwise deplete resources. When paired with continuously monitoring resource utilization and strict input limits, administrators can take a proactive approach to prevent resource exhaustion while still providing users access to AI tooling.

LLM05: Supply Chain

2022 was a year that saw not one but several high-profile supply chain breaches. So impactful were these breaches in fact that guidance from analysts for 2023 foretold that supply chain attacks would continue to grow and proliferate as threat actors continued to set their sights on this large, opportunity-rich target. According to OWASP, “supply-chain vulnerabilities in LLM can affect the entire application lifecycle” – including everything from libraries, containerized instances, images and packages. This extends to cloud service providers that may be hosting models and/or providing services that interface with your LLM, like plugins (but more about them later as they have their own dedicated vulnerabilities that we touch upon). Protecting your AI models from supply chain threats requires a layered approach to your security plan. For starters, thoroughly vetting partners is tantamount to setting up a solid foundation. Performing regular auditing of sources is a key part of the solution to ensure security remains a priority. Implementing model and code signing best practices work best when paired with only working with trusted sources. Of course, active monitoring is a must to detect any vulnerabilities, out-of-scope components being used when they shouldn’t be or even to spot anomalies that could pose a risk to your LLMs security. Lastly, a current inventory of components that are being used in conjunction with Machine Learning Operations (MLOps) to ensure that models are deployed and managed reliably, efficiently and securely.

LLM06: Sensitive information disclosure

Another familiar cybersecurity concern that poses an exponentially unknown risk factor to data security is data leakage. While this too is nothing new to the security industry, the ramifications of AI-based risk cannot be quantified. Information shared with AI technology can (and has) inadvertently revealed confidential data in responses to users, such as it did in three recent issues of leaking proprietary data belonging to Samsung. ML applications particularly learn from all input data and as it builds its database, can and will rely on this data to resolve a query, leading to possible unauthorized data access, compliance and/or privacy violations and of course, possibly lead to a data breach. Hence why it’s critical for users to know and understand the potential consequences of their actions by implementing user training to establish awareness of what should not be shared with AI and why it shouldn’t be shared. Additionally, organizations are well served by aligning user training to organizational policies to further support secure business practices.

LLM07: Insecure plugin design

Touched upon as part of the supply chain vulnerabilities, plugins and their design pose a critical risk to the data accessed and generated by AI due to the sheer nature of how plugins are designed to operate. In many cases, LLMs rely on plugins or APIs to work directly with input data and output data generated by AI models. Insecurely designed plugins may be prone to malicious requests that may result in but are not limited to data leakage, exposure of underlying systems or remote code execution. They may also lead to poisoning results, which will cause the model to generate output that has been compromised or provides sensitive system information that may be used to further an attacker’s aim. As a general precaution, it is advised that all input data be treated as unsafe and therefore, input validation (including parameterized input requirements) is recommended alongside explicit access controls to limit the risk of security issues. Additionally, plugins should be tested thoroughly to validate code and should adhere to best practices for developing secure code at each phase of the development pipeline.

LLM08: Excessive agency

The view and to some extent the marketing of AI, heralds thoughts of a personalized assistant that is always available to perform the “heavy lifting” for us, not unlike the JARVIS protocol used by Tony Stark/Iron Man to handle everything from curating playlists to performing scientific calculations on the fly when identifying an unknown element. And while AI certainly has been tapped to perform autonomous feats, like self-driving cars, the agency granted to the model (direct) or the automated actions that result from the data AI has processed and are executed by plugins or tools (indirect) all share a common trait: they are occurring without human input or authorization. This alone poses one of the more frightening concerns as LLMs or the plugins that rely on their data may perform functions that are not necessary or even intended to perform simply due to the agency or “permissions” given to them – even if the intended operation is one that humans wouldn’t want to be performed. Or as a core tenant of the European Union’s draft of the AI Act, “AI systems should be overseen by people, rather than by automation, to prevent harmful outcomes.”

How does one go about mitigating this risk type? Implementing a risk-based approach. Similar to a Zero Trust model, “LLMs should not be trusted to self-police or self-restrict.” To achieve this, look toward limiting access to plugins and tools to only the functions required. Also, avoid open-ended functions or any functions that are simply unnecessary to harden the attack surface (latter) while strengthening access controls to only interact with the data or perform the actions that are necessary to complete its process (former).

LLM09: Overreliance

If excessive agency is a frightening vulnerability, then overreliance is akin to it but from a more worrisome perspective. Let us explain. Many users have taken quite well to generative AI models, like ChatGPT, among others, to create content like writing articles, capturing captivating imagery or mashing up video content that is hyper-realistic, and yet all of it is completely produced by AI. While on the outset, the ability to generate media content is a capital feat in and of itself, as with many tools, the intent of the user is what drives whether it is used to build or to destroy. This may seem like overdramatization, but the risk posed by users relying on AI content as gospel truth could have disastrous consequences. Take for example the misinformation being generated in a technical paper due to a hallucination by the AI and how that could lead to any number of issues affecting major industries, such as healthcare and IT/IS. Or how relatively easy it is to produce audio recordings of individuals saying anything with only a few seconds' worth of soundbites needed to digitally recreate their voice. Now how about taking that recording and broadcasting it online? Depending on the content of the words, it could be enough to ruin someone’s public reputation or instead, the “faked” recording could be used as part of a crime.

Simply put: we just don’t know how deep that rabbit hole goes in relation to the untold consequences of over-relying on AI. But there are tactics that can help aid discernment between what’s real and what’s generated by LLMs. Let’s begin with fact-checking output with trusted external sources as an additional layer of validation to determine the accuracy and validity of generated content. Similar to plugin development, establishing and adhering to secure coding practices helps to minimize the risk of introducing vulnerabilities into the development environment. In addition to the validation mechanisms and cross-verification of information, clearly and concisely communicating risks, known issues and limitations associated with using AI and AI-generated content is table stakes to ethical and transparency efforts between content creators and content users – not unlike FCC laws that govern truth-in-advertising.

LLM10: Model theft

This vulnerability is among the most straightforward, referring to the unauthorized access and exfiltration of data, in this case, the LLM itself by threat actors. It’s not unlike the data exfiltration threats in cybersecurity seen for years prior to AI where sensitive, private and confidential data is targeted and removed from devices or networks with the express purpose of leaking the information, stealing proprietary details or as part of espionage campaigns. AI model theft, like any piece of confidential data that is stolen, can range in severity from both an economic and a business continuity standpoint. The loss may present a loss of revenue or competitive advantage to unauthorized usage of the model up to and including using it as part of an adversarial attack against the organization the model was stolen from. The key is to secure your LLM using layered security strategies including strong access controls, limiting access to network resources through network segmentation and secure sandboxing, active monitoring of resources, as well as regularly performing audits of logs and activities tied to your LLM. Incident response alerted and deployed upon alerts of suspicious behaviors and to mitigate the detection of anomalous behaviors. In addition to access controls, quick mitigation of other vulnerabilities known to affect LLMs (such as those represented within this article) can help to reduce the risk of malicious actors pivoting or moving laterally from another threat to compromise your model.

Other AI-based security risks

Inadequate sandboxing

Sandboxing data is an excellent way to segment sensitive processes from the rest of a system. Doing so allows data to be effectively processed while it runs securely isolated from the underlying system, including being inaccessible by external threats or exposed to risks outside the sandbox environment. Because of AI’s relative nascency, a number of issues are at the heart of designing a universally accepted or regulated sandbox. However, organizations that wish to take advantage of AI technology today would benefit from sandboxing AI models, tools and systems to promote experimentation with products and services in a secure and ethical manner that minimizes risk while addressing challenges, such as lack of formal safeguards, unforeseen consequences or lack of fidelity across solutions.

AI misalignment

The term AI alignment refers to “research aims to steer AI systems towards humans' intended goals, preferences, or ethical principles”, according to Wikipedia. If despite its competency, an AI system cannot advance the intended goals, then it is considered to be misaligned and its lack of alignment could lead to undesired behaviors, including actions and malfunctions that could further cause harm to businesses and worse still, impact human life. Consider for a moment an AI system used to generate code for a web service. While the aim of the developer is to create complex, secure code that will result in a service that can be used to simplify computer-related tasks, AI can also be subverted to generate powerful malicious code that may pose a threat to the web service mentioned previously or any web service for that matter. Hence why it’s critical to maintain a finger on the digital pulse of AI by identifying what works and refining what doesn’t to help make models safer to use. A key role in the alignment process is human oversight. Not just checking a box off when AI gets something right or wrong, but taking a more pragmatic and scientifically-based approach by documenting problems, performing continuous training, reviewing feedback, conducting evaluations of systems and doing so in a transparent fashion are just some of the key techniques to achieving better alignment.

Key takeaways:

• Develop input validation and output sanitization practices to reduce sensitive data leaks and prompt injection vulnerabilities
• Thoroughly vet supply chain partners to ensure compliance with security and ethical practices
• Ensure that training sets maintain data integrity and have not been tampered with or compromised by working only with trusted sources
• Audit all systems that are used for AI
• Impose limitations on data sharing, especially private and confidential information
• Implement data security and access controls according to industry best practices
• Harden hardware and software with up-to-date patches, vulnerability management and next-generation security tools (including AI/ML-based tooling)
• Provide adversarial training to respond to AI-based threats and improve the resiliency of models
• Integrate regular training so staff understand how to detect and avoid risks stemming from AI-generated threats
• Develop an incident response team for security issues detected and optimized for handling AI-related risks