Understanding Malicious AI Technologies

This new guide from the OWASP® Foundation Agentic Security Initiative for developers, architects, security professionals, and platform engineers building or securing agentic AI applications, published Feb 17, 2025, provides a threat-model-based reference for understanding emerging agentic AI threats and their mitigations. Link: https://lnkd.in/gFVHb2BF * * * The OWASP Agentic AI Threat Model highlights 15 major threats in AI-driven agents and potential mitigations: 1️⃣ Memory Poisoning – Prevent unauthorized data manipulation via session isolation & anomaly detection. 2️⃣ Tool Misuse – Enforce strict tool access controls & execution monitoring to prevent unauthorized actions. 3️⃣ Privilege Compromise – Use granular permission controls & role validation to prevent privilege escalation. 4️⃣ Resource Overload – Implement rate limiting & adaptive scaling to mitigate system failures. 5️⃣ Cascading Hallucinations – Deploy multi-source validation & output monitoring to reduce misinformation spread. 6️⃣ Intent Breaking & Goal Manipulation – Use goal alignment audits & AI behavioral tracking to prevent agent deviation. 7️⃣ Misaligned & Deceptive Behaviors – Require human confirmation & deception detection for high-risk AI decisions. 8️⃣ Repudiation & Untraceability – Ensure cryptographic logging & real-time monitoring for accountability. 9️⃣ Identity Spoofing & Impersonation – Strengthen identity validation & trust boundaries to prevent fraud. 🔟 Overwhelming Human Oversight – Introduce adaptive AI-human interaction thresholds to prevent decision fatigue. 1️⃣1️⃣ Unexpected Code Execution (RCE) – Sandbox execution & monitor AI-generated scripts for unauthorized actions. 1️⃣2️⃣ Agent Communication Poisoning – Secure agent-to-agent interactions with cryptographic authentication. 1️⃣3️⃣ Rogue Agents in Multi-Agent Systems – Monitor for unauthorized agent activities & enforce policy constraints. 1️⃣4️⃣ Human Attacks on Multi-Agent Systems – Restrict agent delegation & enforce inter-agent authentication. 1️⃣5️⃣ Human Manipulation – Implement response validation & content filtering to detect manipulated AI outputs. * * * The Agentic Threats Taxonomy Navigator then provides a structured approach to identifying and assessing agentic AI security risks by leading though 6 questions: 1️⃣ Autonomy & Reasoning Risks – Does the AI autonomously decide steps to achieve goals? 2️⃣ Memory-Based Threats – Does the AI rely on stored memory for decision-making? 3️⃣ Tool & Execution Threats – Does the AI use tools, system commands, or external integrations? 4️⃣ Authentication & Spoofing Risks – Does AI require authentication for users, tools, or services? 5️⃣ Human-In-The-Loop (HITL) Exploits – Does AI require human engagement for decisions? 6️⃣ Multi-Agent System Risks – Does the AI system rely on multiple interacting agents?

New findings from OpenAI reinforce that attackers are actively leveraging GenAI. Palo Alto Networks Unit 42 has observed this firsthand: we've seen threat actors exploiting LLMs for ransomware negotiations, deepfakes in recruitment scams, internal reconnaissance and highly-tailored phishing campaigns. China and other nation-states in particular are accelerating their use of these tools, increasing the speed, scale, and efficacy of attacks. But, we’ve also seen this on the cybercriminal side. Our research uncovered vulnerabilities in LLMs, with one model failing to block 41% of malicious prompts. Unit 42 has jailbroken models with minimal effort, producing everything from malware and phishing lures to even instructions for creating a molotov cocktail. This underscores a critical risk: GenAI empowers attackers, and they are actively using it. Understanding how attackers will leverage AI to advance their attacks but also exploit AI implementations within organizations is crucial. AI adoption and innovation is occurring at breakneck speed and security can’t be ignored. Adapting your organization’s security strategy to address AI-powered attacks is essential.

The Rise of AI Malware: From Creeper to AI Creepy It’s 1971 All Over Again — But This Time, the OS Is the LLM. CVE-2025–32711 (EchoLeak) should be a wake-up call for anyone watching the Cyber for AI space. This isn’t theoretical — it’s real. Rated 9.3 (Critical) on the CVSS scale, EchoLeak is, to my knowledge, the first widely acknowledged, real-world, high-impact prompt injection vulnerability. In a nutshell, the exploit enables a remote attacker to exfiltrate confidential corporate data from Microsoft 365 Copilot, using prompt injection to manipulate how Copilot retrieves and processes internal content via RAG. TL;DR: AI meets real-world data breach! 🔥 Why This Attack Is a Turning Point Unlike previous LLM attacks that involved model poisoning or obscure behaviors (e.g., decompressing malicious Python files), EchoLeak (#CVE-2025–32711) is general, scalable, and dangerously accessible. Any document, email, or file retrievable by a RAG pipeline can be weaponized to issue hidden commands to the LLM. This isn’t a niche vulnerability — I truly think that the weaponization of data is a blueprint for LLM malware at scale. 🔐 What’s the Defense? Yes, an AI firewall (monitoring prompts and outputs) now table stakes. But just like with traditional malware, runtime analysis alone may not be fast enough or early enough to catch sophisticated exploits. Sound familiar again? At Symantec, scanning shared drives for malicious files was a very lucrative business. The same will now happen in AI-native environments: we’ll need “LLM-aware threat scanning” for corporate data — filtering and sanitizing not just inputs and outputs, but the entire enterprise knowledge graph. AI security vendors are already scanning RAG-connected data — for semantic tagging (DSPM), data access governance (DAG), and DLP enforcement (CASB). Startups like Daxa, Inc or Straiker, focused on AI application security, are also scanning corporate data before it enters the RAG index — though their focus is typically on governance and protection, not adversarial misuse. It’s time to broaden the mission — from just classifying and securing sensitive data…to detecting and neutralizing weaponized data. The enterprise knowledge graph is no longer just a source of truth — it’s now an active threat surface. Any data that flows into an LLM can carry malicious intent, just like a macro-enabled Word doc or a Base64-encoded payload in an old-school malware dropper. The next generation of AI security platforms can now evolve from “is this data sensitive?” to “is this data a threat to my AI?” Read the whole story here. https://lnkd.in/g4quUQt5    

'AI models, the subject of ongoing safety concerns about harmful and biased output, pose a risk beyond content emission. When wedded with tools that enable automated interaction with other systems, they can act on their own as malicious agents. Computer scientists affiliated with the University of Illinois Urbana-Champaign (UIUC) have demonstrated this by weaponizing several large language models (LLMs) to compromise vulnerable websites without human guidance. Prior research suggests LLMs can be used, despite safety controls, to assist [PDF] with the creation of malware. Researchers Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang went a step further and showed that LLM-powered agents – LLMs provisioned with tools for accessing APIs, automated web browsing, and feedback-based planning – can wander the web on their own and break into buggy web apps without oversight. They describe their findings in a paper titled, "LLM Agents can Autonomously Hack Websites." "In this work, we show that LLM agents can autonomously hack websites, performing complex tasks without prior knowledge of the vulnerability," the UIUC academics explain in their paper.' https://lnkd.in/gRheYjS5

Good report by OpenAI on malicious use cases of AI 1) Deceptive Employment Scheme: IT Workers was likely linked to North Korea (DPRK) and used AI to automate resume creation, research remote-work setups, and recruit contractors for fraudulent IT job applications.... 2) Covert IO: Operation “Sneer Review,” originating from China, bulk generated social media content, including comments and longer posts, across platforms like TikTok, X, and Reddit, focusing on China's geostrategic interests and creating a false impression of engagement.... 3) Covert IO: Operation “High Five,” linked to a Philippines marketing company, employed AI for political influence, including content analysis, bulk comment generation, and PR materials to promote President Marcos on TikTok and Facebook 4) Social engineering meets IO: Operation “VAGue Focus,” also likely from China, used AI to generate social media personas and translate messages for intelligence collection, posing as European/Turkish professionals and offering payment for sensitive information.... 5) Covert IO: Operation “Helgoland Bite” originated from Russia and produced German-language content for Telegram and X, criticizing the US and NATO, and supporting Germany's AfD party. 6) Cyber Operation: “ScopeCreep,” by a Russian-speaking actor, utilized AI to develop and refine Go-based Windows malware, debug code, and set up command-and-control infrastructure, distributing it via a trojanized gaming tool.... 7) Cyber Operations: Vixen and Keyhole Panda, linked to China's APT groups, used AI across diverse phases of cyber operations, including open-source research, scripting, software development, infrastructure setup, and researching US defense and military networks.... 8) Covert IO: Operation “Uncle Spam,” another China-origin influence operation, generated polarized social media content on US political discourse, like tariffs, on X and Bluesky, using AI to craft fictitious personas and extract personal data.... 9) Recidivist Influence Activity: STORM-2035, likely Iran-linked, generated short comments for X in English and Spanish, covering topics such as US immigration policy, Scottish independence, and Iran's military prowess.... 10) Scam: Operation “Wrong Number,” originating in Cambodia, was a "task" scam using AI to create recruitment messages offering high salaries for trivial tasks or investments, following a workflow designed to extract money from victims....