MIP against agent: Malicious Image Patches hijacking multimodal OS agents

Conference item

Recent advances in operating system (OS) agents have enabled vision-language models (VLMs) to directly control a user’s computer. Unlike conventional VLMs that passively output text, OS agents autonomously perform computer-based tasks in response to a single user prompt. OS agents do so by capturing, parsing, and analysing screenshots and executing low-level actions via application programming interfaces (APIs), such as mouse clicks and keyboard inputs. This direct interaction with the OS significantly raises the stakes, as failures or manipulations can have immediate and tangible consequences. In this work, we uncover a novel attack vector against these OS agents: Malicious Image Patches (MIPs), adversarially perturbed screen regions that, when captured by an OS agent, induce it to perform harmful actions by exploiting specific APIs. For instance, a MIP can be embedded in a desktop wallpaper or shared on social media to cause an OS agent to exfiltrate sensitive user data. We show that MIPs generalise across user prompts and screen configurations, and that they can hijack multiple OS agents even during the execution of benign instructions. These findings expose critical security vulnerabilities in OSagents that have to be carefully addressed before their widespread deployment.

Authors: Aichberger, L; Paren, A; Li, G; Torr, P; Gal, Y

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: NeurIPS
• Host title: Advances in Neural Information Processing Systems 38
• Pages: 18536-18575
• Publication date: 2026-02-02
• Acceptance date: 2025-09-19
• Event title: 39th Annual Conference on Neural Information Processing Systems (NeurIPS 2025)
• Event location: San Diego, CA, USA
• Event website: https://neurips.cc/Conferences/2025
• Event start date: 2025-12-02
• Event end date: 2025-12-07
• Language: English