Automated AI Red Teaming : Elevating Copilot Prompt Injection to Azure Cloud Takeover 

Organizations are heavily adopting Microsoft Copilot Studio and Microsoft Foundry to build custom autonomous agents that do far more than just answer questions, they take actions like automating routine processes and enhancing productivity. 

 

By leveraging Retrieval Augmented Generation (RAG), these assistants can quickly pull highly sensitive corporate data directly from the Microsoft 365 ecosystem including SharePoint, Word and Excel. To achieve true autonomous capability, these agents are given access to advanced tools, such as linking with Power Automate to run workflows.

 

However, connecting AI agents to sensitive corporate data and powerful execution tools introduces new security risks. When agents can read internal documents, they become prime targets for prompt injection attacks that expose confidential data. 

 

The risk increases significantly when these agents possess "excessive agency" to execute tasks without human approval. If an attacker manipulates the chat interface, they can force the AI to abuse its connected tools, triggering unauthorized workflows against the organization’s cloud infrastructure.

 

To find and fix these blind spots, security teams must use Automated AI Red Teaming. In this workshop, you will use the Microsoft’s PyRIT framework to test and compromise a Copilot agent. You’ll learn to design automated single-turn and multi-turn attacks that push the AI beyond its safety guardrails. 

​

The session ends with practical steps to enforce strict tool boundaries and secure your enterprise AI ecosystem.