In an era of rapid advancements in artificial intelligence (AI), organizations face heightened security risks. Such risks include data theft, reputational harm, and operational and financial loss stemming from adversarial abuse of AI, attacks on AI systems or misuse of AI by business users.
The Canadian Centre for Cyber Security (Cyber Centre) plays a critical role in safeguarding Canadian organizations from these threats. Our top AI security actions are designed to help organizations of all sizes and sectors strengthen their cyber resilience. These recommended AI security actions support and strengthen our existing Top IT security actions (ITSM.10.089), rather than replace them.
By adopting AI-specific security actions, organizations can build more resilient AI-enabled infrastructures and processes. This will minimize the likelihood and impact of AI-related intrusions, misuse and system compromise.
Our top AI security actions are organized into the following 3 pillars:
- Pillar 1: Protecting against adversarial use of AI
- Pillar 2: Protecting AI Systems
- Pillar 3: Protecting users and business processes
Given the speed at which AI solutions are being developed and adopted, we expect the risks and actions to evolve over the next 10 years. However, the 3 pillars are expected to remain the same for the foreseeable future, even as AI models and methods evolve.
Pillar 1: Protecting against adversarial use of AI
This pillar provides actions for your organization to enhance your ability to protect your environment from the adversarial use of AI.
Action 1: Implement prompt injection and jailbreak mitigations
- Sanitize inputs
- Isolate system prompts and protect prompt history
- Apply output filtering and policy gating
- Restrict high risk tools and agents via role based access and identity controls
- Validate downstream actions (files, code and tools) before execution
- Quarantine anomalous outputs
- Reduce access to private data by AI models
- Limit exposure to untrusted content
- Reduce the ability for AI systems to communicate externally (such as embedded data in markdown image URLs)
Why this action matters Action 1
Imagine that you’re chatting with a smart chatbot assistant, but a hacker finds a way to secretly slip in sneaky commands inside your questions. These commands trick the AI into doing things it shouldn’t do, such as running harmful computer commands.
This happened in 2025 with GitHub Copilot. Threat actors used a clever “prompt injection” to fool it into running dangerous code remotely. Microsoft quickly fixed it, showing us that spotting these hidden hacks early is key to keeping AI safe.Footnote 1
Action 2: Defend against deepfake and impersonation
- Deploy media authenticity checks and detection
- Enforce strong identity verification and phishing-resistant multi-factor authentication (including meeting PINs) for conferencing and messaging
- Train staff to verify unusual requests across channels
- Implement out of band verification for sensitive actions
- Implement abuse prevention for scams and social engineering at scale
- Monitor for AI generated phishing and voice or video spoofs
- Follow our Top 10 IT security actions (ITSM.10.089), including securing online accounts, to reduce hijacking and limit the impact of disinformation or false narratives
- Set the default for all identify signals (such as voice and video) as untrusted until verified by your organization
- Implement robust identity binding processes for higher risk business functions, such as finance, administrative privileges and access to sensitive information
- Enforce identity risk scoring to AI when it’s used to launch or accelerate specific actions to determine the likelihood of compromise
Why this action matters Action 2
In early 2024, a British design and engineering firm lost millions of dollars/pounds when a threat actor used AI-powered deepfake technology to impersonate the company’s CFO. During a video call, they fooled an employee in Hong Kong into transferring funds to fake accounts. The money quickly vanished offshore.
In May 2025, a separate incident highlighted how a global advertising giant narrowly avoided a similar scam. In this case, a threat actor used deepfake video and audio of the CEO and senior executives in a Microsoft Teams meeting to try to trick employees into leaking confidential information and making payments.Footnote 2
Action 3: Harden defences against AI powered cyber attacks and fraud
- Upgrade analytics for high volume, automated probing and credential stuffing
- Enforce rate limiting, bot detection and adaptive authentication
- Adopt zero trust principles and best practices
- Establish clear standards for data quality, code and documentation
- Regularly refactor models and code as systems evolve
- Ensure ongoing maintenance, automated testing and proactive code review
- Continuously monitor to mitigate “technical debt” and improve long-term AI system reliability and adaptability
- Train staff on AI threats and create tailored awareness programs which include:
- sharing intelligence reporting
- developing resilience by addressing known issues through employee upskilling and vulnerability disclosure
Why this action matters Action 3
Early in 2025, ransomware attacks exploded by almost 150%. A new trend has emerged where threat actors use AI to craft perfect phishing emails that appear to be from your boss or bank. AI-driven malware can shape-shift (called polymorphic malware), acting normally on your computer to avoid being spotted until it’s too late.
Ransom amounts have increased from hundreds of thousands to millions of dollars, while the level of effort for threat actors has gone down significantly.Footnote 3
Pillar 2: Protecting AI systems
This pillar provides actions for your organization to enhance your ability to protect AI systems.
Action 4: Conduct testing and red teaming of AI to identify modifications
- Regularly evaluate models, pipelines and interfaces against known attacks (such as evasion, prompt injection and poisoning)
- Update guardrails as exploits evolve
- Prioritize vetted, signed and well-maintained models based on risk assessments
- Apply timely patches and configuration updates
- Develop recovery plans to provide options for faulty deployments
- Implement disaster recovery and incident response plans for AI systems
Why this action matters Action 4
In July 2025, researchers found a critical data poisoning exploit in Microsoft 365 Copilot and similar retrieval-augmented generation (RAG) AI systems where threat actors injected poisoned documents to manipulate AI outputs persistently. This exploit showed the danger of insufficient continuous testing and updating of AI guardrails against evolving attacks.Footnote 4
Action 5: Safeguard against data poisoning
- Track data provenance
- Curate versioned datasets
- Harden training environments or use sandboxed architectures
- Run anomaly and bias detection on ingested data (including mirrored or manipulated news or content sources)
- Implement gate training and finetuning with approval workflows and model and data rollback plans
- Ensue rigorous identity and access control measures are implemented across all parts of the data and operations chain
Why this action matters Action 5
In 2024, security researchers working with Wiz and Hugging Face uncovered a risk in which malicious actors could upload poisoned data to Hugging Face’s dataset repositories. This vulnerability threatened AI pipelines of multiple organizations using their models and data. It exposed weak tracking of data sources and lack of anomaly detection on ingested data.Footnote 5
Action 6: Implement data usage controls and model theft prevention
- Enforce “no train” defaults and strict data sharing controls
- Implement vendor contractual clauses
- Log and audit all models and data access
- Monitor application programming interfaces (APIs) for extraction patterns (such as mass queries or label harvesting)
- Prevent sensitive data leakage with data loss prevention and secret scrubbing
Why this action matters Action 6
Threat actors carry out model extraction attacks by querying a machine learning model extensively and using the responses to train a replica with similar functionality. Implementing defences like API rate limiting, authentication, query monitoring, model watermarking, and legal protections can prevent unauthorized model replication and data leakage.
Action 7: Secure AI engineering and supply chain processes
- Implement and maintain an AI bill of materials (AIBOM)
- Reduce “technology debt” when AI systems are deployed or advanced very quickly and without secure deployment controls
- Ensure a strong foundational system, in accordance with Cyber security and privacy risk management series: A lifecycle approach (ITSP.10.033), is in place before layering AI systems on top of existing IT infrastructure
- Keep AI applications up to date
- Cryptographically sign models, code and artifacts or use hash-based validation as a start
- Maintain an AI specific software bill of materials (SBOM) for models, datasets, and dependencies
- Align with secure development frameworks (such as ITSP.10.033) and AI risk management standards including the:
- Implement change control, reviews and dependency scanning
Why this action matters Action 7
In 2024, the NullBulge group conducted high-profile supply chain attacks on AI-related open-source repositories hosted on GitHub and Hugging Face. By injecting malicious Python payloads into widely used AI tools and dependencies, they were able to exfiltrate sensitive data and deploy ransomware into downstream systems.
This incident underscores the dangers of technology debt when AI solutions are rapidly adopted without robust security controls. The incident also demonstrates the urgent need for foundational secure engineering, cryptographically signed artifacts, comprehensive SBOMs, and adherence to recognized security frameworks.Footnote 6
Pillar 3: Protecting users and business processes
This pillar provides actions for your organization to enhance your ability to protect users and business processes.
Action 8: Enforce data privacy, vendor and contractual controls
- Classify and minimize personally identifiable information (PII) in prompts
- Encrypt data in transit and at rest
- Apply data loss prevention, access controls and retention limits
- Require transparency, audit rights, liability and use restrictions (for intellectual property and copyright issues)
- Support the labelling of generative AI content to improve transparency
- Map and identify sanctioned and unsanctioned models operating on a network
- Implement policies, process and tools to govern the use of “shadow AI”
- Create an internal business policy for the acceptable use of AI tools
- Apply allow and deny lists for AI solutions
- Draft procurement clauses for vendors of AI-generated tools or solutions to protect against reputational harm
To meet regulatory expectations and manage vendor risk, contracts with AI vendors should include explicit data usage, privacy, audit, and liability clauses. These clauses should include:
- prohibiting unauthorized use of organizational data for model training
- ensuring encryption and access controls for all PII
- requiring vendor transparency and audit rights
- embedding usage restrictions and liability terms to protect against data leakage, unauthorized reuse, and reputational or legal harm
Why this action matters Action 8
Internal policies, including lists of approved AI tools and shadow AI controls, ensure AI tools are used safely and in compliance with privacy laws and contractual obligations.
Action 9: Ensure that human-in-the-loop oversight and execution controls are in place
- Embed human checkpoints in automated and multi‑agent workflows
- Provide explainability tools and auditable decision trails
- Implement escalation, triage, rate limits and kill‑switches or emergency shutdown procedures for high‑impact actions
Why this action matters Action 9
In 2025, an HR technology company faced reputational damage and legal scrutiny after removing human review from its AI-driven candidate screening process. The lack of human oversight allowed unchecked algorithmic bias to influence hiring decisions. In response, the organization reinstated human-in-the-loop controls, added execution checks, and established auditable decision trails to ensure all AI-driven outcomes are monitored and can be corrected by human reviewers.Footnote 7
Action 10: Maintain operational resilience against model drift, hallucinations, bias and overreliance
- Continuously monitor for drift and performance degradation
- Retrain or retire models that exceed validated bounds
- Continuously monitor for AI decision transparency to enable better understanding of model decisions and outputs
- Add truth checking and human review for critical outputs
- Test for bias and misuse
- Maintain fallback procedures and user training to avoid overreliance
- Require human review to prevent introduction of made-up components and containers for software code that could be appropriated by a threat actor
- Implement detection measures and retraining thresholds
Why this action matters Action 10
Financial regulators, including Canada’s Office of the Superintendent of Financial Institutions (OSFI), have identified AI model risk as a growing supervisory concern, particularly risks arising from data quality issues, model drift, lack of transparency, bias, and overreliance on automated outputs. OSFI has warned that poorly governed AI systems can produce unreliable or unexpected results, potentially leading to operational disruption, financial loss, legal exposure, and reputational harm if not actively monitored and controlled.
As AI models evolve over time and conditions change, maintaining operational resilience requires continuous performance monitoring, explainability, human oversight for critical decisions, and clear fallback procedures to ensure AI systems remain within validated and acceptable risk boundaries.Footnote 8