Managing security risks in vibe coding (ITSM.60.043)

Sycophancy

One of the key limitations facing AI and its users today is that it is trained to agree with the user. AI models favour alignment and validation with their perception of what the user wants. This is called sycophancy and can be a serious cyber security risk. If the agent generates responses that reinforce your expectations, you may overlook inaccuracies or security risks.

In vibe coding, if the AI agent behaves in a sycophantic manner, it can jeopardize the robustness and security of the code generated and produce weak or unsecure code. Do not treat the AI’s agreement with the user’s decision as proof that the code is correct, robust or secure.

Understanding how to converse with an AI agent to avoid sycophantic behaviour is important. Ask neutral questions that don’t suggest a preferred answer. The agent will have a harder time judging where you stand on a topic and will likely avoid a sycophantic response. Stay up to date on the latest research and developments related to AI and provide thorough and ongoing training to your employees.

Weak code and code review

AI-generated code may not work as intended and can contain errors, vulnerabilities or poor design choices. AI tools produce code using patterns learned from Internet data and the context you provide, creating a framework for you to review and refine.

AI does not use critical thinking or understand risks and requirements the same way a human does. AI takes the most efficient way to build an application unless told otherwise. This can result in:

• incorrect results
• fragile or bloated code (larger, more complex code than necessary to accomplish its task)
• applications that cannot be scaled to meet operational requirements
• embedded security vulnerabilities
• unintended data leaks or disclosure of sensitive information

You may not understand why an AI tool has made a particular choice. With a person, you can ask why they chose a certain programming language or formatting style. You can discuss the process and learn why and how certain tasks were done. This can be hard to do with AI, but it is possible.

While AI lacks the human ability of critical thought and reflection, you can and should constantly challenge it. Ask the AI tool to explain its decisions or review the code it has written, but do not rely on the explanation or self-review as your only control, especially when related to important organizational tasks. You can prompt an AI model to review code it has generated, but a single model's self-review is a weak control on its own. Where appropriate, use a separate AI tool for review. Rigorous human-in-the-loop (HIL) or governance in-the-loop (GIL) oversight remains essential. In other words, qualified people must still review important decisions and approve the final code before deployment.

Human coders and reviewers also apply their experience, context and judgment, which an AI system cannot do. When a request is ambiguous or outside the AI tool’s training, the tool can produce:

• incorrect conclusions
• strange adaptations
• erroneous results
• hallucinations

When vibe coding, people with limited programming experience may not recognize complex defects in AI-generated code. A qualified computer programmer or reviewer should verify, correct and approve AI-generated code before deployment. This can help prevent reputational damage or financial costs to the organization.

Test your software security using static application security testing (SAST) and dynamic application security testing (DAST) solutions. These tools can check code for vulnerabilities before deployment. You should also program your AI to continuously run application security testing across the software development lifecycle.

Communicating effectively with AI

Prompt engineering is the practice of giving AI clear, effective instructions. Begin by identifying your organization’s key requirements, then address each one in a separate, specific prompt.

You can vibe code in many spoken languages, but response quality may vary by language and by the model’s training data, which is most commonly in English. Structure prompts so the AI can understand and act on them. Reviewing AI-generated code may also require familiarity with English programming terms, although the underlying logic and control flow are language-independent.

Tips for effective prompt engineering

• Provide enough project context to reduce guesswork based on pattern prediction
• Practise with sample prompts before starting the main project to understand how the AI responds
• Describe the expected output and reinforce it throughout development
• Focus each prompt on one main task
• Specify security, privacy and compliance requirements
• Use open-ended instructions for experimentation and brainstorming
• Use follow-up prompts to refine the output
• Provide examples of similar projects for the AI to reference

Supply chain integrity

Supply chain integrity depends on the AI vendor or provider and its dependencies, as well as the information you share with it. Review each provider’s security policies, data-handling terms and processes to make sure they align with those of your organization. It is important to run a trust-tier analysis on AI tool vendors to retain integrity. A trust-tier analysis allows you to manage and assign different levels of access and control for different AI systems and their tools. Some vendor criteria may include:

• where the AI service and data are hosted
  • ownership and legal responsibility
  • public or private
• the geopolitical context
• data use and retention policies
• the provider’s business and security practices
• product vulnerabilities, availability, and support

Vendors that are sanctioned by the government or hosted in-country may carry lower risk. Consult Supply chain integrity risk assessments: Evaluation criteria (ITSAP.10.071) for more information on evaluating risk profiles.

Digital and data sovereignty help strengthen supply chain security. Use a software bill of materials (SBOM) to identify and track open-source dependencies and where they came from. For more information on SBOMs and how to ensure a resilient supply chain framework, consult Joint guidance on minimum elements for a software bill of materials.

In addition to researching and performing proper risk assessments, you can take some of the following actions to protect your supply chain integrity:

• Scan and continuously verify AI-generated code for known common vulnerabilities and dependencies
• Classify threats as critical, high, medium or low risk
• Apply the principles of least privilege and zero trust
• Continuously monitor vendors and dependencies
• Use minimal, hardened container images
• Perform periodic security audits and penetration testing
• Test incident response procedures by simulating supply chain attacks to ensure teams can detect and isolate threats quickly
• Use network detection and response systems and behavioural analytics to identify unusual activity

Code provenance

It can be difficult to determine the origin, or provenance, of AI-generated code. When AI generates code, it draws on patterns learned from training data. That training data can include open-source software, vulnerabilities, outdated practices and intentionally malicious code. Similar patterns may appear in generated output, increasing the threat landscape. Anyone or anything can create the source material that the AI draws from, so do not assume that generated code is secure or robust.

Code generated from unknown provenance can:

• inherit vulnerabilities from its source material
• create licensing, copyright or attribution obligations
• vary in quality
• contain code that has been intentionally modified by threat actors

You can prompt the AI model to review and assess its own code or use a separate AI tool to check for vulnerabilities. Many vulnerabilities can go undetected, and using an AI tool for review may not find vulnerabilities the AI has intentionally inserted. You must have HIL or GIL oversight to confirm your code contains no malicious packages. For more information on how to stay safe while using open-source software see Security considerations when using open-source software (ITSAP.10.059).

Information leaks and architecture

Before vibe coding, organizations should consider information security and robustness. Giving AI access to sensitive information can result in a data leak, either through the resulting code or back through the AI itself. Know where the AI service provider stores information and how the data you input may be used. Depending on the AI service you use, the provider may retain your information or use it to train their models. Review the provider's data-handling terms before entering any organizational information. Select providers whose terms of service or enterprise agreements contractually prohibit training on customer input.

You should also consider the security of code architecture. When a programmer creates code manually, they set up the structure and security of the architecture to their desired parameters, including the permitted information. However, when AI creates code, it generates what it thinks is best for your project but may not understand all of the cyber security pitfalls it encodes. If you do not validate and verify the security of AI-produced code, you can expose your organization to unauthorized access and cyber attacks.

Organizations including, but not limited to, Government of Canada departments and Crown corporations must also seriously consider data residency and cross-border data transfer risks. This goes hand-in-hand with using SBOMs and evaluating your supply chain risk framework. For more information about digital Canadian data, assets and storage requirements, consult Digital sovereignty.

Embedded malware and access in libraries

AI-generated code may use open-source packages or libraries, which could contain vulnerable or outdated components. This can expose your organization to risks such as:

• malicious packages, malware or backdoors
• unknown vulnerabilities
• insecure configurations that give threat actors access to your networks or systems

Maintaining organizational cyber security

Regular updating and patching are important aspects of maintaining organizational cyber security. With cyber threat actors' growing use of agentic and frontier AI, timely maintenance and vulnerability management remains essential.

Consider the following maintenance challenges:

• difficulty understanding and troubleshooting AI-generated code
  • iterative prompts can make it difficult to determine why AI-generated code does not work as expected
  • repeated or revised prompts can change, affect how it works and make it difficult to trace provenance
  • a request to fix one issue, such as a code error or logic flaw, can introduce unnecessary complexity
• need to rebuild vibe-coded software from scratch or create several sub-versions
• the amount of code generated by AI to perform a task can become bloated, which can make it difficult to maintain as it grows

Vibe coding can help people with limited programming experience create code, but AI tools alone cannot maintain a functional code base. Skilled programmers must provide ongoing human oversight.

Human reviewers are essential to code review and maintenance, but they can still make mistakes. Organizations should account for the following human factors when designing their oversight processes.

Automation bias

Automation bias is the tendency to have more confidence in automation systems than your own personal judgment and experience. Reviewers may still accept AI suggestions even if they are incorrect, due to the belief that technology has a better analytical process. A lack of critical human scrutiny can affect the quality and security of your product.

Oversight fatigue

Oversight fatigue can occur when reviewers become exhausted or overwhelmed by a large volume of alerts and threats or by carrying out repetitive tasks. The amount of code generated in vibe coding and the resulting oversight and decision-making tasks can cause oversight fatigue.

Depending on the size of your operation and your project requirements, you may consider various in-house mitigations. Ensure your employees are informed, well-trained and confident in their jobs. Organizations should also develop strategies to manage cognitive load and encourage critical analysis for anyone working in vibe coding.

Common attacks and vulnerabilities

The following section provides details on the most common attacks and vulnerabilities related to vibe coding.

Slopsquatting

Slopsquatting is a software supply chain attack that exploits AI-hallucinated software dependencies. Threat actors register fictitious package names commonly generated by AI coding tools and wait for developers to install them. Unlike typosquatting, which relies on human error, slopsquatting leverages AI-generated errors and misplaced trust in AI-produced code. To mitigate this risk, you should:

• prohibit AI agents from installing packages autonomously
• require an allow list or human review gate
• pin dependency versions and verify package hashes in your build pipeline
• use SBOMs

Prompt injection or manipulation

Prompt injection is a technique that takes advantage of the AI’s inability to distinguish between legitimate and malicious requests. A threat actor can disguise malicious prompts and manipulate the AI to ignore regular instructions or extract sensitive data. Because vibe coding uses natural language to code, an attacker can craft an input to look like a developer’s prompt and override the established system. To mitigate this threat, you should:

• define sensitive topics and implement separate input and output testing
• require regular monitoring and HIL or GIL approval for all high-risk and high-impact requests
• require regular updating and patching
• use a sandboxed environment (a controlled area where the AI-generated code can run with restricted access to systems and data) for the agent and its tools to contain and minimize damage

Cross-site scripting (XSS)

XSS occurs when a threat actor injects malicious input or scripts into a website, causing a user’s browser to execute unauthorized commands. This vulnerability can arise from insecure AI-generated code, including predictable patterns and security gaps. Code that does not validate and authenticate user input is vulnerable to XSS. To mitigate XSS, you should:

• validate and sanitize all untrusted input, and apply context-aware output encoding
• use output encoding and HTML sanitization in your development framework
• use content security policies (CSPs) to restrict the source of content permitted to load
• use a Web Application Firewall (WAF) to monitor and filter HTTP traffic

SQL injection

SQL injection is an attack that targets the Structured Query Language (SQL) used to manage data in databases. Vibe coding often generates the most direct, readable query. Threat actors exploit this pattern, inserting their own SQL query to steal or modify data, execute administrator operations or spoof a client’s identity. To mitigate this vulnerability, you should:

• use secure database queries and prepared statements that keep user input separate from database commands
• be explicit about your database security needs with the AI agent during the production phase, such as requiring secure database interaction