AI Security & Safety
Security is what you defend when someone pushes on the model or the agent. Safety is what you own when the system hurts people with no attacker in the loop. Same CSA spine. Different question.
Ethical requirements it triggers
Control domains that hold the fix
3 harm classes · 7 safety control families · NIST AI 100-1
Harm to People: Harms to individuals and groups — rights, safety, access. The class with the widest ethical-requirement pipeline.
The security half
Start with the OWASP Top 10 for LLM applications (mapped to CSA AICM domains — our judgment, not OWASP’s). Then walk the threats in plain English. The curriculum below is the AI Security Engineer’s primary control set.
OWASP LLM Top 10
OWASP Top 10 for Large Language Model Applications · v2 (2025) ↗Each risk maps to the CSA AI Controls Matrix (AICM) domains where the controls live. That pairing is our judgment, not OWASP’s.
Prompt Injection
Adversarial input that manipulates the model into ignoring its instructions or exposing data. The single most consequential class of LLM attacks today.
Controls live in Model Security · Application & Interface Security
Sensitive Information Disclosure
Models leaking PII, secrets, or proprietary data through outputs — directly or via inference from prompts.
Controls live in Data Security and Privacy Lifecycle Management · Model Security
Supply Chain
Risks from pre-trained models, datasets, and plugins sourced outside the org — the AI version of dependency risk.
Controls live in Supply Chain Management, Transparency, and Accountability · Change Control and Configuration Management
Data and Model Poisoning
Compromised training/fine-tuning data or model artifacts that introduce bias, backdoors, or degraded performance.
Controls live in Model Security · Data Security and Privacy Lifecycle Management
Improper Output Handling
Downstream systems trusting LLM output as code, SQL, shell, or URL without validation — classic injection patterns reborn.
Controls live in Application & Interface Security · Model Security
Excessive Agency
Granting an LLM/agent tool access or permissions far beyond what its task requires. The cousin of over-privileged service accounts.
Controls live in Identity & Access Management · Model Security
System Prompt Leakage
Internal system prompts becoming externally visible — leaking implementation logic and bypassing guardrails.
Controls live in Model Security · Application & Interface Security
Vector and Embedding Weaknesses
Misuse of embeddings/vector stores: information leakage across tenants, embedding inversion attacks, retrieval poisoning.
Controls live in Data Security and Privacy Lifecycle Management · Model Security
Misinformation
Plausible but false output presented as authoritative — hallucinations operationalized into systems of record.
Controls live in Model Security · Governance, Risk and Compliance
Unbounded Consumption
Resource exhaustion, model denial-of-service, runaway cost. Where AI security meets FinOps.
Controls live in Model Security · Threat & Vulnerability Management
MITRE ATLAS shows up as technique cites on the threat map and on skills, not as a second scored crosswalk. We do not invent an ATLAS×AICM matrix here.
Your curriculum · 26 controls
5 skill groups on the CSA controls this role owns primarily: secure the model, attack it first, contain the agent, guard the runtime, defend the training data.
Secure the model
8 controls
Attack it before they do
5 controls
Contain the agent
5 controls
Guard the runtime
4 controls
Defend the training data
4 controls
Skill self-check · nothing saved
Preview the prompts this seat owns, then score them on four concrete rungs. Self-assessed, nothing saved.
The safety half
Look up any AI harm and see who owns it, the controls that hold the fix, and the EU AI Act risk tier it lands at. Then learn the three skills the map keeps asking for.
The three skills this map keeps asking for
Evaluate how a model behaves
You tested code against a spec. A model has no spec, only behaviour: the skill is building the evaluation that catches what it does under pressure, before your users do.
Fit a guardrail, and prove it
An off-the-shelf guard is a claim. The skill is refitting it to your own product and measuring what it catches before and after, on your data.
Trace a harm to its owner
When the system hurts someone, who answers for it? The skill is reading a harm back to the accountable owner, the control that holds the fix, and the regulatory tier it lands at.
The harm classes · NIST AI 100-1
The National Institute of Standards and Technology (NIST) AI Risk Management Framework groups AI harms into three top-level classes. We anchor the safety lens to those classes so the controls we fit map cleanly to ethical requirements, AI Controls Matrix (AICM) domains, and the EU AI Act risk tier each class typically lands at.
Harm to People
Harms experienced by individuals, groups, communities, or society — civil liberties, rights, physical and psychological safety, economic opportunity, equal access. Includes group harms (discrimination, exclusion) and societal harms (erosion of democratic processes, harm to vulnerable populations).
Domains: Model Security · Data Security and Privacy Lifecycle Management · Governance, Risk and Compliance · Human Resources
EU AI Act tier: High risk
Harm to an Organization
Harms to an organization — operational disruption, reputational damage, security and financial loss — caused by AI systems behaving unexpectedly, leaking data, or being weaponised against the organization itself.
Domains: Model Security · Application & Interface Security · Logging and Monitoring · Identity & Access Management · Security Incident Management, E-Discovery, & Cloud Forensics
EU AI Act tier: Limited risk
Harm to an Ecosystem
Harms to the interconnected systems on which society depends — the global financial system, supply chains, critical infrastructure, the natural environment. Includes Environmental harm: AI training and inference are energy-intensive, and at scale they contribute to carbon and water footprint, e-waste, and resource depletion.
Domains: Supply Chain Management, Transparency, and Accountability · Business Continuity Management and Operational Resilience · Change Control and Configuration Management · Threat & Vulnerability Management
EU AI Act tier: High risk
| Harm class | EU AI Act tier | Primary requirements |
|---|---|---|
| Harm to People | High risk | Fairness & bias testing · Human oversight · Transparency & explainability · Contestability |
| Harm to an Organization | Limited risk | Human oversight · Transparency & explainability |
| Harm to an Ecosystem | High risk | Human oversight · Transparency & explainability |
The safety control spine
A lens over the existing 18 AICM domains, not a new partition. Each family names where in the control stack a harm class is mitigated, the AICM domain its controls land in, and the standards hook a Chief Information Security Officer (CISO) is asked to evidence.
Block harmful or unsafe model outputs — hate, violence, self-harm, CSAM, illegal advice, hallucinations in high-impact decisions.
Detect and mitigate discriminatory or skewed model behaviour against groups defined by protected attributes.
Make model behaviour legible to operators, users, and regulators — model cards, attribution, AI-interaction disclosure.
Keep humans in the loop on high-impact decisions and give affected people a route to challenge an AI outcome.
Bound agent autonomy and tool scopes; gate irreversible actions; detect cascading or feedback-loop behaviour. The autonomy frontier — ties to the AI Cyber Maturity Model (AI-CMM) ladder (our model).
Pre-deploy safety assurance — run a benchmark suite + harm-focused red-team and gate releases on the result.
Detect harm over time — behavioural and safety drift, guardrail efficacy in production, harm-incident response.
See also
The measured proof, the skills that power the fix, and where this sits on the maturity model each have their own home, so they are linked, not repeated:
Honesty & attribution
XL-SafetyBench by AIM Intelligence (with Microsoft, Korea AI Safety Institute, KT) — eval code Apache-2.0, dataset CC BY 4.0. Used with attribution. Guardian/RepBend licenses pending verification.
Harm-class taxonomy source — NIST AI Risk Management Framework 1.0 (NIST AI 100-1), 1.0 (January 2023).