ai · security · skills
How it worksPlaybooksInsightsAsk a standardSkillGuardStart an assessment →

For DevSecAIOps · the method

How we solve it.Piece by piece, with provenance.

How AI risk becomes testable inside your pipeline — the gate set, what each gate checks, and the one gate proven end to end.

← Back to the DevSecAIOps pagePublic view · method skeleton
The gate setWhat each gate checksThe worked gateTHE METHODGates in your CIINPUTS → MECHANISM → OUTPUTS · WITH PROVENANCE

Piece 1

The gate set

Inputs

  • Your existing pipeline stages — the gates attach where your checks already run

Mechanism

  • Each tollgate is defined by the AI failure mode it catches (injection, tool misuse, model swap, data leak), the stage it guards, and the evidence it must produce to pass

Outputs

  • A gate list you can wire into CI, each with a stated check and a pass condition

Provenance: The tollgate model at /library/tollgates; gate ids are stable and deep-linkable.

1 deeper item · owner-gated · shown in a walkthrough

Piece 2

What each gate checks

Inputs

  • The AI-specific failure modes your code gates never see

Mechanism

  • Prompt injection is tested with an attack corpus, not a lint rule; model swaps are caught by integrity checks; tool misuse by scope diffs — each check is CI-shaped: deterministic input, binary verdict

Outputs

  • Checks your pipeline can run on every change, like everything else you ship

Provenance: Each gate names its check on the tollgates page.

1 deeper item · owner-gated · shown in a walkthrough

Piece 3

The worked gate

Inputs

  • A live app’s own guard and a fixed attack corpus

Mechanism

  • Gate #4 (prompt-injection red-team as CI) run end to end: baseline the shipped guard, fit it, re-score on the same corpus with a held-out set — then wire the scorer as a pipeline gate

Outputs

  • A measured before/after (modeled, fixed corpus), and a reference implementation you can reproduce

Provenance: The StoryBond case study — every number derived from lib/storybond-demo.ts and labeled with its evidence state.

1 deeper item · owner-gated · shown in a walkthrough

Piece 4

The stack & instruments

Inputs

  • 15 published instruments in three lanes — 13 testers, 3 corpora, 5 controls — validated at every build
  • Python 3 stdlib-only tooling — nothing to install to reproduce a claim

Mechanism

  • Tester-lane instruments ARE the pipeline gates: the jailbreak-resistance scorer that graded StoryBond is the same instrument you wire into CI — deterministic input, binary verdict
  • Lane integrity is machine-enforced: an instrument never grades its own lane, and nothing on this site says "tested" without a named instrument behind it

Outputs

  • A named, reproducible instrument behind every tested claim you read here

Provenance: lib/tools-registry.ts, guarded by validate-tools in the prebuild chain; instrument detail renders in the /controls drill.

3 deeper items · owner-gated · shown in a walkthrough

The exhibits

The working surfaces behind the answer.

DEVSECAIOPS · GATE FRAMEWORK · v0.2

Eight gates. Three lanes. Three lenses.

Every gate checks code, AI systems, and AI artifacts, for security, safety, and privacy.

LANES
L1 Classic L2 AI-augmented L3 AI artifacts
LENSES (within L3)
SecuritySafetyPrivacy
G0

Design

PLAN · before any code or model work begins

L1 ClassicSecurity
Threat model, security requirements, and abuse cases documented.
L2 AI-augmentedSecurity
AI-assisted threat modeling is permitted. A human signs off on every output.
L3 AI artifactsSafetyPrivacy
Risk class, the AI-vs-human allocation decision, a harm/misuse assessment, and the DPIA decision.
G1

Pre-commit

WORKSTATION · before code enters the repo

L1 ClassicSecurity
Pre-commit hooks, secrets scan, no credentials in code, linting.
L2 AI-augmentedSecurity
Agent-authored code must be attributable: which agent, which prompt, which human requested it.
L3 AI artifactsSecurityPrivacy
Prompt hygiene: no PII or secrets in system prompts. Skill files linted against schema.
G2

Source

COMMIT · PR · on every push and pull request

L1 ClassicSecurity
SAST, SCA, dependency scan, license check, IaC scanning.
L2 AI-augmentedSecurity
AI review is advisory only. AI-generated code gets stricter scrutiny, never less.
L3 AI artifactsSecuritySafetyPrivacy
AI-BOM and SkillBOM, a dataset poisoning check, a toxicity/bias screen, and a check for PII in data fixtures.
G3

Build

ARTIFACT · producing the deployable thing

L1 ClassicSecurity
Container scan, artifact signing (Sigstore), SLSA provenance, reproducible builds.
L2 AI-augmentedSecurity
Build agents run isolated. Writes are buffered through safe outputs, never direct.
L3 AI artifactsSecuritySafetyPrivacy
Model signing, a pickle / unsafe-serialization scan, the safety config shipped with the artifact, and anonymization verified.
G4

Test

VERIFY · evals are to AI what unit tests are to code

L1 ClassicSecurity
DAST, IAST, integration security tests.
L2 AI-augmentedSecurity
AI test generation and fuzzing. Results are triaged with a human for severity at or above high.
L3 AI artifacts: three independent release-blocking eval suites
SecurityPrompt injection, jailbreak, and tool-poisoning tests.
SafetyHarmful-output rate, bias and fairness metrics, hallucination thresholds.
PrivacyPII leakage, membership-inference, re-identification resistance.

A model can pass security red-teaming and still fail its fairness threshold. Each eval suite blocks independently.

G5

Deploy

ADMISSION · into the runtime environment

L1 ClassicSecurity
Policy-as-code admission, change approval, environment segregation.
L2 AI-augmentedSecurity
Deployment agents use task-scoped, expiring credentials. No standing access, ever.
L3 AI artifacts

Permission manifest compiled to enforcement. Kill-switch verified before go-live.

Human-in-the-loop checkpoints for consequential decisions; DPIA sign-off as an admission condition.

Retention and data-residency config verified; model card published (tri-lens: security, safety, privacy).

G6

Runtime

CONTINUOUS · not a one-time checkpoint

L1 ClassicSecurity
Monitoring, WAF, CSPM, vulnerability-management cadence.
L2 AI-augmentedSecurity
AI-assisted Security Operations Center (SOC) triage with an auditable investigation ledger. AI conclusions are traceable to evidence.
L3 AI artifacts

Agent behavior monitored against the manifest; drift from declared scope is an incident. Shadow-AI sweep.

Harm monitoring with an incident taxonomy, a user feedback channel, periodic eval re-runs.

Personal information (PII) egress classification, DSAR readiness, retention enforcement as a continuous check.

G7

Attest

MEASURE · closes the loop, feeds the next Design gate

L1 Classic
Evidence from G0 through G6 aggregated. Audit trail complete. DSOMM activity levels updated.
L2 AI-augmented
AI drafts the assessment narrative. The human signs: the attestation signature is always human.
L3 AI artifacts
AI Controls Matrix (AICM) controls, AI Security Maturity Model (AISMM) levels, National Institute of Standards and Technology (NIST) AI RMF, International Organization for Standardization (ISO) 42001, ISO 27701, EU AI Act: baseline and measured slope.
EVIDENCE → BASELINE → SLOPE → NEXT DESIGNG7 closes the loop back into G0, not forward to another artifact.

Maturity per gate: exists → automated → blocking → evidence-mapped → measured

A framework model, not a measured claim. Maps to public frameworks (OWASP DSOMM, Cloud Security Alliance (CSA) AICM/AISMM, SLSA, EU AI Act). The lanes are pipeline lanes; the lenses are evaluation dimensions within the AI-artifacts lane.

Canonical surfaces

The tollgates →The worked example (StoryBond) →