Case study · AI for Security · reading a public exhibit
A real AI SOC, read on our own grid.We name the gaps its own code shows.
Situation
Genuinely governance-forward AI SOCs now exist in the open — this one ships an autonomy dial and hard approval gates.
Complication
Even the best public exhibit can’t say, in a standard vocabulary, where its controls actually land — or where its gate would trip.
The question
“Can an independent grid read a real AI system’s governance from its source — and name the gaps its own docs can’t?”
The answer
Read the published source against Maturity × Autonomy: where it lands per category, where the gate trips, and the two named gaps — each with the fitted skill that closes it. Modeled from source; nothing executed.
AiSOC (beenuar/AiSOC, MIT) is a genuine, open, self-hostable AI Security Operations Center (AI SOC): a LangGraph agent triages, investigates and recommends containment, logged step-by-step. It is the most governance-forward public instance of our own thesis we have found — it ships a per-tenant L0–L4 autonomy dial that gates every autonomous action by blast radius. We read its published source against our own Maturity × Autonomy diagnostic to see where it lands, and where the gate would trip.
Honesty. This reading is Modeled — Real fixed-corpus result, not independently reviewed. It is built entirely from AiSOC’s published, MIT-licensed source (config files, code, and its own documentation) as of 2026-06-30 (commit ec73f26); we did not deploy or execute AiSOC to produce it. No before/after metric is claimed here — that would require actually running the stack and is scoped as future work. AiSOC is named with respect: its own benchmark documentation enforces the same honesty discipline we do (never present a substrate self-check as live performance), which is part of why it is a credible exhibit.
Where it lands.
Four categories, read from the source that governs each — not from a live run. Its default production posture (autonomous inside pre-approved bounded scopes, human on exception) reads as AI-CMM Augmented; its whitelist-gated top tier reads as Autonomous. Its own L0–L4 dial is a different, product-specific ladder — we are not equating the two beyond the general shape both converge on.
Append-only Investigation Ledger: every large language model (LLM) prompt, response, tool call, and decision rationale logged per step, with SHA-256 input/output hashes and row-level security (RLS) tenant isolation.
Source: services/agents/app/investigator/ledger.py
Investigator agent tools are read-only enrichment only (no write access); a prompt sanitizer strips jailbreak patterns and an LLM input contract blocks raw logs/personally identifiable information (PII)/secrets. But no declared per-tool rate limit or aggregate budget on the agent’s own tool scope.
Source: services/agents/app/investigator/prompt_sanitizer.py, app/llm/contract.py
Three-tier confidence-gated policy (auto/review/escalation) per action, in versioned YAML with per-tenant DB overrides. Confidence values come from a calibrated scoring layer, checked against drift only in continuous integration (CI) (a Brier-score gate), not monitored live.
Source: services/agents/config/autonomy_policy.yaml
High-blast-radius actions (isolate_host, disable_user_account, kill_process, run_script) are hard-required to route through human approval regardless of confidence — the gate is real. But an approval request has no enforced timeout at the API layer (a 900s timeout exists only in a Slack-bot integration), and there is no documented kill switch or circuit breaker on the investigator pipeline itself.
Source: services/actions/app/models/action.py (APPROVAL_REQUIRED_ACTIONS)
The gaps, and the fitted few that close them.
Even a governance-forward stack ships ungoverned edges — each one a control our diagnostic already asks about. Naming them is not a critique of AiSOC; it is exactly the exercise the diagnostic is for, run against a target anyone can verify against the source.
No declared rate limit or aggregate budget on the investigator agent’s own tool calls
A runaway loop calling the investigate endpoint repeatedly has no cap — cost and audit-log blast radius, not a security breach, but a governance gap the diagnostic would flag.
Fitted skill: bounding-agent-autonomy-and-tool-scopes-least-privilege →No kill switch or circuit breaker on the agent pipeline; approval requests can pend indefinitely at the API layer
If a provider outage or a guardrail failure causes bad recommendations, there is no on-call manual override to pause investigations; a critical containment approval can wait forever if the on-call analyst is offline.
Fitted skill: implementing-kill-switch-and-rollback-for-autonomous-agents →Both skills are published in the library today; neither was authored for this exhibit — they already existed to close exactly this class of gap. See the library →
Why this matters. The point is not that AiSOC is unsafe — by public-repo standards it is unusually careful. The point is that even a careful, governance-forward AI SOC has named, source-verifiable gaps, and a fitted-few intervention (not a rebuild) is what closes them. A live before/after on a running AiSOC deployment is scoped as a future exhibit; this reading is the source-grounded first step. Run the same diagnostic on your own function →