Case study · modeled · fixed corpus
Quieting a SOC's alert queue,end to end.
One alert queue, one decision policy, one before-and-after — by naming the org’s own noise, not buying a new tool.
Same fixed corpus every run · modeled
Before: The shipped posture: every alert escalates. Attacks are all caught — and so is all the noise.
Situation
Your queue escalates everything — that’s the safe default every tool ships with.
Complication
Analysts burn hours re-dismissing the same known noise, and the one real attack hides inside it. Nobody has written down what your org already knows is noise.
The question
“Can the queue get quiet without missing a real attack?”
The answer
Name the org’s own noise in a decision policy: noise escalated falls 100% → 0% while attacks stay 100% caught, held-out included. Modeled, fixed corpus.
This is the methodology run against a Security Operations Center’s (SOC’s) alert queue. We take a decision policy with no organizational context, name the recurring noise sources that policy can’t tell apart from real attacks, fit it to recognize exactly those sources, and measure the difference. No new vendor tool. We give the policy the context an experienced analyst already carries in their head, and show the lift.
This demonstrates the method is executable and measurable on a synthetic, structurally realistic alert set. It is not enterprise proof — that needs your own log data.
Diagnose.
The target is a SOC’s alert queue: a Security Information and Event Management (SIEM) rule engine with no organizational context, flagging 20alerts across eight alert types. Every one of those alerts lands on an analyst’s desk the same way, whether it’s a real attacker or a known, benign, recurring pattern.
Crucially, the noise isn’t random. This SOC already has, informally, the context to recognize most of it:
None of that context is encoded anywhere the rule engine can read it. So the real question is never “is the SOC flooded?” It is can a policy suppress exactly what the analyst already knows to ignore, without missing what they’d escalate— and that is a number, not an opinion.
Before the fit.
We ran a labelled corpus of 14 training alerts plus 6 held-out alerts through the context-free baseline: escalate everything. It caught 8/8 real attacks — trivially, since it escalates everything — and also flagged 6/6 known-benign alerts as if they were real. Here is where it can’t tell the difference, alert type by alert type:
| Alert class | Known noise | Real attack | Same type, different call |
|---|---|---|---|
| Port scan | 800 ports scanned from 10.50.0.14, inside the scanner’s own /24 (t03) | 600 ports scanned from 203.0.113.44, outside that range (t04) | ✗ |
| Off-hours admin login | svc-backup-01 logs in at 02:14, matching the backup naming pattern (t05) | j.rivera-admin logs in at 03:02 from an external IP (t06) | ✗ |
| Data-exfil volume spike | svc-backup-02 sends 42GB to the internal backup vault (t07) | m.chen sends 9GB to an external host outside the VPN range (t08) | ✗ |
| Suspicious PowerShell | encoded command, hash 9f2b1c7e4a — the one approved IT tool (t09) | encoded command, hash 77aa11bb22 — unrecognized (t10) | ✗ |
| Impossible travel | Austin → Amsterdam in 5 minutes, source IP inside the VPN egress range (t12) | Chicago → Lagos in 40 minutes, no VPN match (t11) | ✗ |
| Malware signature hit | the EICAR industry-standard antivirus test file, on ws-330 (t13) | a real TrojanDownloader signature, on ws-501 (t14) | ✗ |
| Brute-force login | no matching noise source exists for this class | 40 failed VPN-gateway logins from an external IP (t01) | ✗ |
Brute-force login has no matching noise source at all — a useful control row. It stays escalated before and after fitting, which is the point: fitting suppresses named noise, it never suppresses a whole alert type.
The fit.
We don’t suppress by alert type. We add seven narrow predicates, each naming one specific noise source and the exact scope where it applies — so a real attack in the same category still escalates:
The last predicate is validated entirely cold: no privilege-escalation alert exists anywhere in training. If that rule doesn’t generalize, the held-out set below would show it.
After the fit.
Same corpus, same seven predicates. The fitted policy suppresses 6/6 known-noise alerts and still escalates 8/8 real attacks — on the alerts it was fitted to, and on 6 it had never seen:
| Policy | Noise suppressed | Attacks still caught | Held-out |
|---|---|---|---|
| No fitting | 0/6 | 8/8 | 2/6 correct |
Fitted to this org seven named predicates | 6/6 | 8/8 | 6/6 correct |
The held-out column is the credibility test: every one of the 6 unseen alerts was disposed of correctly — the 4 with new literal details (a different backup account, a different scanner IP, a different change-ticket number) suppressed, and the 2 genuine attacks still caught. The policy generalized to the noise pattern, it wasn’t taught the test.
From a fix to a governed control.
A quieter queue is a tactic. A Center of Excellence makes it a governed, repeatable control— mapped to a recognised framework, threaded through the play’s own autonomy tiers, and advanced along a maturity ladder. Here is this one fit threaded down through the control spine.
The domains and the play’s per-tier required controls are real, from the alert-triage play. The skill-level AI Controls Matrix (AICM) control-objective ID, the AI Security Maturity Model (AISMM) target level, and the AI Consensus Assessments Initiative Questionnaire (AI-CAIQ) attestation aren’t authored for this skill yet — shown as mapping in progress, never invented. That loop — diagnose → fit → measure → map → attest → mature, then repeat on the next control — is the Center of Excellence. The single fix is just where it starts.
Why this is the whole point.
The deliverable was never a giant skill catalogue. It was one plain sentence a CISO can act on: the SOC’s triage policy went from escalating everything to suppressing exactly its own known noise, with zero real attacks missed, held-out included. That is a capability you now own — fitted to your stack, ready to mature up the AISMM ladder as another brick of your own AI-Security CoE.
Show your work.
How each repo instrument or skill actually touched this measurement, and which rung it moved.
| Instrument / skill | Applied here | Evidence | Rung |
|---|---|---|---|
| alert_triage_demo/scan.py corpus | Ran 20 labelled alerts through the context-free baseline | 100% false-positive rate; 7 named noise sources it couldn't tell from real attacks | Manual → Assisted |
| classify.py fitted policy | Added 7 narrow, named-noise-source predicates (scanner CIDR, backup pattern, VPN egress, IT-tool hash, EICAR marker, change-ticket prefix) | 0% false-positive rate, 100% attacks still caught, held-out 6/6 correct | Assisted → Augmented |
| Control mapping (governance spine) | Threaded the fix through AICM LOG/SEF, MITRE D3FEND, NIST CSF 2.0, and the play’s per-tier required controls | Mapped domains + controls; skill-level attestation/AISMM-level honestly in progress | Augmented → Autonomous-ready |
A production version of this fit is a repeatable skill, not a one-off script. Three curated core skills operationalize it: performing-alert-triage-with-elastic-siem, automating-ioc-enrichment, performing-ioc-enrichment-automation.
Methodology: the corpus, the baseline policy, the fitted policy, and the scan harness live in tools/alert_triage_demo/. See the AI-for-Security skill in the library →
Honesty. This result is Modeled — Real fixed-corpus result, not independently reviewed.
End of case study.