← Back to Catherene Baker — Business Automation & Operations Systems for Small Business Owners

<aside> 🤖

An AI agent that audits vendor risk management policies against NIST CSF 2.0, autonomously deciding which controls to check and producing a structured gap report.

</aside>

The Problem

Financial institutions are required to assess vendor relationships against cybersecurity frameworks like NIST CSF, but manual gap analysis is slow and inconsistent, and depends heavily on the reviewer's familiarity with the full framework. A reviewer might miss subtle distinctions, like a policy that assigns access at onboarding but never requires reviewing that access over time.

The Solution

I built an AI agent that reads a vendor risk policy, autonomously queries a NIST CSF 2.0 reference dataset for relevant subcategories across five control areas, and produces a structured, evidence-cited gap report, without a human pre-defining which checks to run.

How It Works

  1. A fictional vendor risk policy document is provided as input
  2. The AI Agent (built in n8n, using Google Gemini) reasons about which NIST CSF subcategories are relevant to check
  3. The agent calls a Google Sheets tool on its own initiative to look up the actual subcategory language across five control areas: Supply Chain Risk Management (GV.SC), Asset Management (ID.AM), Risk Assessment (ID.RA), Identity/Access Control (PR.AA), and Data Security (PR.DS)
  4. It evaluates the policy against each subcategory it retrieves, rating each Covered, Partial, or Gap, with cited reasoning
  5. It compiles a full structured report covering all subcategories checked, not just the ones with issues

This is what makes it an agent rather than a workflow: the tool calls, the choice of what to check, and the reasoning behind each rating are all decided by the model at runtime, not hardcoded in advance.

Tech Stack

The Hard Part

To test the agent honestly, I wrote a fictional vendor risk policy with seven deliberate, specific gaps I designed myself, so I had a known answer key to grade the agent's output against, rather than just trusting whatever it produced.

The first run caught 3 of 7 gaps well, but revealed two real reliability problems: the agent stopped after finding a few notable issues instead of systematically checking every relevant control, and it mismarked two subcategories as "Covered" by matching on adjacent topics rather than verifying the full requirement. For example, it treated "access is assigned on a least-privilege basis at onboarding" as satisfying a requirement that also covers periodic access review, which is a distinct control the policy never actually addressed.