AgentShieldA guardrail that learns new PII rules from its misses, and unlearns the ones that misfire.
Agents now hand data to each other. Static filters miss anything they weren't written for: policy numbers, claim IDs, badge formats. AgentShield keeps the inline path fast and cheap, and moves LLM judgment offline, where it gets distilled into compiled, tested rules.
You can run its real engine in the redaction console on the home page.
The problem
Multi-agent systems pass customer data between agents all the time. A PII filter that only knows names, emails, and SSNs lets a policy number like INS-AZ-4471-K straight through, and nobody notices until it shows up somewhere it shouldn't.
How a message is handled
- Detect. Learned regex rules (L1) and Presidio NER (L2) run in a pure function with no I/O, the same code the evals and validator use.
- Block first. Blocking is decided on raw detections, before overlaps are resolved, so an overlapping rule can never hide an SSN.
- Apply the playbook. Per agent pair: tokenize, mask, generalize, or block. Floors no playbook can weaken: SSNs always block, cards are always at least masked.
- Tokenize reversibly. Tokens are a session-salted hash, consistent within a session and unlinkable across sessions. Raw values live only in a vault with a 1-hour TTL.
- Restore for the caller. The reply is shielded too, then only tokens minted in this exchange are swapped back, so the vault can't be used as a lookup oracle.
How it learns
- Audit. After delivery, an LLM auditor reviews the already-redacted payload. A finding is kept only if its span appears verbatim in the text, so hallucinations never become incidents. Incidents store the value's shape and hash, never the value.
- Synthesize. The miss, the three most similar past incidents (Atlas Vector Search), and the domain taxonomy go to a synthesizer that returns a regex with should-match and should-not-match examples, or says the case isn't regex-shaped.
- Validate. Four gates: the pattern compiles; no catastrophic backtracking on 10 KB adversarial strings; under 1% matches on 500 clean look-alike messages; recall rises on the labeled eval set while precision drops by at most a point. Failures go back to the synthesizer, up to three times.
- Approve and hot-load. A human approves in the terminal. A MongoDB change stream loads the rule into the running gateway within about a second, and a new eval generation is written.
How it unlearns
The gateway counts live hits per rule and stores each hit's context with the value replaced by its shape. A reviewer judges whether each hit was really personal data. In the recorded run, a policy-number rule started matching marketing product codes: 6 of its 7 live hits were judged not personal. Past 20% false positives on at least 5 hits from at least 2 sessions, retirement is proposed. The rule stays live until a human approves, then it's demoted to shadow and a new eval generation is written.
Requiring two independent sessions means one sender can't switch off a protection by spamming look-alikes.
Where it falls short
- The first occurrence of a new identifier type gets through: auditing happens after delivery, so the system learns from the leak rather than preventing it.
- Learned rules are regex only. Quasi-identifiers like "the only Cybertruck owner on Elm St" are flagged for a human.
- The retirement thresholds suit a demo; production would want a confidence interval before retiring a rule.
Stack
- FastAPI
- Microsoft Presidio
- spaCy
- MongoDB Atlas
- Change streams
- Atlas Vector Search
- Voyage embeddings
- OpenAI structured outputs
- Rich
- pytest
- Cloud Run (public demo)