The Three Passes
- Regex Pre-pass - Fast removal of obvious PII patterns
- AI Redaction - Contextual removal of names and identifiers
- Jev Verification - A TypeSafe
nouljudgment confirms no identifiers remain
Failure Contract: Withhold, Don’t Badge
If the AI pass throws, returns empty, or the Jev check flags residual identifiers,scrubPII returns { text: null, aiApplied: false }. The caller stores null for free-text fields (transcript_summary, root_cause, offer_presented) and sets metadata.pii_redaction = 'failed'. Regex-only output is never persisted — it can still contain names, and a badge does not satisfy “raw transcripts never stored”. Non-sensitive lifecycle fields (archetype, sentiment, turn_count) are always preserved.
Storage Policy
Development note: the AI SDK DevTools middleware (
lib/ai/provider.ts) captures prompts locally in NODE_ENV=development only — it never wraps production models.
What Gets Scrubbed
Pass 1: Regex Patterns
Pass 2: AI Redaction
The AI model identifies and redacts:- People’s names (first, last, full)
- Company names not in allowlist
- Geographic locations
- Usernames/handles
- Account/reference numbers
- Custom identifiers
Usage
Basic Scrubbing
Regex-Only Scrubbing (Faster, Less Thorough)
Check for Potential PII
Integration
In Interview Tool
Thecomplete_interview tool automatically scrubs before saving:
API Route Middleware
AI Prompt for Redaction
The AI redaction pass uses a carefully crafted prompt to avoid over-redaction:- Specific PII is removed (names, companies, locations)
- Generic business language is preserved (customer, platform, our, their)
- Transcript readability is maintained for analysis purposes
Performance Considerations
Recommendation: Always use
scrubPII for database storage. Use scrubWithRegex only for client-side validation or previews.
Edge Cases
Generic Business Terms (NOT redacted)
The AI is trained to preserve generic business language that isn’t PII:- Generic roles: “customer”, “user”, “client”, “merchant”, “admin”
- Pronouns: “our”, “their”, “my”, “your”, “his”, “her”
- Business terms: “platform”, “service”, “product”, “company”, “team”
- Feature names: “dashboard”, “API”, “integration”, “plugin”
Preserved Information
These are NOT redacted:- Product names (“Pro”, “Enterprise”, “Basic”)
- Feature names (“API access”, “SSO”)
- Pricing information (500/year)
- Technical terms (URLs, API keys in code blocks)
Partial PII
Context-Aware Redaction
Compliance
The PII scrubbing system is designed to help with:- GDPR - Data minimization and pseudonymization
- CCPA - Privacy by design
- SOC 2 - Data protection controls
Note: PII scrubbing is a privacy-enhancing technology, not a legal compliance solution. Consult legal counsel for specific compliance requirements.
Testing
Best Practices
- Always scrub before DB writes - Never store raw transcripts
- Use
scrubPIIfor production - Both passes for maximum coverage - Log scrubbing failures - Alert if AI redaction fails
- Validate scrubbed output - Ensure text is still readable
- Test with real transcripts - Verify PII patterns in your domain