> ## Documentation Index
> Fetch the complete documentation index at: https://docs.offboard.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# PII Scrubbing

> Three-pass PII removal system for protecting customer privacy in interview transcripts.

Offboard.tech uses a three-pass PII (Personally Identifiable Information) scrubbing system to protect customer privacy. All interview free-text fields are scrubbed before database storage.

## The Three Passes

1. **Regex Pre-pass** - Fast removal of obvious PII patterns
2. **AI Redaction** - Contextual removal of names and identifiers
3. **Jev Verification** - A TypeSafe `noul` judgment confirms no identifiers remain

## Failure Contract: Withhold, Don't Badge

If the AI pass throws, returns empty, or the Jev check flags residual identifiers, `scrubPII` returns `{ text: null, aiApplied: false }`. The caller stores `null` for free-text fields (`transcript_summary`, `root_cause`, `offer_presented`) and sets `metadata.pii_redaction = 'failed'`. **Regex-only output is never persisted** — it can still contain names, and a badge does not satisfy "raw transcripts never stored". Non-sensitive lifecycle fields (`archetype`, `sentiment`, `turn_count`) are always preserved.

## Storage Policy

| Data                                            | Policy                                                      |
| ----------------------------------------------- | ----------------------------------------------------------- |
| Visitor free text (summary, root\_cause, offer) | Always scrubbed; withheld on failure                        |
| Merchant URL-param metadata                     | Allowlisted keys only; server lifecycle fields are reserved |
| Server logs                                     | Never print raw message bodies                              |
| Outbox payloads                                 | Scrubbed fields only                                        |
| Email alerts                                    | Sent to workspace owner only                                |

Development note: the AI SDK DevTools middleware (`lib/ai/provider.ts`) captures prompts locally in `NODE_ENV=development` only — it never wraps production models.

## What Gets Scrubbed

### Pass 1: Regex Patterns

| Pattern        | Example                      | Redacted As             |
| -------------- | ---------------------------- | ----------------------- |
| Email          | `john@test.com`              | `[REDACTED_EMAIL]`      |
| Phone (US)     | `555-123-4567`               | `[REDACTED_PHONE]`      |
| Phone (Intl)   | `+44 20 1234 5678`           | `[REDACTED_PHONE]`      |
| URL            | `https://example.com`        | `[REDACTED_URL]`        |
| IP Address     | `192.168.1.1`                | `[REDACTED_IP]`         |
| Credit Card    | `1234-5678-9012-3456`        | `[REDACTED_CREDITCARD]` |
| SSN            | `123-45-6789`                | `[REDACTED_SSN]`        |
| UUID           | `550e8400-e29b-41d4-a716...` | `[REDACTED_UUID]`       |
| Street Address | `123 Main Street`            | `[REDACTED_ADDRESS]`    |

### Pass 2: AI Redaction

The AI model identifies and redacts:

* People's names (first, last, full)
* Company names not in allowlist
* Geographic locations
* Usernames/handles
* Account/reference numbers
* Custom identifiers

## Usage

### Basic Scrubbing

```typescript theme={null}
import { scrubPII } from '@/lib/pii/scrubber'
import { createAIModel } from '@/lib/ai/provider'

const rawTranscript = "Hi, I'm John from Acme Inc. My email is john@acme.com and I'm canceling because the price is too high at $99/month. You can reach me at 555-123-4567."

const model = createAIModel()
const { text, aiApplied } = await scrubPII(rawTranscript, model)

// text: "Hi, I'm [REDACTED] from [REDACTED]. My email is [REDACTED_EMAIL] and I'm canceling because the price is too high at $99/month. You can reach me at [REDACTED_PHONE]."
// If aiApplied is false, text is null — store null, not the raw input.
```

### Regex-Only Scrubbing (Faster, Less Thorough)

```typescript theme={null}
import { scrubWithRegex } from '@/lib/pii'

const quickScrub = scrubWithRegex(rawText)
```

### Check for Potential PII

```typescript theme={null}
import { containsPotentialPII } from '@/lib/pii'

if (containsPotentialPII(userInput)) {
  // Warn user or process differently
}
```

## Integration

### In Interview Tool

The `complete_interview` tool automatically scrubs before saving:

```typescript theme={null}
const complete_interview = {
  description: 'Complete the interview and save redacted summary',
  inputSchema: completeInterviewSchema,
  execute: async (input) => {
    // PII scrub the summary before saving
    const model = createAIModel()
    const redactedSummary = await scrubPII(input.summary, model)

    // Save to database
    await db.interviews.update({
      transcript_summary: redactedSummary
    })
  }
}
```

### API Route Middleware

```typescript theme={null}
import { scrubPII, containsPotentialPII } from '@/lib/pii'
import { createAIModel } from '@/lib/ai'

export async function POST(request: Request) {
  const { summary } = await request.json()

  // Check for PII before processing
  if (containsPotentialPII(summary)) {
    console.log('[PII] Potential PII detected, scrubbing...')
  }

  // Scrub before saving
  const model = createAIModel()
  const scrubbed = await scrubPII(summary, model)

  // Save scrubbed version
  await db.summaries.create({ content: scrubbed })
}
```

## AI Prompt for Redaction

The AI redaction pass uses a carefully crafted prompt to avoid over-redaction:

```
You are a PII redaction specialist. Your job is to identify and replace
personally identifiable information (PII) in the text below.

REPLACE WITH [REDACTED]:
- Names of SPECIFIC people (e.g., "John Smith", "Sarah", "Dr. Johnson")
- Names of SPECIFIC companies or organizations (e.g., "Acme Corp", "Google", "Stark Industries")
- Names of SPECIFIC locations (e.g., "San Francisco", "123 Main St", "Texas")
- Specific usernames, handles, or screen names (e.g., "@johndoe", "user123")
- Specific account/ID/ticket numbers (e.g., "account #12345", "ticket #ABC-123")

DO NOT REDACT - Keep these unchanged:
- Generic terms: "customer", "user", "client", "merchant", "company", "platform", "service", "product"
- Common pronouns: "our", "their", "my", "your", "his", "her", "its"
- Generic descriptors: "the system", "the software", "the tool", "the app"
- Common business terms: "team", "support", "sales", "billing", "management"
- Feature names unless they contain personal info: "dashboard", "API", "integration"

EXAMPLES:
Input: "Contact John Smith at john@example.com or call 555-123-4567"
Output: "Contact [REDACTED] at [REDACTED_EMAIL] or call [REDACTED_PHONE]"

Input: "Customer left due to finding a cheaper alternative with better perceived value"
Output: "Customer left due to finding a cheaper alternative with better perceived value"
(NO redaction - these are generic terms)

Input: "Sarah from Acme Corp said their team was unhappy with our platform's pricing"
Output: "[REDACTED] from [REDACTED] said their team was unhappy with our platform's pricing"
(Redact Sarah and Acme Corp, but keep "their", "our", "platform")

Text to redact:
[TEXT]

Return ONLY the redacted text, no explanation.
```

This prompt design ensures:

* **Specific PII is removed** (names, companies, locations)
* **Generic business language is preserved** (customer, platform, our, their)
* **Transcript readability is maintained** for analysis purposes

## Performance Considerations

| Method                   | Speed            | Coverage | Use Case                       |
| ------------------------ | ---------------- | -------- | ------------------------------ |
| `scrubWithRegex`         | Fast (\~1ms)     | 80-90%   | Pre-validation, quick checks   |
| `scrubPII` (both passes) | Slower (\~500ms) | 95-99%   | Production storage, compliance |

**Recommendation**: Always use `scrubPII` for database storage. Use `scrubWithRegex` only for client-side validation or previews.

## Edge Cases

### Generic Business Terms (NOT redacted)

The AI is trained to preserve generic business language that isn't PII:

```typescript theme={null}
Input: "Customer left due to finding a cheaper alternative with better perceived value, despite our platform's features."
Output: "Customer left due to finding a cheaper alternative with better perceived value, despite our platform's features."
```

Common words that are **preserved**:

* Generic roles: "customer", "user", "client", "merchant", "admin"
* Pronouns: "our", "their", "my", "your", "his", "her"
* Business terms: "platform", "service", "product", "company", "team"
* Feature names: "dashboard", "API", "integration", "plugin"

### Preserved Information

These are **NOT** redacted:

* Product names ("Pro", "Enterprise", "Basic")
* Feature names ("API access", "SSO")
* Pricing information ($99/month, $500/year)
* Technical terms (URLs, API keys in code blocks)

### Partial PII

```typescript theme={null}
Input: "My email is john at test dot com"
Output: "My email is john at test dot com" // Not regex-matched, but AI catches it
```

### Context-Aware Redaction

```typescript theme={null}
Input: "I work at Apple in Cupertino"
Output: "I work at [REDACTED] in [REDACTED]" // Both company and location
```

## Compliance

The PII scrubbing system is designed to help with:

* **GDPR** - Data minimization and pseudonymization
* **CCPA** - Privacy by design
* **SOC 2** - Data protection controls

> **Note**: PII scrubbing is a privacy-enhancing technology, not a legal compliance solution. Consult legal counsel for specific compliance requirements.

## Testing

```typescript theme={null}
import { describe, it, expect } from 'vitest'
import { scrubWithRegex } from '@/lib/pii'

describe('PII Scrubbing', () => {
  it('redacts email addresses', () => {
    const input = 'Contact john@test.com'
    const output = scrubWithRegex(input)
    expect(output).toBe('Contact [REDACTED_EMAIL]')
  })

  it('redacts multiple PII types', () => {
    const input = 'Call John at 555-123-4567 or email jane@test.com'
    const output = scrubWithRegex(input)
    expect(output).toContain('[REDACTED_PHONE]')
    expect(output).toContain('[REDACTED_EMAIL]')
  })
})
```

## Best Practices

1. **Always scrub before DB writes** - Never store raw transcripts
2. **Use `scrubPII` for production** - Both passes for maximum coverage
3. **Log scrubbing failures** - Alert if AI redaction fails
4. **Validate scrubbed output** - Ensure text is still readable
5. **Test with real transcripts** - Verify PII patterns in your domain