llmverify

Stop shipping unsafe AI output. Local-first verification, safety monitoring, and guardrails for any LLM — in one npm install.

100% Local | No Telemetry | Privacy-First | 500 free calls/day

---

The Problem

You're building with GPT-4, Claude, Gemini, or any LLM. Your AI:

• Hallucinates facts and citations that don't exist
• Leaks PII — emails, phone numbers, SSNs in responses
• Gets prompt-injected — users trick it into ignoring instructions
• Returns broken JSON that crashes your parser
• Drifts in quality over time without anyone noticing

You need a safety layer between your LLM and your users. That's llmverify.

What You Get

npm install llmverify

Feature	One-liner	What it catches
Verify output	await verify(aiResponse)	Hallucinations, harmful content, quality issues
Block injections	isInputSafe(userMessage)	Prompt injection, jailbreaks, malicious input
Redact PII	redactPII(text)	Emails, phones, SSNs, credit cards, API keys
Fix broken JSON	repairJSON(broken)	Missing brackets, trailing commas, unquoted keys
Monitor health	monitorLLM(client)	Latency spikes, token drift, behavioral changes
Sentinel tests	sentinel.quick(client, model)	LLM regressions before users see them
Classify output	classify(prompt, response)	Intent detection, content categorization
Audit trail	audit.log(event)	SOC2, HIPAA, GDPR compliance evidence

Everything runs locally. No data leaves your machine. No API keys needed for free tier.

Complete Feature Map

Category	Feature	Function	What it does
Safety	Prompt injection defense	isInputSafe()	Block jailbreaks, instruction override, data exfiltration
Safety	PII detection & redaction	redactPII() / containsPII()	Strip emails, phones, SSNs, credit cards, API keys
Safety	Harmful content filtering	checkHarmfulContent()	Flag toxic, violent, or inappropriate responses
Safety	Injection risk scoring	getInjectionRiskScore()	0–1 numerical severity score
Quality	Hallucination risk	calculateHallucinationRisk()	Detect hedging, self-contradiction, unsupported claims
Quality	JSON validation & repair	detectAndRepairJson()	Fix missing brackets, trailing commas, unquoted keys
Quality	Output classification	classify(prompt, output)	Categorize by intent, topic, compliance risk
Quality	Consistency analysis	ConsistencyEngine	Detect divergent answers to the same question
DevEx	One-line verify	verify(text)	Full risk assessment in one call
DevEx	Zod-like guard API	guard() / safe() / parse()	Familiar patterns — {ok, data, risk}
DevEx	9 LLM adapters	createAdapter({provider})	OpenAI, Anthropic, Groq, Google, DeepSeek, Mistral, Cohere, local, custom
DevEx	Plugin system	use(createPlugin())	Add custom verification rules
DevEx	CLI presets	--preset dev|prod|strict	One command for any environment
Ops	Runtime health monitor	monitorLLM(client)	Track latency, token drift, behavioral changes
Ops	Sentinel regression tests	runAllSentinelTests()	4 proactive tests to catch LLM regressions
Ops	Audit logging	AuditLogger	Local-only trail for SOC2/HIPAA/GDPR evidence
Ops	Baseline drift detection	BaselineStorage	Alert when LLM behavior changes from baseline
Privacy	100% local processing	—	Zero network requests, verifiable with tcpdump
Privacy	No telemetry	—	No tracking, no phone-home, no data collection

All detection is 100% deterministic — regex + heuristics, zero ML, zero AI inference. Same input = same output, always.

Part of the HAIEC AI governance platform. Use alongside AI Security Scanner, CI/CD Pipeline, and Runtime Injection Testing.

---

Install

npm install llmverify

---

Quick Start (30 seconds)

const { verify, isInputSafe, redactPII } = require('llmverify');

// 1. Check user input before sending to AI
if (!isInputSafe(userMessage)) {
  return { error: 'Invalid input detected' };
}

// 2. Get AI response (your existing code)
const aiResponse = await yourLLM.generate(userMessage);

// 3. Verify the output before showing to users
const result = await verify(aiResponse);

if (result.risk.level === 'critical') {
  return { error: 'Response failed safety check' };
}

// 4. Redact any PII before logging
const { redacted } = redactPII(aiResponse);
console.log(redacted); // "Contact [REDACTED] at [REDACTED]"

That's it. Three lines of safety between your LLM and your users.

---

For IDE Users (Windsurf, VS Code, Cursor)

llmverify includes a REST server and clipboard monitor for real-time verification inside your IDE.

# Terminal 1: Start verification server
npm run serve

# Terminal 2: Start clipboard monitor
npm run monitor

# Now copy any AI response → see risk scores automatically

Port conflict? Run npm run serve:force

You can also use the programmatic IDE extension:

import { LLMVerifyIDE, createIDEExtension } from 'llmverify';

const ide = new LLMVerifyIDE({ useLocalFallback: true });
const result = await ide.verify(clipboardText);

Full IDE Setup Guide →

---

Core Functions

1. Verify AI Output

Checks AI responses for safety, accuracy, and quality before showing to users.

const { verify } = require('llmverify');

// Object form
const result = await verify({ content: aiResponse });

// String shorthand (same thing)
const result = await verify(aiResponse);

console.log(result.risk.level);   // "low", "moderate", "high", "critical"
console.log(result.risk.overall); // 0.172 (17.2% risk)
console.log(result.limitations);  // What wasn't checked (transparency)

Risk Levels:

• LOW (0-25%): Safe to use
• MODERATE (26-50%): Review recommended
• HIGH (51-75%): Fix before using
• CRITICAL (76-100%): Do not use

Risk Levels Guide →

---

2. Check Input Safety

What it does: Validates user input before sending to AI
When to use: On every user message
Benefit: Prevent prompt injection attacks

const { isInputSafe } = require('llmverify');

if (!isInputSafe(userMessage)) {
  return { error: 'Invalid input detected' };
}

// Safe to send to AI
const aiResponse = await callAI(userMessage);

What it checks:

• Prompt injection patterns
• Malicious commands
• Suspicious formatting
• Excessive length

---

3. Remove PII

What it does: Redacts personal information from text
When to use: Before logging or storing AI responses
Benefit: GDPR/CCPA compliance, privacy protection

const { redactPII } = require('llmverify');

const { redacted, piiCount } = redactPII(aiResponse);

console.log(redacted);  // "Contact us at [REDACTED]"
console.log(piiCount);  // 2

Detects:

• Email addresses
• Phone numbers
• Social Security Numbers
• Credit card numbers
• API keys

---

4. Fix Malformed JSON

What it does: Repairs broken JSON from AI responses
When to use: When AI returns invalid JSON
Benefit: Avoid parsing errors, improve reliability

const { repairJSON } = require('llmverify');

const brokenJSON = '{"name": "John", "age": 30';  // Missing }
const fixed = repairJSON(brokenJSON);

console.log(fixed);  // {"name": "John", "age": 30}

Fixes:

• Missing brackets/braces
• Trailing commas
• Unquoted keys
• Escape issues

---

5. Classify Output

What it does: Categorizes AI responses by intent/type
When to use: Routing, filtering, or organizing responses
Benefit: Automated content organization

const { classify } = require('llmverify');

const result = classify('Summarize this article', aiResponse);

console.log(result.intent);       // "question", "answer", "code", "explanation"
console.log(result.confidence);  // 0.85

Categories:

• Question
• Answer
• Code
• Explanation
• Instruction
• Creative

---

6. Monitor Performance

What it does: Tracks AI system health and performance
When to use: Production monitoring
Benefit: Detect issues before users do

const { monitor } = require('llmverify');

monitor.track({
  model: 'gpt-4',
  latency: 1200,
  tokens: 500,
  cost: 0.02
});

const health = monitor.getHealth();
console.log(health.status);  // "healthy", "degraded", "critical"

Monitors:

• Response latency
• Token usage
• Error rates
• Cost tracking
• Behavioral drift

---

7. Audit Logging

What it does: Creates compliance-ready audit trails
When to use: Regulated industries, compliance requirements
Benefit: Meet SOC2, HIPAA, GDPR requirements

const { audit } = require('llmverify');

audit.log({
  action: 'ai_response_verified',
  user: 'user_123',
  riskLevel: 'low',
  timestamp: new Date()
});

const logs = audit.query({ user: 'user_123', limit: 10 });

Captures:

• All verifications
• Risk scores
• User actions
• Timestamps
• Findings

---

8. Generate Compliance Badge

What it does: Creates verification badges for your content
When to use: Show users content is verified
Benefit: Build trust, demonstrate safety

const { generateBadge } = require('llmverify');

const badge = generateBadge({
  riskLevel: 'low',
  score: 17.2,
  verified: true
});

// Use in HTML
// <img src="data:image/svg+xml;base64,..." alt="Verified" />

Badge Types:

• Verification status
• Risk level
• Score display
• Custom branding

---

Server Mode (REST API)

What it does: Run llmverify as a standalone service
When to use: IDE integration, microservices, external tools
Benefit: Language-agnostic, easy integration

# Start server
npm run serve

# Server runs on http://localhost:9009

Endpoints:

# Health check
GET /health

# Verify content
POST /verify
Body: {"content": "AI response text"}

# Check input
POST /check-input
Body: {"input": "User message"}

# Detect PII
POST /check-pii
Body: {"content": "Text with PII"}

# Classify
POST /classify
Body: {"content": "AI response"}

Example:

const response = await fetch('http://localhost:9009/verify', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ content: aiResponse })
});

const result = await response.json();

Server API Reference →

---

CLI Tools

What it does: Verify content from command line
When to use: Scripts, automation, testing
Benefit: No code required

# Verify a file
llmverify verify response.txt

# Check input
llmverify check-input "User message"

# Redact PII
llmverify redact-pii document.txt

# System health
llmverify doctor

CLI Reference →

---

Integration Examples

With OpenAI

const OpenAI = require('openai');
const { verify, isInputSafe } = require('llmverify');

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function safeChat(userMessage) {
  // 1. Validate input
  if (!isInputSafe(userMessage)) {
    throw new Error('Invalid input');
  }
  
  // 2. Get AI response
  const completion = await openai.chat.completions.create({
    model: "gpt-4",
    messages: [{ role: "user", content: userMessage }]
  });
  
  const aiResponse = completion.choices[0].message.content;
  
  // 3. Verify output
  const verification = await verify(aiResponse);
  
  if (verification.risk.level === 'critical') {
    throw new Error('Response failed safety check');
  }
  
  return {
    response: aiResponse,
    risk: verification.risk
  };
}

With Express API

const express = require('express');
const { verify, isInputSafe, redactPII } = require('llmverify');

const app = express();

app.post('/chat', async (req, res) => {
  const { message } = req.body;
  
  // Validate input
  if (!isInputSafe(message)) {
    return res.status(400).json({ error: 'Invalid input' });
  }
  
  // Get AI response
  const aiResponse = await callAI(message);
  
  // Verify output
  const verification = await verify(aiResponse);
  
  // Redact PII
  const { redacted } = redactPII(aiResponse);
  
  // Block critical risk
  if (verification.risk.level === 'critical') {
    return res.status(400).json({ error: 'Response failed safety check' });
  }
  
  res.json({
    response: redacted,
    riskScore: verification.risk.overall,
    riskLevel: verification.risk.level
  });
});

With Streaming

const { verify } = require('llmverify');

async function streamWithVerification(userMessage) {
  let fullResponse = '';
  
  // Stream AI response
  const stream = await openai.chat.completions.create({
    model: 'gpt-4',
    messages: [{ role: 'user', content: userMessage }],
    stream: true
  });
  
  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content || '';
    fullResponse += content;
    process.stdout.write(content);  // Stream to user
  }
  
  // Verify after streaming completes
  const verification = await verify(fullResponse);
  
  if (verification.risk.level === 'high' || 
      verification.risk.level === 'critical') {
    console.log('\n[WARNING] Response has elevated risk');
    console.log(`Risk: ${verification.risk.overall * 100}%`);
  }
  
  return { fullResponse, verification };
}

---

Configuration

Create .llmverify.json in your project:

{
  "ignoreFindings": ["injection-marker"],
  "context": "ai-agent-development",
  "whitelist": {
    "pii": ["example.com", "test.com"]
  },
  "strictMode": false,
  "logLevel": "info"
}

Configuration Guide →

---

Pricing & Tiers

All features available on every tier. Usage tracked locally — no data leaves your machine.

Tier	Price	Daily Calls	Content Limit	Support
Free	$0	500/day	50KB	Community
Starter	$79/mo	5,000/day	200KB	Email (72hr)
Pro	$299/mo	50,000/day	1MB	Email (24hr)
Business	$999/mo	Unlimited	Unlimited	Slack (4hr SLA)

# Check your usage anytime
npx llmverify usage

# See current tier and limits
npx llmverify tier

All calls share a single daily pool: verify, guard, safe, parse, sentinel, monitor.
Usage resets at midnight local time. A 10% grace period lets you finish work without hard cutoffs.

---

Sentinel Tests

Proactive behavioral tests that verify your LLM is responding correctly — run in CI/CD, health checks, or before deployment.

const { sentinel } = require('llmverify');

// One-liner — runs all 4 sentinel tests
const suite = await sentinel.quick(myClient, 'gpt-4');
console.log(suite.passed);    // true/false
console.log(suite.passRate);  // 0.75 = 3/4 passed

// Run a single test
const echo = await sentinel.test('staticEchoTest', myClient, 'gpt-4');

Built-in tests:

• Static Echo — Does the LLM echo back exact content?
• Duplicate Query — Are responses consistent across identical prompts?
• Structured List — Can it follow output format instructions?
• Short Reasoning — Does it show step-by-step reasoning?

---

Health Monitoring

Wraps any LLM client with real-time health tracking. Detects latency spikes, token drift, and behavioral changes automatically.

const { monitorLLM } = require('llmverify');

const monitored = monitorLLM(openaiClient, {
  hooks: {
    onUnstable: (report) => alert('LLM unstable!'),
    onDegraded: (report) => console.warn('Degraded'),
    onRecovery: (report) => console.log('Recovered')
  }
});

const response = await monitored.generate({ prompt: 'Hello' });
console.log(response.llmverify.health);  // 'stable' | 'degraded' | 'unstable'

---

LLM Provider Adapters

Unified interface for any LLM provider. Swap providers without changing your verification code.

const { createAdapter, monitorLLM } = require('llmverify');

// Works with OpenAI, Anthropic, Groq, Google, DeepSeek, Mistral, Cohere, local models
const client = createAdapter({ provider: 'openai', apiKey: process.env.OPENAI_API_KEY });
const monitored = monitorLLM(client);

const response = await monitored.generate({ prompt: 'Hello', model: 'gpt-4' });

---

Plugin System

Extend llmverify with custom verification rules.

const { use, createPlugin, createRegexPlugin } = require('llmverify');

// Built-in plugin helpers
use(createRegexPlugin('no-urls', /https?:\/\/\S+/g, 'URL detected in output'));

// Custom plugin
use(createPlugin('custom-check', async (ctx) => {
  const hasBadWord = ctx.content.includes('forbidden');
  return { passed: !hasBadWord, message: hasBadWord ? 'Forbidden word found' : 'Clean' };
}));

---

Error Handling

Every error includes a code, severity, recoverability flag, and a human-readable suggestion.

const { verify, LLMVerifyError } = require('llmverify');

try {
  const result = await verify(aiResponse);
} catch (err) {
  if (err instanceof LLMVerifyError) {
    console.log(err.code);                // "LLMVERIFY_1003"
    console.log(err.metadata.severity);   // "medium"
    console.log(err.metadata.recoverable); // true
    console.log(err.metadata.suggestion); // "Reduce content size or split into smaller chunks"
  }
}

Error code ranges:

• 1xxx — Input validation (empty, too large, bad format)
• 2xxx — Configuration (malformed config, invalid tier)
• 3xxx — Runtime (timeout, engine failure)
• 4xxx — Server (port in use, rate limit)
• 5xxx — Privacy/security (privacy violation, unauthorized)
• 6xxx — Plugins (not found, load failed)
• 7xxx — Usage/tier (daily limit, content length)

All errors are typed — use instanceof checks or the code property for programmatic handling.

---

DX Shortcuts

// String shorthand — no object wrapper needed
const result = await verify("The Earth is flat.");

// Default export
import llmverify from 'llmverify';
const result = await llmverify.verify(text);

// Shorthand APIs
import { ai, llm, guardrails } from 'llmverify';
const safe = await ai.safe(text);       // returns text or null
const guard = await ai.guard(text);     // returns { ok, risk, findings }
const score = ai.riskScore(text);       // injection risk 0-1

---

Performance

• Verification: <100ms for typical responses
• PII Detection: <50ms
• Input Validation: <10ms
• Memory: <50MB baseline
• Bundle: No heavy ML models — pattern-based, runs anywhere Node runs

---

Requirements

• Node.js: ≥18.0.0
• OS: Windows, Linux, macOS
• Memory: 512MB minimum
• Disk: 50MB

---

Quick Reference

Task	Function	Example
Verify AI output	verify()	await verify(response)
Verify (string)	verify()	await verify("text")
Check input	isInputSafe()	isInputSafe(message)
Remove PII	redactPII()	redactPII(text)
Fix JSON	repairJSON()	repairJSON(broken)
Classify	classify()	classify(prompt, response)
Monitor health	monitorLLM()	monitorLLM(client)
Sentinel tests	sentinel.quick()	await sentinel.quick(client, model)
Guard (Zod-like)	guard()	await guard(text)
Safe check	safe()	await safe(text)
Plugin	use()	use(createRegexPlugin(...))
Check usage	CLI	npx llmverify usage
Audit	audit.log()	audit.log(event)
Badge	generateBadge()	generateBadge(result)
Adapter	createAdapter()	createAdapter({ provider: 'openai', apiKey })

---

Documentation

• Quick Start Guide — IDE setup in 5 minutes
• Integration Guide — Add to your app
• API Reference — Complete API docs
• Risk Levels — What scores mean
• Error Guide — Fix common errors
• CLI Reference — Command line tools
• AI Guide — Help AI assistants use llmverify

---

Support

• Issues: GitHub Issues
• Docs: Full Documentation
• Examples: /examples

---

License

MIT License — See LICENSE for details.

---

Made with care for AI safety and developer experience.