GitHub - Crashlens/crashlens_detector: Core CLI engine that scans JSONL logs for LLM cost-waste patterns — retry loops, fallback chains, and overkill model usage — with built-in detection rules.

🧠 What is Crashens Detector?

Crashens Detector is a developer tool to analyze GPT API logs and uncover hidden token waste, retry loops, and overkill model usage. It helps you optimize your OpenAI, Anthropic, or Langfuse API usage by generating a cost breakdown and suggesting cost-saving actions.

🔍 Use it when you want to:

Understand how your GPT API budget is being spent
Reduce unnecessary model calls or retries
Audit logs for fallback logic inefficiencies
Analyze Langfuse/OpenAI JSONL logs locally, with full privacy

🧾 Supports: OpenAI, Anthropic, Langfuse JSONL logs
💻 Platform: 100% CLI, 100% local

💡 Why use Crashens Detector?

"You can't optimize what you can't see." Crashens Detector gives you visibility into how you're actually using LLMs — and how much it's costing you.

👨‍💻 Use Cases

Track and reduce monthly OpenAI bills
Debug retry loops and fallback logic in LangChain or custom agents
Detect inefficient prompt-to-model usage (e.g., using GPT-4 for 3-token completions)
Generate token audit logs for compliance or team analysis
CLI tool to audit GPT usage and optimize OpenAI API costs
Analyze GPT token usage and efficiency in LLM logs
Reduce LLM spending with actionable insights

TL;DR

pip install crashlens-detector
crashlens scan path/to/your-logs.jsonl
# Generates report.md with per-trace waste, cost, and suggestions

⚠️ Python Requirement

Crashens Detector requires Python 3.12 or higher. Download Python 3.12+ here.

⚠️ Windows PATH Warning

If you see a warning like:

WARNING: The script crashlens-detector.exe is installed in 'C:\Users\<user>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

This means the crashlens command may not work from any folder until you add the above Scripts directory to your system PATH.

How to fix:

Copy the path shown in the warning (ending with \Scripts).
Open the Windows Start menu, search for "Environment Variables", and open "Edit the system environment variables".
Click "Environment Variables...".
Under "User variables" or "System variables", select Path and click "Edit".
Click "New" and paste the Scripts path.
Click OK to save. Restart your terminal/command prompt.

Now you can run crashlens-detector from any folder.

Crashens Detector analyzes your logs for patterns like fallback failures, retry loops, and overkill model usage, and generates a detailed Markdown report (report.md) with cost breakdowns and actionable insights.

📝 Example Crashens Detector Report

Below is a sample of what the actual report.md looks like after running Crashens Detector:

🚨 Crashens Detector Token Waste Report 🚨 📊 Analysis Date: 2025-07-31 15:24:48

Metric	Value
Total AI Spend	$1.18
Total Potential Savings	$0.82
Wasted Tokens	19,831
Issues Found	73
Traces Analyzed	156

❓ Overkill Model | 59 traces | $0.68 wasted | Fix: optimize usage 🎯 Wasted tokens: 16,496 🔗 Traces (57): trace_overkill_01, trace_norm_02, trace_overkill_02, trace_overkill_03, trace_norm_06, +52 more

📢 Fallback Failure | 7 traces | $0.08 wasted | Fix: remove redundant fallbacks 🎯 Wasted tokens: 1,330 🔗 Traces (7): trace_fallback_success_01, trace_fallback_success_02, trace_fallback_success_03, trace_fallback_success_04, trace_fallback_success_05, +2 more

⚡ Fallback Storm | 5 traces | $0.07 wasted | Fix: optimize model selection 🎯 Wasted tokens: 1,877 🔗 Traces (5): trace_fallback_failure_01, trace_fallback_failure_02, trace_fallback_failure_03, trace_fallback_failure_04, trace_fallback_failure_05

🔄 Retry Loop | 2 traces | $0.0001 wasted | Fix: exponential backoff 🎯 Wasted tokens: 128 🔗 Traces (2): trace_retry_loop_07, trace_retry_loop_10

Top Expensive Traces

Rank	Trace ID	Model	Cost
1	trace_norm_76	gpt-4	$0.09
2	trace_norm_65	gpt-4	$0.07
3	trace_norm_38	gpt-4	$0.06

Cost by Model

Model	Cost	Percentage
gpt-4	$1.16	98%
gpt-3.5-turbo	$0.02	2%

🚀 Features

Detects token waste patterns: fallback failures, retry loops, overkill/short completions
Supports OpenAI, Anthropic, and Langfuse-style logs (JSONL)
Robust error handling for malformed or incomplete logs
Configurable model pricing and thresholds via pricing.yaml
Generates a professional Markdown report (report.md) after every scan
100% local: No data leaves your machine

1. Clone the Repository

Replace <repo-link> with the actual GitHub URL:

git clone <repo-link>
cd crashlens

2. Install Python & Poetry

Crashens Detector requires Python 3.8+ and Poetry for dependency management.

MacOS

Install Python (if not already):
```
brew install python@3.12
```

Install Poetry (stable version):

curl -sSL https://install.python-poetry.org | python3 - --version 1.8.2
# Or with Homebrew:
brew install poetry

Add Poetry to your PATH if needed:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zprofile
source ~/.zprofile

Verify installation:

poetry --version
# Should show: Poetry (version 1.8.2)

Windows

⚠️ Use PowerShell, not CMD, for these commands.

Install Python from python.org

Install Poetry (stable version):

(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | python - --version 1.8.2

Add Poetry to your PATH if poetry --version returns "not found":

$userPoetryBin = "$HOME\AppData\Roaming\Python\Scripts"

if (Test-Path $userPoetryBin -and -not ($env:Path -like "*$userPoetryBin*")) {
    $env:Path += ";$userPoetryBin"
    [Environment]::SetEnvironmentVariable("Path", $env:Path, "User")
    Write-Output "✅ Poetry path added. Restart your terminal."
} else {
    Write-Output "⚠️ Poetry path not found or already added. You may need to locate poetry.exe manually."
}

⚠️ Restart your terminal/PowerShell after adding to PATH.

Verify installation:

poetry --version
# Should show: Poetry (version 1.8.2)

3. Set Up the Environment

# From the project root:
poetry install

This will create a virtual environment and install all dependencies.

To activate the environment :

poetry shell

4. Running Crashens Detector

You can run Crashens Detector via Poetry or as a Python module:

Basic Scan (from file)

crashlens scan examples/retry-test.jsonl

Demo Mode (built-in sample data)

crashlens scan --demo

🚨 Crashens Detector Token Waste Report 🚨 📊 Analysis Date: 2025-07-31 15:22:08

Metric	Value
Total AI Spend	$0.09
Total Potential Savings	$0.07
Wasted Tokens	1,414
Issues Found	8
Traces Analyzed	12

📢 Fallback Failure | 5 traces | $0.07 wasted | Fix: remove redundant fallbacks 🎯 Wasted tokens: 1,275 🔗 Traces (5): demo_fallback_01, demo_fallback_02, demo_fallback_03, demo_fallback_04, demo_fallback_05

❓ Overkill Model | 2 traces | $0.0007 wasted | Fix: optimize usage 🎯 Wasted tokens: 31 🔗 Traces (2): demo_overkill_01, demo_overkill_02

🔄 Retry Loop | 1 traces | $0.0002 wasted | Fix: exponential backoff 🎯 Wasted tokens: 108 🔗 Traces (1): demo_retry_01

Top Expensive Traces

Rank	Trace ID	Model	Cost
1	demo_norm_03	gpt-4	$0.03
2	demo_norm_04	gpt-4	$0.02
3	demo_fallback_05	gpt-3.5-turbo	$0.02

Cost by Model

Model	Cost	Percentage
gpt-4	$0.09	99%
gpt-3.5-turbo	$0.0012	1%

Why Crashens Detector? (vs. grep + Excel, LangSmith, or basic logging)

🔁 grep + spreadsheet: Too manual, error-prone, no cost context
💸 LangSmith: Powerful but complex, requires full tracing/observability stack
🔍 Logging without cost visibility: You miss $ waste and optimization opportunities
🔒 Crashens Detector runs 100% locally—no data leaves your machine.

Features (Ultra-Specific)

✅ Detects retry-loop storms across trace IDs
✅ Flags gpt-4, Claude, Gemini, and other expensive model usage where a cheaper model (e.g., gpt-3.5, Claude Instant) would suffice
✅ Scans stdin logs from LangChain, LlamaIndex, custom logging
✅ Generates Markdown cost reports with per-trace waste

What Makes Crashens Detector Different?

💵 Model pricing fallback (auto-detects/corrects missing cost info)
🔒 Security-by-design (runs 100% locally, no API calls, no data leaves your machine)
🚦 Coming soon: Policy enforcement, live CLI firewall, more integrations

📄 Log File Structure

Your logs must be in JSONL format (one JSON object per line) and follow this structure:

{"traceId": "trace_9",  "startTime": "2025-07-19T10:36:13Z", "input": {"model": "gpt-3.5-turbo", "prompt": "How do solar panels work?"}, "usage": {"prompt_tokens": 25, "completion_tokens": 110, "total_tokens": 135}, "cost": 0.000178}

Each line is a separate API call (no commas or blank lines between objects).
Fields must be nested as shown: input.model, input.prompt, usage.completion_tokens, etc.

Required fields:

traceId (string): Unique identifier for a group of related API calls
input.model (string): Model name (e.g., gpt-4, gpt-3.5-turbo)
input.prompt (string): The prompt sent to the model
usage.completion_tokens (int): Number of completion tokens used

Optional fields:

cost (float): Cost of the API call
name, startTime, etc.: Any other metadata

💡 Crashens Detector expects JSONL with per-call metrics (model, tokens, cost). Works with LangChain logs, OpenAI api.log, Claude, Gemini, and more.

🚀 Usage: Command Line Examples

After installation, use the crashlens command in your terminal (or python -m crashlens if running from source).

1. Scan a log file

crashlens scan path/to/your-logs.jsonl

Scans the specified log file and generates a report.md in your current directory.

2. Demo mode (built-in sample data)

crashlens scan --demo

Runs analysis on built-in example logs (requires examples-logs/demo-logs.jsonl file).
Note: If installing from PyPI, you'll need to create sample logs or use your own data.
From source: Demo data is included in the repository.

3. Scan from stdin (pipe)

cat path/to/your-logs.jsonl | crashlens scan --stdin

Reads logs from standard input (useful for pipelines or quick tests).

4. Paste logs interactively

crashlens scan --paste

Reads JSONL data from clipboard (paste and press Enter to finish).

5. Generate detailed category reports

crashlens scan --detailed

Creates grouped JSON files in detailed_output/ by issue type (fallback_failure.json, retry_loop.json, etc.).

6. Get cost summaries

crashlens scan --summary          # Cost summary with breakdown
crashlens scan --summary-only     # Summary without trace IDs

Shows cost analysis with or without detailed trace information.

7. Change output format

crashlens scan --format json      # JSON output
crashlens scan --format markdown  # Markdown format

Default format is slack for team sharing.

8. Get help

crashlens --help
crashlens scan --help

Shows all available options and usage details.

🧩 Example Workflow

Install Crashens Detector:

pip install crashlens
# OR clone and install from source as above

Scan your logs:

crashlens scan path/to/your-logs.jsonl
# OR
python -m crashlens scan path/to/your-logs.jsonl

Open report.md in your favorite Markdown viewer or editor to review the findings and suggestions.

📝 Logging Helper

To make log analysis seamless, you can use our crashlens-logger package to emit logs in the correct structure for Crashens Detector. This ensures compatibility and reduces manual formatting.

Example usage:

pip install --upgrade crashlens_logger

from crashlens_logger import CrashLensLogger

logger = CrashLensLogger()
logger.log_event(
    traceId=trace_id,
    startTime=start_time,
    endTime=end_time,
    input={"model": model, "prompt": prompt},
    usage=usage
    # Optionally add: type, level, metadata, name, etc.
)

The logger writes each call as a JSONL line in the required format.
See the crashlens-logger repo for full docs and advanced usage.

🆘 Troubleshooting & Tips

File not found: Make sure the path to your log file is correct.
No traces found: Your log file may be empty or not in the expected format.
Cost is $0.00: Check that your log’s model names match those in the pricing config.
Virtual environment issues: Make sure you’re using the right Python environment.
Need help? Use crashlens --help for all options.

🛠️ Full Installation (Advanced/Dev)

Alternative: Install from Source (GitHub)

If you want the latest development version or want to contribute, you can install Crashens Detector from source:

Clone the repository:
```
git clone <repo-link>
cd crashlens
```

(Optional but recommended) Create a virtual environment:

On Mac/Linux:

python3 -m venv .venv
source .venv/bin/activate

On Windows:

python -m venv .venv
.venv\Scripts\activate

Install dependencies:

pip install -r requirements.txt
# Or, if using Poetry:
poetry install

Run Crashens Detector:

python -m crashlens scan path/to/your-logs.jsonl
# Or, if using Poetry:
poetry run crashlens scan path/to/your-logs.jsonl

📬 Support

For questions, issues, or feature requests, open an issue on GitHub or contact the maintainer.

📄 License

MIT License - see LICENSE file for details.

Crashens Detector: Find your wasted tokens. Save money. Optimize your AI usage.

Scan from stdin (pipe or paste)

cat examples/retry-test.jsonl | poetry run crashlens scan --stdin

5. Output: The Markdown Report

After every scan, Crashens Detector creates or updates report.md in your current directory.

Example Structure

# Crashens Detector Token Waste Report

🧾 **Total AI Spend**: $0.123456
💰 **Total Potential Savings**: $0.045678

| Trace ID | Model | Prompt | Completion Length | Cost | Waste Type |
|----------|-------|--------|------------------|------|------------|
| trace_001 | gpt-4 | ... | 3 | $0.00033 | Overkill |
| ...      | ...   | ...    | ...              | ...  | ...        |

## Overkill Model Usage (5 issues)
- ...

## Retry Loops (3 issues)
- ...

## Fallback Failures (2 issues)
- ...

6. Troubleshooting

File not found: Ensure the path to your log file is correct.
No traces found: Your log file may be empty or malformed.
Cost is $0.00: Check that your pricing.yaml matches the model names in your logs.
Virtual environment issues: Use poetry run to ensure dependencies are available.

7. Example Commands

# Scan a log file
poetry run crashlens scan examples/demo-logs.jsonl

# Use demo data
poetry run crashlens scan --demo

# Scan from stdin
cat examples/demo-logs.jsonl | poetry run crashlens scan --stdin

📚 Complete Command Reference

Basic Usage

crashlens scan [OPTIONS] [LOGFILE]

🎯 Examples

# Scan a specific log file
crashlens scan logs.jsonl

# Run on built-in sample logs
crashlens scan --demo

# Pipe logs via stdin
cat logs.jsonl | crashlens scan --stdin

# Read logs from clipboard
crashlens scan --paste

# Generate detailed category JSON reports
crashlens scan --detailed

# Cost summary with categories
crashlens scan --summary

# Show summary only (no trace details)
crashlens scan --summary-only

🔧 All Options

Option	Description	Example
`-f, --format`	Output format: `slack`, `markdown`, `json`	`--format json`
`-c, --config`	Custom pricing config file path	`--config my-pricing.yaml`
`--demo`	Use built-in demo data (requires examples-logs/demo-logs.jsonl)	`crashlens scan --demo`
`--stdin`	Read from standard input	`cat logs.jsonl \| crashlens scan --stdin`
`--paste`	Read JSONL data from clipboard	`crashlens scan --paste`
`--summary`	Show cost summary with breakdown	`crashlens scan --summary`
`--summary-only`	Summary without trace IDs	`crashlens scan --summary-only`
`--detailed`	Generate detailed category JSON reports	`crashlens scan --detailed`
`--detailed-dir`	Directory for detailed reports (default: detailed_output)	`--detailed-dir my_reports`
`--help`	Show help message	`crashlens scan --help`

📂 Detailed Reports

When using --detailed, Crashens Detector generates grouped category files:

detailed_output/fallback_failure.json - All fallback failure issues
detailed_output/retry_loop.json - All retry loop issues
detailed_output/fallback_storm.json - All fallback storm issues
detailed_output/overkill_model.json - All overkill model issues

Each file contains:

Summary with total issues, affected traces, costs
All issues of that type with trace IDs and details
Specific suggestions for that category

🔍 Input Sources

Crashens Detector supports multiple input methods:

File input: crashlens scan path/to/logs.jsonl
Demo mode: crashlens scan --demo (requires examples-logs/demo-logs.jsonl file)
Standard input: cat logs.jsonl | crashlens scan --stdin
Clipboard: crashlens scan --paste (paste logs interactively)

📊 Output Formats

slack (default): Slack-formatted report for team sharing
markdown: Clean Markdown for documentation
json: Machine-readable JSON for automation

💡 Pro Tips

Use --demo to test Crashens Detector without your own logs
Use --detailed to get actionable JSON reports for each issue category
Use --summary-only for executive summaries without trace details
Combine --stdin with shell pipelines for automation

🤝 Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

Quick Start for Contributors

Fork and clone the repository
Set up development environment:
```
poetry install
```
Run tests:
```
poetry run pytest
```

Run linting:

poetry run black crashlens/ tests/
poetry run flake8 crashlens/ tests/

Test CLI:

poetry run crashlens scan examples-logs/demo-logs.jsonl

Development Requirements

Python 3.12+
Poetry for dependency management
Black for code formatting
Flake8 for linting
Pytest for testing
MyPy for type checking (optional)

Branch Protection & CI

This repository uses branch protection rules that require:

✅ Pull request reviews before merging
✅ Status checks to pass (CI tests, linting, etc.)
✅ Conversations resolved before merging
✅ Branch up-to-date before merging

All contributions must:

Pass automated tests
Follow code style guidelines
Include appropriate documentation
Be submitted via pull request (no direct pushes to main)

See CONTRIBUTING.md for detailed contribution guidelines.

🔒 Security

For security issues, please see our Security Policy.

8. Support

💬 Questions: Use GitHub Discussions
🐛 Bug Reports: Open a GitHub Issue
📧 Private Issues: Contact security@crashlens.dev

📄 License

MIT License - see LICENSE file for details.

Crashlens Detector: Find your wasted tokens. Save money. Optimize your AI usage. 🎯

Name		Name	Last commit message	Last commit date
Latest commit History 53 Commits
.github		.github
crashlens		crashlens
detailed_output		detailed_output
examples-logs		examples-logs
examples-reports		examples-reports
tests		tests
.gitignore		.gitignore
BRANCH_PROTECTION_IMPLEMENTATION.md		BRANCH_PROTECTION_IMPLEMENTATION.md
DETECTOR_PRIORITY_LOGIC.md		DETECTOR_PRIORITY_LOGIC.md
LAZY_EVALUATION.md		LAZY_EVALUATION.md
LICENSE		LICENSE
PRIORITY_SYSTEM_CHECKLIST.md		PRIORITY_SYSTEM_CHECKLIST.md
README.md		README.md
SECURITY.md		SECURITY.md
commands-report.md		commands-report.md
friend_conversation_simulation.md		friend_conversation_simulation.md
pyproject.toml		pyproject.toml
report.md		report.md
setup.cfg		setup.cfg

License

Crashlens/crashlens_detector

Folders and files

Latest commit

History

Repository files navigation

🧠 What is Crashens Detector?

🔍 Use it when you want to:

💡 Why use Crashens Detector?

👨‍💻 Use Cases

TL;DR

⚠️ Python Requirement

⚠️ Windows PATH Warning

📝 Example Crashens Detector Report

Top Expensive Traces

Cost by Model

🚀 Features

1. Clone the Repository

2. Install Python & Poetry

MacOS

Windows

3. Set Up the Environment

4. Running Crashens Detector

Basic Scan (from file)

Demo Mode (built-in sample data)

Top Expensive Traces

Cost by Model

Why Crashens Detector? (vs. grep + Excel, LangSmith, or basic logging)

Features (Ultra-Specific)

What Makes Crashens Detector Different?

📄 Log File Structure

🚀 Usage: Command Line Examples

1. Scan a log file

2. Demo mode (built-in sample data)

3. Scan from stdin (pipe)

4. Paste logs interactively

5. Generate detailed category reports

6. Get cost summaries

7. Change output format

8. Get help

🧩 Example Workflow

📝 Logging Helper

🆘 Troubleshooting & Tips

🛠️ Full Installation (Advanced/Dev)

Alternative: Install from Source (GitHub)

📬 Support

📄 License

Scan from stdin (pipe or paste)

5. Output: The Markdown Report

Example Structure

6. Troubleshooting

7. Example Commands

📚 Complete Command Reference

Basic Usage

🎯 Examples

🔧 All Options

📂 Detailed Reports

🔍 Input Sources

📊 Output Formats

💡 Pro Tips

🤝 Contributing

Quick Start for Contributors

Development Requirements

Branch Protection & CI

🔒 Security

8. Support

📄 License

About

Topics

Resources

License

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Packages