Bug: /api/v1/prompts/models returns HTTP 500 when PDD_MODEL_DEFAULT is not set

[Serhan-Asad]

Description

The /api/v1/prompts/models endpoint in pdd connect crashes with HTTP 500 for all users who don't have the PDD_MODEL_DEFAULT environment variable set.

Since PDD_MODEL_DEFAULT is undocumented and not configured by pdd setup, .pddrc, .env.example, or any setup path, this effectively means every user hits this bug when the frontend calls this endpoint.

Root Cause

DEFAULT_BASE_MODEL is None when the env var is unset (llm_invoke.py:635):

DEFAULT_BASE_MODEL = os.getenv("PDD_MODEL_DEFAULT", None)

The endpoint passes this directly to a Pydantic model that requires str (server/routes/prompts.py:94-97):

class ModelsResponse(BaseModel):
    models: list[ModelInfo] = Field(...)
    default_model: str = Field(...)  # rejects None

Result: ValidationError → HTTP 500.

Reproduction

# 1. Ensure PDD_MODEL_DEFAULT is not set (default for all users)
unset PDD_MODEL_DEFAULT

# 2. Start the server
pdd connect

# 3. Hit the endpoint
curl http://127.0.0.1:9876/api/v1/prompts/models

Response:

{
  "detail": "Error getting available models: 1 validation error for ModelsResponse\ndefault_model\n  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]"
}

Why Tests Don't Catch This

The existing test at tests/server/routes/test_prompts.py:129 mocks DEFAULT_BASE_MODEL to "claude-sonnet-4-20250514", so it never sees the None case:

mock_llm_invoke.DEFAULT_BASE_MODEL = "claude-sonnet-4-20250514"

Impact

• The Dev Units view model selector fails to load for every user
• PDD_MODEL_DEFAULT is not documented anywhere (README, help text, setup guide, .env.example)
• No setup path configures it — users have no way to know it exists

Suggested Fix

Either:

1. Make default_model optional: default_model: Optional[str] = Field(None, ...)
2. Or fall back to the first model from CSV when DEFAULT_BASE_MODEL is None:

   default_model=DEFAULT_BASE_MODEL or (models[0].name if models else "")

Option 2 is better UX since the frontend always gets a usable default.

[Serhan-Asad]

Step 1: Duplicate Check

Status: No duplicates found

Search Performed

• Searched for: "500 models endpoint", "PDD_MODEL_DEFAULT", "ModelsResponse ValidationError", "prompts models 500"
• Issues reviewed: 0 potential matches (only this issue returned)

Findings

This is a new issue. No existing open or closed issues report the same /api/v1/prompts/models HTTP 500 error or reference PDD_MODEL_DEFAULT / DEFAULT_BASE_MODEL being None.

---

Proceeding to Step 2: Documentation Check

[Serhan-Asad]

Step 2: Documentation Check

Status: Confirmed Bug

Documentation Reviewed

• CLAUDE.md (project docs) - PDD_MODEL_DEFAULT not listed in environment variables
• pdd/llm_invoke.py - DEFAULT_BASE_MODEL = os.getenv("PDD_MODEL_DEFAULT", None)
• pdd/server/routes/prompts.py:97 - default_model: str = Field(...) rejects None
• pdd/server/routes/prompts.py:514 - passes DEFAULT_BASE_MODEL directly without fallback

Findings

• PDD_MODEL_DEFAULT is not documented in CLAUDE.md, README, or any configuration guide
• No setup path (pdd setup, .pddrc, .env.example) configures this variable
• The endpoint unconditionally passes a potentially None value to a Pydantic model requiring str
• This affects all users who haven't manually discovered and set this undocumented env var

This is a clear bug: an HTTP 500 on a standard API endpoint with no user-actionable workaround documented anywhere.

---

Proceeding to Step 3: Triage

[Serhan-Asad]

Step 3: Triage

Status: Proceed

Summary

The bug report provides complete information for reproduction:

• Expected: /api/v1/prompts/models returns a valid JSON response with available models and a default model
• Actual: HTTP 500 due to Pydantic validation error when PDD_MODEL_DEFAULT is unset
• Reproduction steps: Clearly provided (unset env var, start server, curl endpoint)
• Code locations identified: llm_invoke.py:635, server/routes/prompts.py:94-97
• Root cause hypothesized: DEFAULT_BASE_MODEL = None passed to default_model: str field

All information needed to reproduce and fix is present.

---

Proceeding to Step 4: Reproduction

[Serhan-Asad]

Step 4: Reproduction

Status: Reproduced

Environment

• macOS Darwin 24.3.0
• Python 3.11+, Pydantic 2.11.4
• PDD_MODEL_DEFAULT not set (default state for all users)

Steps Taken

1. Ensured PDD_MODEL_DEFAULT is unset (unset PDD_MODEL_DEFAULT)
2. Verified DEFAULT_BASE_MODEL is None when imported from pdd.llm_invoke
3. Attempted to construct ModelsResponse(models=[], default_model=None)

Observed Behavior

1 validation error for ModelsResponse
default_model
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]

Exact match with the reported error.

Affected Files

• pdd/llm_invoke.py:635 - DEFAULT_BASE_MODEL = os.getenv("PDD_MODEL_DEFAULT", None) returns None when unset
• pdd/server/routes/prompts.py:94-97 - ModelsResponse.default_model: str rejects None
• pdd/server/routes/prompts.py:512-514 - Endpoint passes DEFAULT_BASE_MODEL directly without null check

Notes

• Bug is 100% reproducible on every install since PDD_MODEL_DEFAULT is undocumented and never configured by any setup path
• The existing test mocks DEFAULT_BASE_MODEL to a valid string, masking this issue

---

Proceeding to Step 5: Root Cause Analysis

[Serhan-Asad]

Step 5: Root Cause Analysis

Root Cause Identified: Yes

Summary

When PDD_MODEL_DEFAULT is not set (the default for all users), DEFAULT_BASE_MODEL resolves to None. This None is passed directly to ModelsResponse(default_model=DEFAULT_BASE_MODEL), but the Pydantic model declares default_model: str = Field(...) which rejects None, causing a ValidationError that surfaces as HTTP 500.

Technical Details

• Location 1: pdd/llm_invoke.py:635

  DEFAULT_BASE_MODEL = os.getenv("PDD_MODEL_DEFAULT", None)

  Returns None when env var is unset.

• Location 2: pdd/server/routes/prompts.py:94-97

  class ModelsResponse(BaseModel):
      models: list[ModelInfo] = Field(...)
      default_model: str = Field(...)  # rejects None

• Location 3: pdd/server/routes/prompts.py:512-515

  return ModelsResponse(
      models=models,
      default_model=DEFAULT_BASE_MODEL,  # passes None
  )

• Why it fails: Pydantic v2 strict type validation rejects None for a str field, raising ValidationError.

External Research

• Search terms: "pydantic Field None str validation error", "FastAPI 500 pydantic optional field"
• Sources checked: Pydantic v2 docs, FastAPI error handling docs
• Relevant findings: Standard Pydantic behavior — str fields reject None unless declared Optional[str]. This is a well-known pattern when env vars are unset.

Repository History

• Introduced by: Commit 1974a161 (Greg Tanaka, 2026-01-19) — initial server routes implementation
• Never caught: The test at tests/server/routes/test_prompts.py:129 mocks DEFAULT_BASE_MODEL to "claude-sonnet-4-20250514", always hiding the None case
• Related PRs: No PRs have modified the ModelsResponse construction since initial commit

Experiments Performed

1. Confirmed DEFAULT_BASE_MODEL is None when env var is unset by reading llm_invoke.py:635
2. Confirmed ModelsResponse.default_model is typed str (not Optional[str]) at prompts.py:97
3. Confirmed the endpoint passes DEFAULT_BASE_MODEL directly without null-check at prompts.py:514

Fix Direction

At prompts.py:514, fall back to the first model from the CSV when DEFAULT_BASE_MODEL is None:

default_model=DEFAULT_BASE_MODEL or (models[0].model if models else ""),

This gives the frontend a usable default without requiring an undocumented env var. The Pydantic model type stays str (no need for Optional).

---

Proceeding to Step 5.5: Prompt Classification

[Serhan-Asad]

Step 5.5: Prompt Classification

Classification: Code Bug

DEFECT_TYPE: code

Analysis

The server route code in pdd/server/routes/prompts.py is hand-coded (not generated from a .prompt file). The /api/v1/prompts/models endpoint passes DEFAULT_BASE_MODEL (which is None when PDD_MODEL_DEFAULT is unset) directly to a Pydantic model requiring str, without any fallback.

Evidence

• Prompt specifies: N/A — no .prompt file generates the server routes; they are hand-coded
• Code implements: Passes DEFAULT_BASE_MODEL directly to ModelsResponse(default_model=...) with no None handling
• User expects: The endpoint should return a usable default model (e.g., the first model from CSV) when PDD_MODEL_DEFAULT is unset

Conclusion

The code does not handle the common case where DEFAULT_BASE_MODEL is None. This is a straightforward code bug — a missing fallback. Proceeding with test generation.

---

Proceeding to Step 6: Test Plan

[Serhan-Asad]

Step 6: Test Plan

Existing Test Coverage

• Test file: tests/server/routes/test_prompts.py
• Current coverage: test_get_available_models (line 622) tests the happy path with DEFAULT_BASE_MODEL mocked to "claude-sonnet-4-20250514" (line 129 of fixture)
• Gap: DEFAULT_BASE_MODEL is never set to None, so the bug (Pydantic ValidationError when env var is unset) is never exercised

Proposed Tests

Test 1: test_get_available_models_default_model_none

• Input: DEFAULT_BASE_MODEL = None (the default when PDD_MODEL_DEFAULT env var is unset)
• Expected: Endpoint returns HTTP 200 with a valid ModelsResponse (falling back to first model from CSV or empty string)
• Actual (before fix): Pydantic ValidationError → HTTP 500

Test 2: test_get_available_models_empty_model_list_and_no_default

• Input: DEFAULT_BASE_MODEL = None AND _load_model_data returns an empty DataFrame
• Expected: Endpoint returns HTTP 200 with empty models list and a fallback default_model (e.g., "")
• Actual (before fix): Pydantic ValidationError → HTTP 500

Test 3: test_get_available_models_default_model_set (existing test — no change needed)

• Input: DEFAULT_BASE_MODEL = "claude-sonnet-4-20250514"
• Expected: Returns that model as default_model — already passes

Test Location

• File: tests/server/routes/test_prompts.py (append after existing test_get_available_models)
• Framework: pytest + pytest-asyncio
• Pattern: Uses existing prompts_test_env fixture, overriding mock_llm_invoke.DEFAULT_BASE_MODEL = None

Notes

• The fixture at line 129 hardcodes DEFAULT_BASE_MODEL = "claude-sonnet-4-20250514" — new tests must override this to None after fixture setup
• Tests call get_available_models() directly (not via HTTP client), matching existing test patterns
• Test 1 is the primary regression test; Test 2 covers the edge case of no models + no default

---

Proceeding to Step 7: Generate Test

[Serhan-Asad]

Step 7: Generated Test

Test File

tests/server/routes/test_prompts.py (append)

Test Code

@pytest.mark.asyncio
async def test_get_available_models_default_model_none(prompts_test_env):
    """Test that get_available_models does not crash when DEFAULT_BASE_MODEL is None."""
    import pandas as pd
    get_available_models = prompts_test_env["get_available_models"]
    mock_llm_invoke = prompts_test_env["mock_llm_invoke"]
    mock_llm_invoke.DEFAULT_BASE_MODEL = None
    mock_df = pd.DataFrame([{
        "model": "claude-sonnet-4-20250514", "provider": "Anthropic",
        "input": 3.0, "output": 15.0, "coding_arena_elo": 1400,
        "max_reasoning_tokens": 0, "reasoning_type": "none", "structured_output": True,
    }])
    mock_llm_invoke._load_model_data.return_value = mock_df
    response = await get_available_models()
    assert len(response.models) == 1
    assert response.default_model is not None
    assert isinstance(response.default_model, str)

@pytest.mark.asyncio
async def test_get_available_models_default_model_none_empty_models(prompts_test_env):
    """Test None default + empty model list."""
    import pandas as pd
    get_available_models = prompts_test_env["get_available_models"]
    mock_llm_invoke = prompts_test_env["mock_llm_invoke"]
    mock_llm_invoke.DEFAULT_BASE_MODEL = None
    mock_llm_invoke._load_model_data.return_value = pd.DataFrame()
    response = await get_available_models()
    assert len(response.models) == 0
    assert isinstance(response.default_model, str)

What This Test Verifies

Both tests set DEFAULT_BASE_MODEL = None (simulating unset PDD_MODEL_DEFAULT env var) and call get_available_models(). On the current buggy code, this triggers a Pydantic ValidationError → HTTP 500. Once fixed (with a fallback), the tests will pass.

Running the Test

pytest tests/server/routes/test_prompts.py::test_get_available_models_default_model_none tests/server/routes/test_prompts.py::test_get_available_models_default_model_none_empty_models -v

---

Proceeding to Step 8: Verification

[Serhan-Asad]

Step 8: Verification Complete

Test Execution

FAILED tests/server/routes/test_prompts.py::test_get_available_models_default_model_none
  fastapi.exceptions.HTTPException: 500: Error getting available models: 1 validation error for ModelsResponse
  default_model
    Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]

FAILED tests/server/routes/test_prompts.py::test_get_available_models_default_model_none_empty_models
  (same ValidationError)

2 failed in 0.50s

Verification Status

PASS: Test correctly detects the bug

Both tests fail with the exact Pydantic ValidationError described in the issue — DEFAULT_BASE_MODEL=None is passed to ModelsResponse(default_model=...) which requires str.

Summary

Step	Result
Duplicate Check	✅ No duplicates
Documentation	✅ Confirmed undocumented
Triage	✅ Proceed
Reproduction	✅ Bug reproduced
Root Cause	✅ Identified
Prompt Classification	✅ Code bug (not prompt)
Test Plan	✅ Plan created
Test Generation	✅ Tests written
Verification	✅ Tests correctly detect the bug

Bug Details

• Location: pdd/server/routes/prompts.py:514 (return ModelsResponse(... default_model=DEFAULT_BASE_MODEL))
• Root Cause: DEFAULT_BASE_MODEL is None when PDD_MODEL_DEFAULT env var is unset; Pydantic rejects None for default_model: str
• Test File: tests/server/routes/test_prompts.py (two new tests: test_get_available_models_default_model_none, test_get_available_models_default_model_none_empty_models)

Next Steps

1. Fix the bug: add fallback DEFAULT_BASE_MODEL or (models[0].model if models else "")
2. Run the new tests to confirm the fix
3. Run full test suite to check for regressions
4. Submit PR with fix and test

---

Proceeding to Step 9: E2E Tests

[Serhan-Asad]

Step 9: E2E Test

Environment Discovery

• Repository language: Python
• Test framework: pytest + FastAPI TestClient
• Existing E2E tests found: No dedicated E2E directory; tests/server/test_app.py uses TestClient pattern
• Test configuration: pyproject.toml (pytest section)
• Fixtures/helpers used: Custom e2e_app fixture with sys.modules mocking (matches existing pattern in test_prompts.py)
• Environment variables required: None (the bug is that PDD_MODEL_DEFAULT is not set)
• Auth/credentials: Mocked (security module patched out)

E2E Test File

tests/server/routes/test_prompts_e2e.py

Test Type

API (HTTP endpoint via FastAPI TestClient)

What This Test Verifies

When a user (or the frontend) calls GET /api/v1/prompts/models and PDD_MODEL_DEFAULT is not set (the default for all users), the endpoint should return HTTP 200 with a valid models list — not HTTP 500.

Test Code

def test_models_endpoint_returns_200_when_default_model_none(e2e_app):
    """GET /api/v1/prompts/models should return 200 when DEFAULT_BASE_MODEL is None."""
    response = e2e_app.get("/api/v1/prompts/models")
    assert response.status_code == 200
    data = response.json()
    assert "models" in data
    assert "default_model" in data
    assert isinstance(data["default_model"], str)

def test_models_endpoint_error_message_on_500(e2e_app):
    """Verify the exact Pydantic ValidationError users see in the 500 response."""
    response = e2e_app.get("/api/v1/prompts/models")
    if response.status_code == 500:
        detail = response.json().get("detail", "")
        assert "default_model" in detail
        pytest.fail(f"Bug #85 confirmed: endpoint returns 500 with ValidationError: {detail}")

Test Execution Result

FAILED test_models_endpoint_returns_200_when_default_model_none - AssertionError: Expected 200 but got 500.
  Response: {'detail': 'Error getting available models: 1 validation error for ModelsResponse\ndefault_model\n  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]'}

FAILED test_models_endpoint_error_message_on_500 - Failed: Bug #85 confirmed: endpoint returns 500 with ValidationError:
  default_model - Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]

============================== 2 failed in 0.49s ===============================

E2E Test Status

PASS: E2E test correctly detects the bug — Both tests fail on buggy code, confirming HTTP 500 from the Pydantic ValidationError when DEFAULT_BASE_MODEL is None. Tests will pass once the fix adds a fallback for None.

---

Proceeding to Step 10: Create Draft PR

[Serhan-Asad]

Step 10: Draft PR Created

Pull Request

PR #112: Add failing tests for #85: models endpoint HTTP 500

Branch

fix/issue-85

What's Included

• Failing unit test at tests/server/routes/test_prompts.py
• Failing E2E test at tests/server/routes/test_prompts_e2e.py
• Commits: 1

Next Steps for Maintainers

1. Review the failing tests to understand the expected behavior
2. Implement the fix at the identified location (pdd/server/routes/prompts.py — add None fallback for DEFAULT_BASE_MODEL)
3. Verify both unit and E2E tests pass with your fix
4. Run full test suite to check for regressions
5. Mark the PR as ready for review

PDD Fix Command

To auto-fix this bug using PDD:

pdd fix https://github.com/Serhan-Asad/pdd/issues/85

Tip: Use --protect-tests if the tests are known to be correct, or --max-cycles N to limit fix attempts.

---

Investigation complete. A draft PR with failing tests has been created and linked to this issue.

[Serhan-Asad]

Step 1: Unit Test Execution (Cycle 1)

Status: All Tests Pass

Unit Tests Identified

• tests/server/routes/test_prompts.py::test_get_available_models_default_model_none
• tests/server/routes/test_prompts.py::test_get_available_models_default_model_none_empty_models

Initial Test Results

• Passed: 0
• Failed: 2

Both tests failed with fastapi.exceptions.HTTPException: 500 due to Pydantic ValidationError — DEFAULT_BASE_MODEL is None when PDD_MODEL_DEFAULT env var is unset, and ModelsResponse.default_model: str rejects None.

Fix Applied

File: pdd/server/routes/prompts.py:514

Changed:

default_model=DEFAULT_BASE_MODEL,

To:

default_model=DEFAULT_BASE_MODEL or (models[0].model if models else ""),

This falls back to the first model from the CSV when DEFAULT_BASE_MODEL is None, or empty string if no models are available. No pdd fix --manual used since this is hand-coded server route code (no .prompt file).

pdd fix Runs

Dev Unit	Result
server/routes/prompts	Fixed (direct edit — no prompt file)

Final Test Results

• Passed: 2
• Failed: 0

Full regression check: all 31 tests in test_prompts.py pass.

---

Proceeding to Step 2: E2E Test Check

[Serhan-Asad]

Step 2: E2E Test Check (Cycle 1)

Status: ALL_TESTS_PASS

E2E Tests Run

• tests/server/routes/test_prompts_e2e.py::test_models_endpoint_returns_200_when_default_model_none ✓
• tests/server/routes/test_prompts_e2e.py::test_models_endpoint_error_message_on_500 ✓

Summary

All 2 e2e tests pass. Bug fix complete!

---

Workflow complete - all tests passing