Replace Microsoft Entra ID with Keycloak as sole authentication provider

[Copilot]

Completely replaces Microsoft Entra ID with Keycloak as the exclusive identity provider across all application components. This is a full substitution (not a migration) that removes all Azure AD dependencies and simplifies the authentication architecture to use only Keycloak.

Changes

Terraform Configuration

• Removed: Complete removal of terraform/modules/azure-ad-app/ module
• Removed: Azure providers (azuread, azurerm) from terraform.tf and providers.tf
• Removed: All Azure-specific variables (azure_tenant_id, azure_client_id, azure_client_secret, jwt_oidc_discovery_url, jwt_bound_issuer)
• Updated: main.tf to use only Keycloak module for vault authentication
• Updated: outputs.tf to expose only Keycloak client IDs, secrets, and OIDC endpoints
• Simplified: Renamed ad_user_password to user_password for clarity

Keycloak Terraform Module (terraform/modules/keycloak/)

• Automated provisioning of Keycloak realm (confused-deputy-realm), clients, users, and groups
• OAuth2 token exchange configuration for on-behalf-of flows between products-web → products-agent → products-mcp
• Protocol mappers for group claims in JWT tokens
• Vault integration via OIDC discovery and group-to-policy mapping
• Test users: alice (readonly), bob (admin) with appropriate group memberships

Application Code

• Removed: products-agent/auth/entra_token_service.py and token_service_factory.py
• Updated: products-agent/main.py to directly use Keycloak token service
• Updated: products-web/app.py to use only Keycloak OAuth URLs with hardcoded endpoint construction
• Simplified: Removed all provider auto-detection logic

Infrastructure

• Docker Compose service for local Keycloak instance (quay.io/keycloak/keycloak:26.0)
• Simplified: export-env.sh with single provider support: ./export-env.sh [local|docker|aws]
• Removed dual-provider configuration options

Documentation

• Updated: README.md to describe Keycloak-only architecture
• Updated: terraform/README.md with complete removal of Azure/Entra ID references
• Updated: terraform.tfvars.example to show only Keycloak configuration variables
• Removed: Migration guide (no longer applicable)
• Removed: Implementation summary (no longer needed)

Architecture

The application now uses Keycloak exclusively for:

• OAuth2/OIDC authentication flows
• JWT token issuance and validation
• Token exchange for on-behalf-of delegation
• Group-based authorization with Vault policy mapping

Security Model Preservation

All zero-trust patterns maintained:

• JWT validation and token signature verification
• Audience and issuer checks using Keycloak endpoints
• Group-based Vault policies (dbread, dbadmin)
• Dynamic database credentials via HCP Vault
• On-behalf-of token exchange flow using RFC 8693 token-exchange grant type

Result

• Zero references to Microsoft Entra ID or Azure AD remain in the codebase
• Single authentication path through Keycloak
• Simplified configuration with no dual-provider logic
• Fully automated Terraform deployment of identity infrastructure
• No manual steps required beyond Terraform and Docker Compose

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Migrate Authentication from Microsoft Entra ID to Keycloak (Terraform-based)</issue_title>
<issue_description>Role: You are a DevOps automation engineer specialized in Identity and Access Management with Terraform, Keycloak, and multi-service application deployments.

Task:
Convert the current authentication and authorization setup from Microsoft Entra ID to Keycloak, maintaining all current security and agentic behavior. The new configuration must be fully automated through Terraform (under /terraform) and integrate with the existing multi-service app structure:

• products-web → Streamlit UI (Frontend)
• products-agent → FastAPI Agent API
• products-mcp → Model Context Protocol (MCP) Server

---

🧩 Current Architecture Summary

• Each service authenticates via OAuth2 + JWT validation using Microsoft Entra ID.
• products-agent acts as an OBO (On-Behalf-Of) intermediary, exchanging and validating tokens between the UI and MCP.
• products-mcp uses HCP Vault to generate dynamic database credentials and validate identity scopes.
• All configuration and environment variables are generated via Terraform (export-env.sh), and deployments occur via Docker Compose or AWS bastion host.

---

🎯 Goal

Replace Microsoft Entra ID with Keycloak as the identity provider across all components, keeping the same functional and security model.

---

🛠️ Required Actions

1. Terraform Integration

   • Use the Keycloak Terraform provider.
   • Create a new Terraform module under /terraform/keycloak that:
     • Deploys or references a Keycloak realm (e.g., confused-deputy-realm).
     • Creates clients for:
       • products-web (public client, PKCE flow)
       • products-agent (confidential client with client secret)
       • products-mcp (confidential client)
     • Defines realm roles for user groups (e.g., readonly, admin).
     • Adds protocol mappers for group/role claims in JWTs.
     • Configures OBO token exchange between clients using Keycloak’s token_exchange feature.
   • Expose Terraform outputs for each client ID and secret to be used in .env files.

2. Environment & App Configuration

   • Update each app’s .env files generated by export-env.sh:
     • Replace all MS_ENTRA_* variables with KEYCLOAK_* equivalents:
       • KEYCLOAK_URL, KEYCLOAK_REALM, KEYCLOAK_CLIENT_ID, KEYCLOAK_CLIENT_SECRET
     • Update redirect URIs and audience claims for each service.
   • Adjust JWT validation logic in products-agent and products-mcp:
     • Use Keycloak’s OpenID Connect discovery endpoint for public keys.
     • Replace Entra ID issuer/audience validation with Keycloak’s issuer.
     • Preserve the same OBO flow between products-web → products-agent → products-mcp.

3. Docker Compose Integration

   • Add a Keycloak service to docker-compose/ for local testing:

     keycloak:
       image: quay.io/keycloak/keycloak:latest
       command: start-dev --import-realm
       ports:
         - "8080:8080"
       environment:
         KEYCLOAK_ADMIN: admin
         KEYCLOAK_ADMIN_PASSWORD: admin

   • Mount the exported realm JSON file generated by Terraform (if applicable).

4. Documentation Updates

   • Update /README.md and /terraform/README.md:
     • Replace all mentions of Microsoft Entra ID with Keycloak.
     • Include instructions for initializing and applying the new Keycloak configuration:

       cd terraform
       terraform init
       terraform apply -target=module.keycloak
       ./export-env.sh local

     • Note: JWT validation and token delegation now use Keycloak OIDC endpoints.

---

🧾 Expected Output

• Complete Terraform configuration for Keycloak under /terraform/keycloak/.
• Updated .env templates for all apps (products-web, products-agent, products-mcp).
• Optional docker-compose snippet to run Keycloak locally.
• Updated documentation reflecting Keycloak-based auth flow.
• Maintained end-to-end secure OBO token exchange between services.

---

✅ Closure Conditions

• All apps can authenticate via Keycloak and communicate securely.
• Terraform deploys the realm, clients, and roles automatically.
• Token validation, delegation, and policy enforcement remain consistent with the previous Entra ID setup.
• No manual steps are required outside Terraform and Docker Compose.

</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Migrate Authentication from Microsoft Entra ID to Keycloak (Terraform-based) #1

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[shoootyou]

@copilot here's more instructions:

We’ll continue from the previous work on authentication and identity setup.
The goal is to completely replace Microsoft Entra ID (Azure AD) with Keycloak — this is not a migration, but a full substitution of the identity layer across all components.

---

🎯 Objective

Refactor the entire authentication and authorization system so that Keycloak becomes the only identity provider for:

• products-web (Frontend)
• products-agent (Agent API)
• products-mcp (MCP Server)

All identity provisioning, roles, and client registrations must be handled automatically with Terraform.
All apps must authenticate, validate, and exchange tokens exclusively via Keycloak.

---

🛠️ Implementation Guidelines

• Use the Keycloak Terraform provider to define the full realm, clients, and token exchange setup.
• Each app should have its own Keycloak client with proper roles, redirect URIs, and token exchange policies.
• Maintain the same security model and agentic flow:
  • JWT validation and signature checks
  • OBO (On-Behalf-Of) token delegation between products-web → products-agent → products-mcp
  • Group and role propagation for authorization
• Replace all Microsoft Entra ID–specific variables, endpoints, and logic with Keycloak equivalents (KEYCLOAK_URL, REALM, CLIENT_ID, CLIENT_SECRET, etc.).
• Update any code that currently references Entra ID or Microsoft Graph to use Keycloak OIDC endpoints instead.
• Adjust environment variables and documentation so that the setup runs end-to-end using only Keycloak.

---

🧩 Expected Behavior

• Terraform automatically provisions all necessary Keycloak resources.
• Each service authenticates and validates tokens through Keycloak.
• Token delegation between services continues to function securely.
• The Docker and AWS deployments remain compatible.
• Documentation reflects the new Keycloak-based identity system.

---

✅ Acceptance Criteria

• No references to Microsoft Entra ID remain anywhere in the project.
• Authentication and authorization rely solely on Keycloak.
• All existing functionality (OBO flow, JWT validation, Vault access) continues to work using Keycloak tokens.
• The setup can be applied, tested, and run without any manual configuration beyond Terraform and environment variables.

---

🚀 Next Step

Continue from the existing project structure and let Copilot decide what needs to be created, updated, or removed to complete the transition to Keycloak.

[Copilot]

@copilot here's more instructions:

We’ll continue from the previous work on authentication and identity setup.
...

✅ Complete replacement done! All Microsoft Entra ID references have been removed.

Changes Summary:

Removed:

• Azure AD Terraform module (terraform/modules/azure-ad-app/)
• Azure providers (azuread, azurerm)
• Entra ID token service files
• All Azure variables and outputs
• Provider selection logic
• Migration documents

Updated:

• Terraform configs to use only Keycloak module
• Application code to directly use Keycloak token service
• export-env.sh simplified to single provider
• All documentation updated to reference only Keycloak

Result: Zero references to Microsoft Entra ID remain. The application now authenticates exclusively through Keycloak with full Terraform automation.

Commits: ec05859, 65fc19a