Docs: Add PDP Explorer documentation

README.md

@@ -1,16 +1,23 @@
-# pdp-explorer
+# PDP Explorer
+
 Data model and UI for exploring the PDP hot storage network
 
-# Mockup 
+## Documentation
+
+Detailed documentation is available in the following file:
+
+- [Documentation](docs/README.md) - System architecture, database schema, and development guide
+
+## Mockup
 
 This is a first draft at what the PDP explorer will look like ![pdpexplorer](https://github.com/user-attachments/assets/e0595422-fa77-490b-ab57-0c9516ea5d8a)
 
 # Usage
 
-A few user journeys: 
-As a user storing data with PDP I can use the explorer to: 
-* Check if my SP has had any faults.  And I can check which data in particular was faulted
-* Validate that all of the data added to my proofset is data that I asked to store, not anything else
-* Look at fault rate of SPs in the network when deciding who to store my data with
-* Learn about data that has been removed from my proofset
-  
+A few user journeys:
+As a user storing data with PDP I can use the explorer to:
+
+- Check if my SP has had any faults. And I can check which data in particular was faulted
+- Validate that all of the data added to my proofset is data that I asked to store, not anything else
+- Look at fault rate of SPs in the network when deciding who to store my data with
+- Learn about data that has been removed from my proofset

docs/ARCHITECTURE.md

@@ -0,0 +1,160 @@
+# PDP Explorer Backend Architecture
+
+## Overview
+
+The PDP Explorer backend is composed of two main components:
+
+1. **Indexer Service**: Responsible for processing blockchain events and maintaining the database state
+2. **API Server**: Provides REST endpoints for the frontend to query indexed data
+
+## System Components
+
+### 1. Indexer Service
+
+The indexer is responsible for:
+
+- Processing blockchain events in real-time
+- Maintaining database consistency during chain reorganizations
+- Managing historical data for providers and proof sets
+
+#### Key Components:
+
+- **Block Processor**: Handles block-by-block processing of chain events
+- **Event Handlers**: Process specific event types (ProofSetCreated, RootsAdded, etc.)
+
+### 2. API Server
+
+Provides RESTful endpoints for the frontend application.
+
+#### Available Endpoints:
+
+- `GET /providers`: List all providers with pagination
+- `GET /providers/:providerId`: Get detailed provider information
+- `GET /providers/:providerId/proofsets`: List all proof sets for a provider with sorting and pagination
+- `GET /providers/:providerId/activities`: Get activity statistics for a provider
+- `GET /proofsets`: List all proof sets with sorting and pagination
+- `GET /proofsets/:proofSetId`: Get detailed proof set information
+- `GET /proofsets/:proofSetId/event-logs`: Get proof set event logs
+- `GET /proofsets/:proofSetId/txs`: Get proof set transactions
+- `GET /proofsets/:proofSetId/roots`: Get proof set roots
+- `GET /network-metrics`: Get network-wide metrics
+- `GET /search`: Search for proof sets and providers
+
+## Database Schema
+
+The system uses PostgreSQL with the following key tables:
+
+### Core Tables
+
+1. **blocks**
+
+   - Tracks processed blocks and their finalization status
+   - Used for handling chain reorganizations
+
+2. **providers**
+
+   - Stores provider information with version control
+   - Tracks historical changes to provider data
+
+3. **proof_sets**
+
+   - Maintains proof set metadata and status
+   - Includes version control for ownership changes
+
+4. **roots**
+
+   - Stores root data associated with proof sets
+   - Maintains historical versions for chain reorgs
+
+5. **transactions**
+
+   - Records all relevant blockchain transactions
+   - Links to related entities (proof sets, providers)
+
+6. **event_logs**
+
+   - Stores blockchain events and their metadata
+   - Used for real-time processing and indexing
+
+7. **proofs**
+
+   - Stores individual proof submissions
+   - Used for analytics and provider activity tracking
+
+8. **proof_fees**
+
+   - Records fee information for proof submissions
+   - Used for economic analysis
+
+9. **fault_records**
+   - Tracks provider fault events
+   - Used for provider reliability metrics
+
+## Architecture Diagram
+
+![System Architecture Diagram](./assets/pdp-arch.png)
+
+## Integration Points
+
+### Blockchain Integration
+
+- Connects to Filecoin Eth RPC endpoint for blockchain data
+- Handles chain reorganizations gracefully
+- Recovery process for historical data
+
+### Frontend Integration
+
+- RESTful API with JSON responses
+- Pagination support for large datasets
+- Sorting and filtering capabilities
+- Real-time data consistency
+
+## Data Flow
+
+1. **Event Processing Flow**
+
+   ```
+   Blockchain Event → Filecoin Eth RPC Endpoints → Indexer → Processor → Handlers → Database
+   ```
+
+2. **Query Flow**
+
+   ```
+   Frontend Request → API Server → Database → JSON Response
+   ```
+
+3. **Chain Reorganization Flow**
+   ```
+   Reorg Detection → Block Reversal → Historical Data Cleanup → State Recovery
+   ```
+
+## Configuration
+
+The system is configured through environment variables:
+
+- Database connection settings
+- Filecoin Eth RPC endpoint configuration
+- Other runtime settings
+
+## Security Considerations
+
+- Database connections use SSL/TLS
+- API endpoints follow REST best practices
+- Environment variables for sensitive configuration
+- No direct blockchain write access from API server
+
+## Maintenance and Operations
+
+### Database Maintenance
+
+- Regular cleanup of historical data
+- Index optimization for query performance
+- Monitoring of database size and growth
+
+## Future Considerations
+
+- **Feature Additions**
+  - Event processing pipeline optimization
+  - Use Filecoin Method Group instead of Eth
+  - Advanced analytics endpoints
+  - Historical data API

docs/DEVELOPMENT.md

@@ -0,0 +1,128 @@
+# Development Guide
+
+## Overview
+
+This document provides guidelines and instructions for developers working on the PDP Explorer backend. It covers setup procedures, development workflows, and best practices to ensure consistent and high-quality contributions.
+
+## Development Environment Setup
+
+### Prerequisites
+
+- Go 1.19 or higher
+- PostgreSQL 14 or higher
+- Docker and Docker Compose (optional, for containerized development)
+- Make
+
+### Initial Setup
+
+1. **Clone the repository**:
+
+   ```bash
+   git clone https://github.com/FilOzone/pdp-explorer.git
+   cd pdp-explorer
+   ```
+
+2. **Set up environment variables**:
+   Create a `.env` file with following variables at:
+
+   - `backend/indexer/.env`
+
+     ```
+     # Database
+     DATABASE_URL=postgresql://localhost:5432/pdp
+
+     # RPC provider
+     LOTUS_API_ENDPOINT=https://api.calibration.node.glif.io/rpc/v0
+     LOTUS_API_KEY= # Your API key (from https://api.node.glif.io)
+
+     # Trigger config
+     TRIGGERS_CONFIG=./config/pdp.yaml
+     START_BLOCK= # Start block number
+     ```
+
+   - `backend/server/.env`
+
+     ```
+     # Database
+     DATABASE_URL=postgresql://localhost:5432/pdp
+
+     # Server Port
+     PORT=3000
+     ```
+
+   - `client/.env`
+     ```
+     VITE_SERVER_URL=http://localhost:3000
+     VITE_NETWORK=calibration
+     ```
+
+3. **Initialize the database**:
+
+   ```bash
+   cd backend/indexer
+   make migrate-up
+   ```
+
+## Development Workflow
+
+### Running the Indexer
+
+```bash
+# From root of the repository
+cd backend/indexer
+
+# Run the indexer in development mode
+make dev
+```
+
+### Running the API Server
+
+```bash
+# From root of the repository
+cd backend/server
+
+# Run the API server in development mode
+make dev
+```
+
+### Running the Frontend
+
+```bash
+# From root of the repository
+cd client
+
+# Run the frontend in development mode
+npm run dev
+```
+
+## Code Structure
+
+### Indexer Directory Layout
+
+```
+backend/indexer/
+├── cmd/
+│   └── indexer/          # Indexer application entrypoint
+├── config/               # Configuration files
+├── internal/
+│   ├── client/           # RPC client library
+│   ├── contract/         # Contract related code
+│   ├── indexer/          # Blockchain indexing logic
+│   ├── infrastructure/   # Infrastructure layer
+│   │   ├── config/       # Infrastructure configuration
+│   │   └── database/     # Database access layer
+│   ├── logger/           # Logging package
+│   ├── models/           # Data models
+│   ├── processor/        # Transactions and events processor
+│   │   └── handlers/     # Event and transaction handlers
+│   └── types/            # Common types
+├── migrations/           # Database migrations
+└── scripts/              # Utility sql scripts
+```
+
+## Additional Resources
+
+- [Go Documentation](https://golang.org/doc/)
+- [Echo Framework Documentation](https://echo.labstack.com/)
+- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
+- [Filecoin Documentation](https://docs.filecoin.io/)