Add support for PostgreSQL datastore by jasonish · Pull Request #361 · jasonish/evebox

jasonish · 2026-01-03T19:01:08Z

No description provided.

Copilot

Pull request overview

This PR adds comprehensive PostgreSQL datastore support to EveBox as an alternative to SQLite and Elasticsearch. The implementation includes partition-based storage, efficient retention management, and full API integration.

Key Changes:

Implements a complete PostgreSQL event repository with partitioning by day for efficient retention management
Adds query builder infrastructure for constructing parameterized PostgreSQL queries with JSON path support
Integrates PostgreSQL datastore across all API endpoints and event operations

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
`src/postgres/mod.rs`	Module organization and exports for the new PostgreSQL subsystem
`src/postgres/connection.rs`	PostgreSQL connection pool management and database initialization with migrations
`src/postgres/partition.rs`	Time-based partition management with caching for efficient event storage and retention
`src/postgres/query_builder.rs`	Query builder for constructing parameterized PostgreSQL queries with JSON field support
`src/postgres/eventrepo.rs`	Core event repository implementation with all query operations (alerts, events, stats, DNS, DHCP)
`src/postgres/importer.rs`	Event sink for batch importing events with metrics tracking and error handling
`src/postgres/retention.rs`	Background task for automatic partition dropping based on retention policies
`src/eventrepo/mod.rs`	Integration of PostgreSQL variant into the EventRepo enum with routing to implementation
`src/eventrepo/stats.rs`	Routes stats aggregation queries to PostgreSQL implementation
`src/importer.rs`	Adds PostgreSQL event sink to the EventSink enum
`src/server/mod.rs`	Adds postgres_url configuration field to ServerConfig
`src/server/main.rs`	Configures PostgreSQL datastore with URL from config/env variables
`src/server/metrics.rs`	Adds PostgreSQL-specific metrics tracking
`src/server/api/*.rs`	Updates API handlers to support PostgreSQL datastore variant
`src/logger.rs`	Adjusts logging configuration to reduce sqlx query noise
`src/eve/reader.rs`	Changes log level from info to trace for incomplete line reads
`resources/postgres/migrations/0001_Initial.sql`	SQL schema with partitioned events table and management functions
`docker/postgres/*`	Docker configuration for running PostgreSQL for testing
`Cargo.toml`	Adds postgres feature to sqlx dependency

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-04T16:42:21Z

src/postgres/eventrepo.rs

+                                                path.push_str(&format!("->>'{}' ", part));
+                                            } else {
+                                                path.push_str(&format!("->'{}'", part));
+                                            }
+                                        }
+                                        let p = builder.next_placeholder();
+                                        builder
+                                            .push_where(format!("{} {} {}", path.trim(), op, p))


There's a trailing space after the JSON path construction on this line. The code formats the path with a trailing space and then trims it on line 1060. Consider removing the trailing space from the format string on line 1055 to eliminate the need for trimming.

Suggested change

path.push_str(&format!("->>'{}' ", part));

} else {

path.push_str(&format!("->'{}'", part));

}

}

let p = builder.next_placeholder();

builder

.push_where(format!("{} {} {}", path.trim(), op, p))

path.push_str(&format!("->>'{}'", part));

} else {

path.push_str(&format!("->'{}'", part));

}

}

let p = builder.next_placeholder();

builder

.push_where(format!("{} {} {}", path, op, p))

Copilot · 2026-01-04T16:42:22Z

resources/postgres/migrations/0001_Initial.sql

+    partition_record RECORD;
+    dropped_count INTEGER := 0;
+    nanos_per_day BIGINT := 86400000000000;
+BEGIN


Consider adding validation for negative or zero values of retention_days. While the calling code appears to check for days > 0, this function should be defensive and handle edge cases gracefully. A negative retention period could result in incorrect cutoff calculations.

Suggested change

BEGIN

BEGIN

-- Validate retention_days to avoid incorrect cutoff calculations.

-- For NULL, zero, or negative values, do nothing and return 0.

IF retention_days IS NULL OR retention_days <= 0 THEN

RETURN 0;

END IF;

Copilot · 2026-01-04T16:42:22Z

src/postgres/partition.rs

+    pub fn partition_name(ts_nanos: i64) -> String {
+        let ts_seconds = ts_nanos / 1_000_000_000;
+        let dt = chrono::DateTime::from_timestamp(ts_seconds, 0)
+            .unwrap_or_else(|| chrono::DateTime::from_timestamp(0, 0).unwrap());


The unwrap_or_else fallback creates a timestamp at epoch 0 (1970-01-01), which may not be the intended behavior for invalid timestamps. Consider using a more appropriate error handling strategy, such as returning a Result or using a more meaningful default like the current time.

Suggested change

.unwrap_or_else(|| chrono::DateTime::from_timestamp(0, 0).unwrap());

.unwrap_or_else(|| chrono::Utc::now());

Copilot · 2026-01-04T16:42:22Z

src/server/main.rs

+    let userinfo = match (user, password) {
+        (Some(u), Some(p)) => format!("{}:{}@", u, p),
+        (Some(u), None) => format!("{}@", u),
+        _ => String::new(),
+    };
+
+    Some(format!(
+        "postgres://{}{}:{}/{}",
+        userinfo, host, port, database


The PostgreSQL URL construction does not URL-encode the username and password, which could lead to connection failures or security issues if these credentials contain special characters (e.g., @, :, /, ?). Consider using proper URL encoding for the userinfo component.

Copilot · 2026-01-04T16:42:23Z

src/postgres/eventrepo.rs

+
+        if n == 0 {
+            warn!("Archive by event ID request did not update any events");
+            bail!("postgres: event not found");


The error message "postgres: event not found" uses lowercase "postgres" while other parts of the codebase use "PostgreSQL". For consistency with the rest of the error messages and logging, consider using "PostgreSQL" or "Postgres" with capital P.

Suggested change

bail!("postgres: event not found");

bail!("PostgreSQL: event not found");

Copilot · 2026-01-04T16:42:24Z

src/eventrepo/mod.rs

            }
-            EventRepo::SQLite(ds) => ds.comment_event_by_id(event_id, comment, session).await,
+            EventRepo::SQLite(ds) => {
+                ds.comment_event_by_id(event_id, comment, session.clone())


The clone() operation in the parameter should be removed. This creates an unnecessary clone of the Arc before passing ownership.

Suggested change

ds.comment_event_by_id(event_id, comment, session.clone())

ds.comment_event_by_id(event_id, comment, session)

Copilot · 2026-01-04T16:42:24Z

src/eventrepo/mod.rs

        match self {
            EventRepo::Elastic(ds) => Ok(ds.agg(field, size, order, query).await?),
-            EventRepo::SQLite(ds) => Ok(ds.agg(field, size, order, query).await?),
+            EventRepo::SQLite(ds) => Ok(ds.agg(field, size, order, query.clone()).await?),


The clone() operation should be removed here. The SQLite variant now clones the query parameter while Elastic and Postgres don't. For consistency and to avoid unnecessary allocations, all variants should avoid cloning unless absolutely necessary. Consider if SQLite actually requires the clone or if it can be removed there as well.

Suggested change

EventRepo::SQLite(ds) => Ok(ds.agg(field, size, order, query.clone()).await?),

EventRepo::SQLite(ds) => Ok(ds.agg(field, size, order, query).await?),

Copilot · 2026-01-04T16:42:24Z

src/postgres/query_builder.rs

+impl EventQueryBuilder {
+    pub(crate) fn new() -> Self {
+        Self {
+            ..Default::default()


The default struct initialization syntax should use explicit field names for clarity. Instead of ..Default::default(), consider explicitly setting all fields. The current code creates an instance with all default values and then immediately overwrites them, which is unclear.

Suggested change

..Default::default()

select: Vec::new(),

from: Vec::new(),

wheres: Vec::new(),

order_by: None,

limit: 0,

args: PgArguments::default(),

arg_count: 0,

Copilot · 2026-01-04T16:42:25Z

src/postgres/eventrepo.rs

+                path.push_str(&format!("->>'{}' ", part));
+            } else {
+                path.push_str(&format!("->'{}' ", part));
+            }
+        }
+        path.trim().to_string()


There's an extra space character after the JSON path expression that gets trimmed later. Consider removing the trailing space in the format string to make the code cleaner and avoid the need for the trim() call.

Suggested change

path.push_str(&format!("->>'{}' ", part));

} else {

path.push_str(&format!("->'{}' ", part));

}

}

path.trim().to_string()

path.push_str(&format!("->>'{}'", part));

} else {

path.push_str(&format!("->'{}'", part));

}

}

path

Copilot · 2026-01-04T16:42:25Z

src/postgres/eventrepo.rs

+            if i == parts.len() - 1 {
+                path.push_str(&format!("->>'{}' ", part));
+            } else {
+                path.push_str(&format!("->'{}' ", part));


Similar to line 1055, there's an extra space after the JSON path expression. Consider removing the trailing space in the format string on line 1057 as well for consistency.

Suggested change

path.push_str(&format!("->'{}' ", part));

path.push_str(&format!("->'{}'", part));

- Add postgres module with connection pooling and lifecycle management - Implement EventRepository for PostgreSQL with full query/search capabilities - Add SQL migrations for initial schema with events, stats, and bookmarks tables - Implement Importer for inserting events into PostgreSQL - Add QueryBuilder for dynamic SQL construction from search filters - Implement table partitioning by date for performance - Add retention policy support for automatic old data cleanup - Update EventRepository trait to abstract datastore differences - Add Docker Compose setup for PostgreSQL development - Update server API layer to work with PostgreSQL backend - Add postgres feature flag to Cargo.toml Co-authored-by: Henri 🐕 <henri@codemonkey.net>

jasonish force-pushed the postgres/v1 branch from df1c280 to dad54e5 Compare January 4, 2026 07:02

jasonish requested a review from Copilot January 4, 2026 16:37

Copilot started reviewing on behalf of jasonish January 4, 2026 16:37 View session

jasonish force-pushed the postgres/v1 branch from 2893db1 to 124cfb3 Compare January 4, 2026 16:38

Copilot AI reviewed Jan 4, 2026

View reviewed changes

jasonish force-pushed the postgres/v1 branch from 124cfb3 to 8e30d17 Compare January 6, 2026 01:48

jasonish closed this Jan 8, 2026

jasonish deleted the postgres/v1 branch January 8, 2026 23:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for PostgreSQL datastore#361

Add support for PostgreSQL datastore#361
jasonish wants to merge 1 commit intomainfrom
postgres/v1

jasonish commented Jan 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Copilot AI Jan 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

-BEGIN
+BEGIN
+    -- Validate retention_days to avoid incorrect cutoff calculations.
+    -- For NULL, zero, or negative values, do nothing and return 0.
+    IF retention_days IS NULL OR retention_days <= 0 THEN
+        RETURN 0;
+    END IF;

	.unwrap_or_else(\|\| chrono::DateTime::from_timestamp(0, 0).unwrap());
	.unwrap_or_else(\|\| chrono::Utc::now());

	bail!("postgres: event not found");
	bail!("PostgreSQL: event not found");

	ds.comment_event_by_id(event_id, comment, session.clone())
	ds.comment_event_by_id(event_id, comment, session)

	EventRepo::SQLite(ds) => Ok(ds.agg(field, size, order, query.clone()).await?),
	EventRepo::SQLite(ds) => Ok(ds.agg(field, size, order, query).await?),

-            ..Default::default()
+            select: Vec::new(),
+            from: Vec::new(),
+            wheres: Vec::new(),
+            order_by: None,
+            limit: 0,
+            args: PgArguments::default(),
+            arg_count: 0,

	path.push_str(&format!("->'{}' ", part));
	path.push_str(&format!("->'{}'", part));

Uh oh!

Conversation

jasonish commented Jan 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant