experiment: Add windmill-local crate with libSQL/Turso for local mode preview#7662
experiment: Add windmill-local crate with libSQL/Turso for local mode preview#7662rubenfiszel wants to merge 2 commits intomainfrom
Conversation
This experimental crate demonstrates running Windmill preview endpoints
with libSQL (SQLite/Turso) instead of PostgreSQL. Key features:
- Schema: SQLite-compatible schema for jobs, queue, and results
- ENUMs → TEXT with CHECK constraints
- JSONB → JSON (TEXT)
- Arrays → JSON arrays
- No FOR UPDATE SKIP LOCKED (single worker, mutex coordination)
- Database: Supports three modes via libsql crate:
- In-memory SQLite (for testing)
- File-based SQLite (local persistence)
- Remote Turso (multi-writer scenarios)
- API: Compatible preview endpoints:
- POST /api/w/{workspace}/jobs/run/preview
- POST /api/w/{workspace}/jobs/run_wait_result/preview
- POST /api/w/{workspace}/jobs/run/preview_flow
- POST /api/w/{workspace}/jobs/run_wait_result/preview_flow
- Executor: Simple script execution for bash, python3, deno, bun
- Worker: Single embedded worker that processes queue
Run with: cargo run -p windmill-local --example local_server
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add flow_executor.rs that supports executing complex flows in local mode: - ForloopFlow: iterate over arrays/ranges with sequential execution - WhileloopFlow: execute modules while condition is true - BranchOne: if/else branching based on conditions - BranchAll: parallel branch execution (sequential for now) - RawScript: inline script execution (bash, python, deno, bun) - Identity: pass-through module Key features: - Uses windmill-common FlowValue types for compatibility - Expression evaluation for input transforms with comparisons - Proper flow status tracking with module results - Recursive async execution with async_recursion crate Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
Claude finished @rubenfiszel's task in 2m 9s —— View job PR Review Complete: windmill-local crate with libSQL/Turso
SummaryThis is a well-structured experimental implementation of a lightweight local mode for Windmill using libSQL instead of PostgreSQL. The architecture is clean with good separation of concerns across modules. Strengths
Issues Found (9 inline comments)
Recommendations
Testing InstructionsTo verify the changes through the app interface: This is a backend-only experimental crate that runs as a standalone local server, not integrated into the main Windmill app UI. To test:
|
Deploying windmill with
|
| Latest commit: |
741526b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://45bd47c0.windmill.pages.dev |
| Branch Preview URL: | https://experiment-libsql-local-mode.windmill.pages.dev |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| import sys | ||
|
|
||
| # Args passed as JSON | ||
| args = json.loads('''{}''') |
There was a problem hiding this comment.
Security Concern: JSON Injection in Python Wrapper
The args are injected directly into the Python code using triple-quoted strings, which could allow code injection if the JSON contains special Python sequences.
Consider using base64 encoding or passing via environment variable:
| args = json.loads('''{}''') | |
| // Wrap the code to handle args and return JSON result | |
| let args_b64 = base64::engine::general_purpose::STANDARD.encode(serde_json::to_string(args)?); | |
| let wrapped_code = format!( | |
| r#" | |
| import json | |
| import sys | |
| import base64 | |
| # Args passed as base64-encoded JSON | |
| args = json.loads(base64.b64decode('{}').decode('utf-8')) | |
| # User code | |
| {} | |
| # Call main if it exists | |
| if 'main' in dir(): | |
| result = main(**args) | |
| print(json.dumps(result)) | |
| "#, | |
| args_b64, | |
| code | |
| ); |
Alternatively, pass via stdin or environment variable for better security.
| } | ||
|
|
||
| /// Get a reference to the connection (locked) | ||
| pub async fn conn(&self) -> tokio::sync::MutexGuard<'_, Connection> { |
There was a problem hiding this comment.
Potential Deadlock Risk
Exposing the raw mutex guard can lead to deadlocks if callers hold the guard across await points or while calling other LocalDb methods.
Consider removing this public method and ensuring all database operations go through the internal methods (execute, query, execute_batch) which properly manage the lock scope. If raw connection access is truly needed, document the risks clearly:
/// Get a reference to the connection (locked)
///
/// **Warning**: Hold this lock for the minimum time necessary.
/// Do not call other LocalDb methods while holding this guard or
/// await across the guard - this will cause deadlocks.
pub async fn conn(&self) -> tokio::sync::MutexGuard<'_, Connection> {
| }; | ||
|
|
||
| let id_str: String = row.get(0)?; | ||
| let id = Uuid::parse_str(&id_str).map_err(|e| LocalError::InvalidJobState(e.to_string()))?; |
There was a problem hiding this comment.
Race Condition: Non-Atomic Job Pull
The SELECT and UPDATE operations are not atomic. Between the SELECT (line 23-37) and UPDATE (line 48-52), another concurrent request could theoretically pull the same job.
While this is mitigated by the single-connection mutex design mentioned in comments, if the crate is ever used with multiple connections (e.g., Turso remote mode with multiple clients), this could cause duplicate job processing.
Consider using a single atomic UPDATE...RETURNING statement:
UPDATE v2_job_queue SET running = 1, started_at = ?1
WHERE id = (
SELECT q.id FROM v2_job_queue q
JOIN v2_job j ON q.id = j.id
WHERE q.running = 0 AND q.scheduled_for <= ?1
ORDER BY q.priority DESC, q.scheduled_for ASC
LIMIT 1
)
RETURNING idThen fetch the full job data separately.
| skip_failures: bool, | ||
| ctx: &mut FlowContext, | ||
| ) -> Result<serde_json::Value> { | ||
| const MAX_ITERATIONS: usize = 1000; |
There was a problem hiding this comment.
Magic Number: MAX_ITERATIONS
The value 1000 is hardcoded without configuration. For an experimental crate this is fine, but consider making it configurable:
const DEFAULT_MAX_ITERATIONS: usize = 1000;And potentially adding it as a parameter to the flow context or a configuration struct for future flexibility.
| // Handle string literals | ||
| if (expr.starts_with('"') && expr.ends_with('"')) || | ||
| (expr.starts_with('\'') && expr.ends_with('\'')) { | ||
| return Ok(serde_json::json!(&expr[1..expr.len()-1])); |
There was a problem hiding this comment.
String Slice Safety Issue
Slicing &expr[1..expr.len()-1] could panic on multi-byte UTF-8 characters if the string boundaries don't align with character boundaries. While unlikely for typical inputs, it's safer to use:
| return Ok(serde_json::json!(&expr[1..expr.len()-1])); | |
| if (expr.starts_with('"') && expr.ends_with('"')) || | |
| (expr.starts_with('\'') && expr.ends_with('\'')) { | |
| let inner = &expr[1..expr.len()-1]; | |
| return Ok(serde_json::json!(inner)); |
Or even better, use chars():
if (expr.starts_with('"') && expr.ends_with('"')) ||
(expr.starts_with('\'') && expr.ends_with('\'')) {
let inner: String = expr.chars().skip(1).take(expr.chars().count() - 2).collect();
return Ok(serde_json::json!(inner));
}Though for ASCII quotes this is fine as-is.
|
|
||
| /// Create the API router | ||
| fn create_router(state: Arc<AppState>) -> Router { | ||
| let cors = CorsLayer::new() |
There was a problem hiding this comment.
Security Note: Wide-Open CORS
Allowing any origin/method/header is appropriate for local development but should be documented:
// Note: Wide-open CORS is intentional for local development mode.
// This should NOT be used in production deployments.
let cors = CorsLayer::new()Consider adding a feature flag or configuration option to restrict CORS for more secure deployments.
| }; | ||
| use windmill_common::scripts::ScriptLang as WmScriptLang; | ||
|
|
||
| use crate::db::LocalDb; |
There was a problem hiding this comment.
Unused Import
LocalDb is imported but the db parameter in many functions appears unused for the actual flow execution logic (it's passed through but not used for persistence during flow execution). This is noted as "by design" but the import and passing db everywhere adds noise.
If db is intended for future use (e.g., persisting intermediate results), that's fine - otherwise consider removing it from function signatures to simplify the API.
|
|
||
| let mut child = Command::new("deno") | ||
| .arg("eval") | ||
| .arg("--unstable") |
There was a problem hiding this comment.
Deprecation Warning: --unstable Flag
The --unstable flag for Deno is deprecated in recent versions. Consider using specific unstable feature flags instead, or removing it entirely if no unstable features are needed:
| .arg("--unstable") | |
| let mut child = Command::new("deno") | |
| .arg("eval") | |
| .arg(&wrapped_code) |
Or if specific unstable features are needed:
.arg("--unstable-kv") // Example: specific unstable feature
| windmill-common = { path = "../windmill-common", default-features = false } | ||
|
|
||
| # For input transforms (JavaScript evaluation) | ||
| rquickjs = { version = "0.8", features = ["bindgen", "classes", "loader", "array-buffer", "futures"] } |
There was a problem hiding this comment.
Unused Dependency
rquickjs is listed as a dependency but doesn't appear to be used anywhere in the codebase. The expression evaluation in flow_executor.rs uses custom string parsing instead.
If this was intended for a more robust JS evaluation implementation, consider either:
- Using it for the
evaluate_exprfunction to handle complex expressions properly - Removing it to reduce compilation time and binary size
| rquickjs = { version = "0.8", features = ["bindgen", "classes", "loader", "array-buffer", "futures"] } | |
| # For input transforms (JavaScript evaluation) - TODO: implement proper JS runtime | |
| # rquickjs = { version = "0.8", features = ["bindgen", "classes", "loader", "array-buffer", "futures"] } |
1 participant
Summary
Experimental implementation of a local mode for Windmill using libSQL (Turso's SQLite fork) instead of PostgreSQL. This enables lightweight preview execution without requiring a full PostgreSQL setup.
What's included:
windmill-localcrate with embedded libSQL databaseKnown limitations (by design for this experiment):
To test:
Test plan
cargo test -p windmill-local)🤖 Generated with Claude Code