[PoC] Direct op-sqlite connections

[rkistner]

This exposes an additional OPSqliteOpenFactory.openDirectConnection() method, to get direct access to an op-sqlite DB.

This is similar to opening another op-sqlite DB directly, but:

1. Uses the same sqlite options directly from the OPSqliteOpenFactory.
2. Auto-loads the powersync extension.
3. Propagates any updates in that connection to the PowerSyncDatabase, to trigger watch queries and crud uploads.

We do this rather than exposing the existing DB, since that would bypass any existing transaction locking mechanisms, leading to likely race conditions when the DB is used directly.

One specific advantage of this is allowing synchronous queries to be executed, to make it easier to migrate existing applications currently built on synchronous queries. I would not recommend this for other use cases.

Even with this approach, synchronous writes should be wrapped with PowerSyncDatabase.writeLock(), otherwise it could fail with SQLITE_BUSY if another transaction is busy. We cannot really use BUSY_TIMEOUT for this, since the synchronous query would block the JS thread, preventing the other lock from being released.

Synchronous reads do not have the same restriction, and can run concurrently with any other transaction on the PowerSyncDatabase.

Other changes

1. This makes the number of read connections configurable.
2. This changes write transactions to use BEGIN EXCLUSIVE instead of BEGIN: Same as in the web SDK. This reduces the number of lock steps, and avoids triggering SQLITE_BUSY errors later in the transaction.

TODOs

• This currently builds on many private APIs of OPSQLiteDBAdapter - this can probably be done better.
• Needs testing - I haven't actually tested this at all yet.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 711a96a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[a11rew]

+1 on this being added. Appreciate it!

My use-case for this would be replaying write notifications to the database that occurred in another connection so the OPSqliteConnection the UI and watched queries listen on gets updated. Access to the connection would enable calling addTableUpdate on the UI connection directly

[rkistner]

@a11rew Could you expand on your specific use case? Specifically, where would you like to do (1) writes, (2) reads and (3) watched queries, between the PowerSyncDatabase, op-sqlite connection, or another connection?

We probably won't go ahead with this PR in the current form, since it doesn't solve any actual use case I'm aware of: synchronous writes on the op-sqlite connection have issues regarding write locks (see description above), and synchronous reads can be done directly using op-sqlite, without the changes here (see https://github.com/powersync-community/powersync-react-native-synchronous-operations).

[a11rew]

@rkistner use case here is reflecting changes made to the underlying database outside the Powersync connection in the UI. i.e. if a change is made to the sqlite db in a connection outside of the one Powersync establishes, watched queries should still update.

In my specific case:

1. Writes are done in native iOS code to the same SQLite DB Powersync creates, this allows the app to respond to App Intents without the JS runtime being available
2. Reads are done in the JS runtime when the app is foregrounded. They're done through Powersync.
3. The app uses watched queries to display data in some views. Since sqlite3_update_hook and subsequently op-sqlite's update hook functionality work on a per-connection basis, the connection Powersync is listening on doesn't receive update notifications.

What we're currently doing to work around this is registering a sqlite3_update_hook on that other sqlite connection and then calling tablesUpdated on all the listeners registered on the op-sqlite connection the app is using for UI. This also means we've had do things like re-implement the buffering and flushing OPSQLiteConnection does. Direct access to the/a connection instance would make the ergonomics of working with the connection better.

[rkistner]

Thanks, that makes sense. So you're making database updates in native iOS code, and need those to trigger update notifications / watched queries in the PowerSyncDatabase.

I think this PR as-is isn't quite suitable for that, since opening another op-sqlite connection wouldn't help you. But adding a way to trigger those update notifications directly on the PowerSyncDatabase instance would help.