Add new databases new users new roles with selective permission #56

lecping · 2025-12-16T14:57:41Z

This pull request consists 6 SQL which can be run in psql bash.

SQL to initialize database for dataset metadata, file metadata of dataset, tool registry data and user usage and history data.
SQL to generate username and system account for scripts.
Configure roles permission for datasetDB.
Configure roles permission for fileDB.
Configure roles permission for toolDB.
Configure roles permission for userDB.

lecping · 2025-12-16T14:59:18Z

@Michal-Kolomanski and @tobiasschweizer , please have a first look on the SQL, it is very likely that we will move most of the dataset metadata to the DB called datasetDB instead of admin.

lecping · 2025-12-16T15:02:38Z

ownership definition:
createDB.sql -> Metadata Warehouse Owner
user.sql -> Metadata Warehouse Owner
datasetDBconfig -> WP4
fileDBconfig -> WP5-6
toolDBconfig -> WP5-6
userDBconfig -> WP4

tobiasschweizer

How should these single SQL statements be executed? I think it would make sense to have a script that executes them in a certain order. What happens if the DBB already exists?

Note that there is already a script that creates the admin DB: https://github.com/EOSC-Data-Commons/metadata-warehouse/blob/main/scripts/postgres_data/create_db.py

tobiasschweizer · 2025-12-17T15:50:27Z

scripts/postgres_data/create_sql/createDB.sql

@@ -0,0 +1,8 @@
+-- database to store metadata about datasets
+CREATE DATABASE datasetDB;


How does this relate to existing DB structure? Is datasetDB supposed to replace the existing DB structure?

If yes, then this structure would have to comply with what we currently have since the existing code depends on it (SQL queries, FastAPI)

if admin is going to be replaced by datasetDB, then this would need to be adapted when creating the psycopg client as now it is assumed that the db user and the db name are the same.

yes it will be the case, I can update accordingly. with a database and user of the same name "admin"is not easy for monitoring purpose.

tobiasschweizer · 2025-12-17T15:52:24Z

scripts/postgres_data/create_sql/user.sql

+---------------------------------------------------------
+-- CREATE GOLOBAL USERS
+
+CREATE ROLE reggie LOGIN PASSWORD 'reggie';


Is there another way to manage user? This seems hard to maintain as new users will join the project.

we can write a procedure to skip those roles which have been already created before. So far we still do not have procedure.

Michal-Kolomanski

To avoid code duplication, suggested template:

-- =========================================
-- Database RBAC template for {{DB_NAME}}
-- =========================================

\c {{DB_NAME}}

-- -----------------------------------------
-- 1. CREATE ROLE GROUPS (NOLOGIN)
-- -----------------------------------------
CREATE ROLE {{DB_NAME}}_read NOLOGIN;
CREATE ROLE {{DB_NAME}}_readwrite NOLOGIN;
CREATE ROLE {{DB_NAME}}_admin NOLOGIN;

-- -----------------------------------------
-- 2. DATABASE-LEVEL SECURITY
-- -----------------------------------------

-- Nobody connects by default
REVOKE CONNECT ON DATABASE {{DB_NAME}} FROM PUBLIC;

-- Only role groups may connect
GRANT CONNECT ON DATABASE {{DB_NAME}}
  TO {{DB_NAME}}_read,
     {{DB_NAME}}_readwrite,
     {{DB_NAME}}_admin;

-- -----------------------------------------
-- 3. SCHEMA LOCKDOWN
-- -----------------------------------------

-- Remove dangerous defaults
REVOKE ALL ON SCHEMA public FROM PUBLIC;
REVOKE CREATE ON SCHEMA public FROM PUBLIC;

-- -----------------------------------------
-- 4. SCHEMA ACCESS BY ROLE
-- -----------------------------------------

-- Read-only: see objects, nothing else
GRANT USAGE ON SCHEMA public TO {{DB_NAME}}_read;

-- Read-write: create & modify objects
GRANT USAGE, CREATE ON SCHEMA public TO {{DB_NAME}}_readwrite;

-- Admin: full control
GRANT ALL ON SCHEMA public TO {{DB_NAME}}_admin;

-- -----------------------------------------
-- 5. EXISTING OBJECT PRIVILEGES
-- -----------------------------------------

-- Tables
GRANT SELECT ON ALL TABLES IN SCHEMA public TO {{DB_NAME}}_read;
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO {{DB_NAME}}_readwrite;
GRANT ALL ON ALL TABLES IN SCHEMA public TO {{DB_NAME}}_admin;

-- Sequences 
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO {{DB_NAME}}_readwrite;
GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO {{DB_NAME}}_admin;

-- Functions
GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA public
  TO {{DB_NAME}}_read, {{DB_NAME}}_readwrite;
GRANT ALL ON ALL FUNCTIONS IN SCHEMA public TO {{DB_NAME}}_admin;

-- -----------------------------------------
-- 6. DEFAULT PRIVILEGES (FUTURE OBJECTS)
-- -----------------------------------------

-- Tables
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT ON TABLES TO {{DB_NAME}}_read;

ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE ON TABLES TO {{DB_NAME}}_readwrite;

ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT ALL ON TABLES TO {{DB_NAME}}_admin;

-- Sequences
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT USAGE, SELECT ON SEQUENCES TO {{DB_NAME}}_readwrite;

ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT ALL ON SEQUENCES TO {{DB_NAME}}_admin;

-- Functions
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT EXECUTE ON FUNCTIONS
  TO {{DB_NAME}}_read, {{DB_NAME}}_readwrite;

ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT ALL ON FUNCTIONS TO {{DB_NAME}}_admin;

Then we can have a dedicated sql file for granting access - the only place users get power - easy to maintain and monitor, debug etc.

e.g.

-- datasetDB
GRANT datasetDB_read      TO reggie;
GRANT datasetDB_readwrite TO ping;

-- fileDB
GRANT fileDB_read TO reggie;

Michal-Kolomanski · 2025-12-17T15:19:21Z

scripts/postgres_data/create_sql/createDB.sql

@@ -0,0 +1,8 @@
+-- database to store metadata about datasets
+CREATE DATABASE datasetDB;


we probably should define here the owner, encoding and change the default template to template0 to achieve separation (no inheritance)

CREATE DATABASE datasetDB WITH OWNER = postgres ENCODING = 'UTF8' TEMPLATE = template0; CREATE DATABASE fileDB WITH OWNER = postgres ENCODING = 'UTF8' TEMPLATE = template0; CREATE DATABASE toolDB WITH OWNER = postgres ENCODING = 'UTF8' TEMPLATE = template0; CREATE DATABASE userDB WITH OWNER = postgres ENCODING = 'UTF8' TEMPLATE = template0;

https://stackoverflow.com/questions/51121889/why-do-we-need-template0-and-template1-in-postgresql

@Michal-Kolomanski, that is a good idea but I suggest to let each owner to maintain the configuration of their database. I think that it will take more time to debug and delay if we force each DB to have the same template.

The main goal is to let DB owner to start fast and populate the required data for now. If we expected to have more than 100 databases and 100 user logins... your idea will be exactly needed for the refactoring.

But the access control shouldn't be exactly the same for each DB?
We need the same roles in each DB, and only certain people should be able to read/write.
In my scenario, we only maintain a single file that pairs people with roles:

-- datasetDB GRANT datasetDB_read TO reggie; GRANT datasetDB_readwrite TO ping;

We basically supply only access control and roles, everything else is custom made by the owners

Michal-Kolomanski · 2025-12-17T15:25:29Z

scripts/postgres_data/create_sql/user.sql

+---------------------------------------------------------
+-- CREATE GOLOBAL USERS
+
+CREATE ROLE reggie LOGIN PASSWORD 'reggie';


By default, we should set no privileges like:

CREATE ROLE reggie LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE; CREATE ROLE jusong LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE; CREATE ROLE ritwik LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE; CREATE ROLE tobias LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE; CREATE ROLE josip LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE; CREATE ROLE michal LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE; CREATE ROLE ping LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE; CREATE ROLE vincent LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE; CREATE ROLE eko LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE; CREATE ROLE script LOGIN PASSWORD 'CHANGE_ME' NOSUPERUSER NOCREATEDB NOCREATEROLE;

that is good, should I change them now?

Michal-Kolomanski · 2025-12-17T15:40:19Z

scripts/postgres_data/create_sql/datasetDBconfig.sql

@@ -0,0 +1,41 @@
+-- ---------------------------
+-- DATABASE 1 – Dataset / FAIR, connect to datasetDB first


This applies to every DB

I believe that firstly, we need to lock down the PUBLIC schema. Without this any user that can connect to the DB, should be able to do anything

REVOKE ALL ON SCHEMA public FROM PUBLIC; REVOKE CREATE ON SCHEMA public FROM PUBLIC;

Result, anyone can still:

Open a connection to datasetDB

Start a session

Consume a connection slot

See database metadata

-> They just can’t do much after connecting

to avoid this add also:
REVOKE CONNECT ON DATABASE datasetDB FROM PUBLIC;

Michal-Kolomanski · 2025-12-17T15:41:11Z

scripts/postgres_data/create_sql/datasetDBconfig.sql

+CREATE ROLE datasetDB_read;
+CREATE ROLE datasetDB_readwrite;
+CREATE ROLE datasetDB_admin;


Rule:

Humans & services → LOGIN
Permissions → NOLOGIN

CREATE ROLE datasetDB_read NOLOGIN; CREATE ROLE datasetDB_readwrite NOLOGIN; CREATE ROLE datasetDB_admin NOLOGIN;

Michal-Kolomanski · 2025-12-17T15:44:44Z

scripts/postgres_data/create_sql/datasetDBconfig.sql

+GRANT SELECT ON ALL TABLES IN SCHEMA public TO datasetDB_read;
+
+-- READWRITE
+GRANT USAGE, CREATE ON SCHEMA public TO datasetDB_readwrite;


Missing sequences & functions privileges

-- Default privileges for sequences ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO datasetDB_readwrite; ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO datasetDB_admin; -- Default privileges for functions ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT EXECUTE ON FUNCTIONS TO datasetDB_read, datasetDB_readwrite; ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO datasetDB_admin;

scripts/postgres_data/create_sql/userDBconfig.sql

lecping added 6 commits December 16, 2025 15:33

feat(datamodel): add username to metadata warehouse

edd10e3

feat(datamodel): add four databases for metadata warehouse

e5f04dc

feat(datamodel): configure dataset DB for user roles and permission

a2aeb82

feat(datamodel): configure file DB for user roles and permission

e274909

feat(datamodel): configure tool DB for user roles and permission

6e29501

feat(datamodel): configure user DB for user roles and permission

69ca6f7

lecping requested review from Michal-Kolomanski and tobiasschweizer and removed request for Michal-Kolomanski December 16, 2025 14:58

tobiasschweizer reviewed Dec 17, 2025

View reviewed changes

Michal-Kolomanski reviewed Dec 17, 2025

View reviewed changes

vemonet reviewed Jan 13, 2026

View reviewed changes

scripts/postgres_data/create_sql/userDBconfig.sql Outdated Show resolved Hide resolved

refactor: rename userDB to appDB

01d85de

		@@ -0,0 +1,8 @@
		-- database to store metadata about datasets
		CREATE DATABASE datasetDB;

		@@ -0,0 +1,41 @@
		-- ---------------------------
		-- DATABASE 1 – Dataset / FAIR, connect to datasetDB first

Add new databases new users new roles with selective permission #56

Are you sure you want to change the base?

Add new databases new users new roles with selective permission #56

Uh oh!

Conversation

lecping commented Dec 16, 2025

Uh oh!

lecping commented Dec 16, 2025

Uh oh!

lecping commented Dec 16, 2025

Uh oh!

tobiasschweizer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michal-Kolomanski left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tobiasschweizer left a comment •

edited

Loading