Skip to content

Export files to s3 and crons to scripts#368

Draft
antoineludeau wants to merge 8 commits intomasterfrom
antoineludeau/export-files-to-S3
Draft

Export files to s3 and crons to scripts#368
antoineludeau wants to merge 8 commits intomasterfrom
antoineludeau/export-files-to-S3

Conversation

@antoineludeau
Copy link
Contributor

@antoineludeau antoineludeau commented Feb 13, 2024

WARNING - pre-requisite : before merging this PR, need to have deployed the build-addok-bundle cloud service.

Context

In order to make ban-plateforme cloud ready, we need to modify some mechanisms that won't be possible on the cloud. This PR is targeting the file storage system. For now, the storage used in the server storage system. We need to migrate to an S3 storage system. As a consequence, we need to migrate the way export files are created and stored. The export files targeted are :
1- Department files (csv, csv-bal, csv-with-ids, addok)
2- National files :

  • csv : adresses-france.csv.gz, lieux-dits-beta-france.csv.gz
  • csv-with-ids : adresses-with-ids-france.csv.gz, lieux-dits-with-ids-beta-france.csv.gz
  • csv-bal : adresses-france.csv.gz
  • addok : adresses-addok-france.ndjson.gz, addok-france-bundle.zip

Enhancements

1- Departement files : modify the current behavior that write the files on the server file storage to a behavior that does not right anything on the server but that upload the files on a S3. In order to work locally in dev mode, this PR also adds a "minio" container to the docker-compose.

2- National files : modify the current behavior using crons outside ban-plateforme (on the server) to a behavior that starts scripts using crons on the ban-plateforme worker.

How to test

1- Clone this branch.
2- Enter the right values for the following env variables :

MINIO_ROOT_USER=minio
MINIO_ROOT_PASSWORD=minio123
MINIO_PORT=9000
MINIO_FRONT_PORT=9001

3- Start the docker compose :
docker compose up --build -d

4- For department export files, start :
yarn dist

5- For national files, start :
node scripts/create-and-upload-national-files-to-s3.js

@antoineludeau antoineludeau force-pushed the antoineludeau/export-files-to-S3 branch from 8d9add8 to 1405e35 Compare February 13, 2024 16:25
@antoineludeau antoineludeau force-pushed the antoineludeau/export-files-to-S3 branch 2 times, most recently from 1cd5112 to dafbe96 Compare March 4, 2024 14:45
@antoineludeau antoineludeau force-pushed the antoineludeau/export-files-to-S3 branch from dafbe96 to 2b84afa Compare March 5, 2024 15:59
@antoineludeau antoineludeau marked this pull request as ready for review March 6, 2024 16:00
@nkokla nkokla marked this pull request as draft April 4, 2024 14:40
@vinsag vinsag requested review from jbouhadoun and removed request for FLoreauIGN and mmortier February 28, 2025 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

transformer les scripts lancés par cron en scripts intégrés au code produire les fichiers d'exports sur le S3

1 participant