-
Notifications
You must be signed in to change notification settings - Fork 0
feat: Improve reproducibility and templating using mise #13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Bafbi
wants to merge
44
commits into
main
Choose a base branch
from
feat/improve-deployment
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
44 commits
Select commit
Hold shift + click to select a range
6ecced8
feat: Add initial configuration and automation tasks for infrastructu…
Bafbi a6d15ff
refactor: Remove unused Terraform resources and variables for cleaner…
Bafbi 50e6fdf
feat: Update environment configuration and add service account key ge…
Bafbi 772ed8d
refactor(infra): rename dbt service account var and cleanup outputs
Bafbi 1baf6e0
feat(dbt): update profiles template to use dynamic env vars
Bafbi f20d011
chore(env): update mise config and env example for dynamic vars
Bafbi 2062ccb
feat(prefect): replace static config with template for dynamic git url
Bafbi bde4aad
feat(infra): use dataset suffix vars and local dataset ids (rework ma…
Bafbi 6799391
feat(dbt): update template to use env and tf context + add rendered p…
Bafbi 9297949
feat(scripts): add render_template and setup_prefect_blocks CLIs (ren…
Bafbi d1a7d79
chore(mise): add dbt render & prefect setup tasks; remove deprecated …
Bafbi 59638a1
feat(prefect): add rendered prefect.yml, update template usage and te…
Bafbi 103ed47
chore(gitignore): stop tracking terraform-outputs.json (now added to …
Bafbi 1e67b4a
chore(gitignore): add terraform-outputs.json to .gitignore
Bafbi c3a753c
refactor(infra): remove deprecated infrastructure/setup_profiles modu…
Bafbi b4cffc8
docs: update mise workflow
Bafbi 9ade7bf
chore: scripts and ignore tweaks
Bafbi fbaf7e9
fix: some nitpick from code review
Bafbi bc6cfe1
chore: update .gitignore and untrack templated files
Bafbi 74014b0
chore: apply nitpicks
Bafbi 6014ce8
fix: update service account and variable definitions in mise.toml
Bafbi 35c7c1f
fix: needing the .env to enable mise
Bafbi 438d16f
fix: update environment variable references in mise.toml for consistency
Bafbi fb6255b
fix: replace local variables with inline expressions for dataset IDs …
Bafbi 55e47ec
fix: update .gitignore to ignore tfvars files and clean up mise.toml …
Bafbi f7fda74
feat: add 'infra:apply' task to run 'opentofu apply' and update usage…
Bafbi 31c5834
feat: enhance mise.toml with setup task and improve infra dependencies
Bafbi 38f677d
feat: update task dependencies in mise.toml for improved setup and in…
Bafbi d29790f
feat: update service account key generation task to use default email…
Bafbi 1c65a91
feat: update dbt and prefect tasks to include additional sources and …
Bafbi 1d12e68
Merge branch 'main' into feat/improve-deployment
Bafbi 2fceac6
feat: Setup Prefect and IaC config to support prod environment (#17)
CyprienKelma 114c56a
hotfix: refactor/prod dataset (#18)
CyprienKelma 030b7ca
chore: fix description content of fact table
CyprienKelma 6738666
hotfix: update Prefect deployment
CyprienKelma fdcc637
hotfix: update dbt commands to include target specification in setup_…
CyprienKelma bd19068
refactor: minor changes on Prefect deployment scripts
CyprienKelma aa32d8a
chore: add dbt dependency installation step to Prefect deployment scr…
CyprienKelma 74ade0e
hotfix: update dbt command execution
CyprienKelma 1fe66c5
chore: update Prefect deployment configuration and refine model descr…
CyprienKelma 1be56f3
Feat/bucket idempotency (#19)
CyprienKelma 917f8bd
chore: update .gitignore to include presentation assets and remove ob…
CyprienKelma 844955f
docs: add MOE POC and related documentation (presentation, annexes, g…
Bafbi e886bb2
chore(mise): add typst to tools in mise.toml
Bafbi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| GCLOUD_PROJECT= | ||
| # list for available regions: https://cloud.google.com/about/locations ; auto : https://cloud.withgoogle.com/region-picker/ | ||
| GCLOUD_REGION= | ||
| PREFECT_API_URL= | ||
| PREFECT_API_KEY= | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
.env.examplefile is missing several environment variables that are referenced in the templates and scripts, includingGCP_SA_KEY_PATH,BQ_DEV_DATASET,BQ_PROD_DATASET, and potentially others. Add these missing variables with appropriate comments to guide users in setting up their environment correctly.