Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
6ecced8
feat: Add initial configuration and automation tasks for infrastructu…
Bafbi Nov 17, 2025
a6d15ff
refactor: Remove unused Terraform resources and variables for cleaner…
Bafbi Nov 17, 2025
50e6fdf
feat: Update environment configuration and add service account key ge…
Bafbi Nov 17, 2025
772ed8d
refactor(infra): rename dbt service account var and cleanup outputs
Bafbi Nov 21, 2025
1baf6e0
feat(dbt): update profiles template to use dynamic env vars
Bafbi Nov 21, 2025
f20d011
chore(env): update mise config and env example for dynamic vars
Bafbi Nov 21, 2025
2062ccb
feat(prefect): replace static config with template for dynamic git url
Bafbi Nov 21, 2025
bde4aad
feat(infra): use dataset suffix vars and local dataset ids (rework ma…
Bafbi Nov 28, 2025
6799391
feat(dbt): update template to use env and tf context + add rendered p…
Bafbi Nov 28, 2025
9297949
feat(scripts): add render_template and setup_prefect_blocks CLIs (ren…
Bafbi Nov 28, 2025
d1a7d79
chore(mise): add dbt render & prefect setup tasks; remove deprecated …
Bafbi Nov 28, 2025
59638a1
feat(prefect): add rendered prefect.yml, update template usage and te…
Bafbi Nov 28, 2025
103ed47
chore(gitignore): stop tracking terraform-outputs.json (now added to …
Bafbi Nov 28, 2025
1e67b4a
chore(gitignore): add terraform-outputs.json to .gitignore
Bafbi Nov 28, 2025
c3a753c
refactor(infra): remove deprecated infrastructure/setup_profiles modu…
Bafbi Nov 28, 2025
b4cffc8
docs: update mise workflow
Bafbi Nov 28, 2025
9ade7bf
chore: scripts and ignore tweaks
Bafbi Nov 28, 2025
fbaf7e9
fix: some nitpick from code review
Bafbi Nov 28, 2025
bc6cfe1
chore: update .gitignore and untrack templated files
Bafbi Nov 28, 2025
74014b0
chore: apply nitpicks
Bafbi Nov 29, 2025
6014ce8
fix: update service account and variable definitions in mise.toml
Bafbi Nov 29, 2025
35c7c1f
fix: needing the .env to enable mise
Bafbi Nov 29, 2025
438d16f
fix: update environment variable references in mise.toml for consistency
Bafbi Nov 29, 2025
fb6255b
fix: replace local variables with inline expressions for dataset IDs …
Bafbi Nov 29, 2025
55e47ec
fix: update .gitignore to ignore tfvars files and clean up mise.toml …
Bafbi Nov 29, 2025
f7fda74
feat: add 'infra:apply' task to run 'opentofu apply' and update usage…
Bafbi Nov 29, 2025
31c5834
feat: enhance mise.toml with setup task and improve infra dependencies
Bafbi Nov 29, 2025
38f677d
feat: update task dependencies in mise.toml for improved setup and in…
Bafbi Nov 29, 2025
d29790f
feat: update service account key generation task to use default email…
Bafbi Nov 29, 2025
1c65a91
feat: update dbt and prefect tasks to include additional sources and …
Bafbi Nov 29, 2025
1d12e68
Merge branch 'main' into feat/improve-deployment
Bafbi Nov 29, 2025
2fceac6
feat: Setup Prefect and IaC config to support prod environment (#17)
CyprienKelma Nov 29, 2025
114c56a
hotfix: refactor/prod dataset (#18)
CyprienKelma Nov 29, 2025
030b7ca
chore: fix description content of fact table
CyprienKelma Nov 29, 2025
6738666
hotfix: update Prefect deployment
CyprienKelma Nov 29, 2025
fdcc637
hotfix: update dbt commands to include target specification in setup_…
CyprienKelma Nov 29, 2025
bd19068
refactor: minor changes on Prefect deployment scripts
CyprienKelma Nov 29, 2025
aa32d8a
chore: add dbt dependency installation step to Prefect deployment scr…
CyprienKelma Nov 30, 2025
74ade0e
hotfix: update dbt command execution
CyprienKelma Nov 30, 2025
1fe66c5
chore: update Prefect deployment configuration and refine model descr…
CyprienKelma Nov 30, 2025
1be56f3
Feat/bucket idempotency (#19)
CyprienKelma Nov 30, 2025
917f8bd
chore: update .gitignore to include presentation assets and remove ob…
CyprienKelma Dec 4, 2025
844955f
docs: add MOE POC and related documentation (presentation, annexes, g…
Bafbi Dec 13, 2025
e886bb2
chore(mise): add typst to tools in mise.toml
Bafbi Dec 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
GCLOUD_PROJECT=
# list for available regions: https://cloud.google.com/about/locations ; auto : https://cloud.withgoogle.com/region-picker/
GCLOUD_REGION=
PREFECT_API_URL=
PREFECT_API_KEY=
Comment on lines +1 to +5
Copy link

Copilot AI Nov 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The .env.example file is missing several environment variables that are referenced in the templates and scripts, including GCP_SA_KEY_PATH, BQ_DEV_DATASET, BQ_PROD_DATASET, and potentially others. Add these missing variables with appropriate comments to guide users in setting up their environment correctly.

Copilot uses AI. Check for mistakes.
5 changes: 2 additions & 3 deletions .github/workflows/publish-dbt-doc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ jobs:
run: |
cd dbt
uv run dbt deps

- name: Generate dbt documentation
run: |
cd dbt
Expand All @@ -97,7 +97,7 @@ jobs:
mkdir -p public/dbt/assets/img/mart/core/fct__viewings/
cp dbt/assets/img/mart/core/fct__viewings/fct__viewings_model.html public/dbt/assets/img/mart/core/fct__viewings/
cp dbt/assets/img/mart/core/fct__viewings/fct__viewings_model.mmd public/dbt/assets/img/mart/core/fct__viewings/

- name: Setup Pages
uses: actions/configure-pages@v4

Expand All @@ -109,4 +109,3 @@ jobs:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4

270 changes: 141 additions & 129 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,73 +1,67 @@
# Created by https://www.toptal.com/developers/gitignore/api/dbt,linux,macos,python,windows,terraform,dotenv
# Edit at https://www.toptal.com/developers/gitignore?templates=dbt,linux,macos,python,windows,terraform,dotenv

# Local .terraform directories
.terraform/
### dbt ###
target/
dbt_modules/
dbt_packages/
logs/

# .tfstate files
*.tfstate
*.tfstate.*
### dotenv ###
.env

# Crash log files
crash.log
crash.*.log
### Linux ###
*~

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version
# control as they are data points which are potentially sensitive and subject
# to change depending on the environment.
*.tfvars
*.tfvars.json
# temporary files which can be created if a process still has a handle open of a deleted file
.fuse_hidden*

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json
# KDE directory preferences
.directory

# Ignore transient lock info files created by terraform apply
.terraform.tfstate.lock.info
# Linux trash folder which might appear on any partition or disk
.Trash-*

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf
# .nfs files are created when an open file is removed but is still being accessed
.nfs*

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*
### macOS ###
# General
.DS_Store
.AppleDouble
.LSOverride

# Ignore CLI configuration files
.terraformrc
terraform.rc
# Icon must end with two \r
Icon

# Optional: ignore graph output files generated by `terraform graph`
# *.dot

# Optional: ignore plan files saved before destroying Terraform configuration
# Uncomment the line below if you want to ignore planout files.
# planout
# Thumbnails
._*

# OpenTofu
.tofu/
tofu.tfstate
tofu.tfstate.backup
# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent

# Secrets
.secrets/
# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk

# Virtualenv
# https://realpython.com/python-virtual-environments-a-primer/#the-virtualenv-project
.Python
[Bb]in
[Ii]nclude
[Ll]ib
[Ll]ib64
[Ll]ocal
[Ss]cripts
pyvenv.cfg
.venv
pip-selfcheck.json
### macOS Patch ###
# iCloud generated files
*.icloud

### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[codz]
*.py[cod]
*$py.class

# C extensions
Expand All @@ -94,8 +88,8 @@ share/python-wheels/
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

Expand All @@ -113,7 +107,7 @@ htmlcov/
nosetests.xml
coverage.xml
*.cover
*.py.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
Expand Down Expand Up @@ -141,7 +135,6 @@ docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints
Expand All @@ -160,37 +153,22 @@ ipython_config.py
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
# Pipfile.lock

# UV
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# uv.lock
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
# poetry.lock
# poetry.toml
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
# pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
# https://pdm-project.org/en/latest/usage/project/#working-with-version-control
# pdm.lock
# pdm.toml
.pdm-python
.pdm-build/

# pixi
# Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
# pixi.lock
# Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
# in the .venv directory. It is recommended not to include this directory in version control.
.pixi
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
Expand All @@ -199,25 +177,10 @@ __pypackages__/
celerybeat-schedule
celerybeat.pid

# Redis
*.rdb
*.aof
*.pid

# RabbitMQ
mnesia/
rabbitmq/
rabbitmq-data/

# ActiveMQ
activemq-data/

# SageMath parsed files
*.sage.py

# Environments
.env
.envrc
.venv
env/
venv/
Expand Down Expand Up @@ -250,50 +213,99 @@ dmypy.json
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
# .idea/

# Abstra
# Abstra is an AI-powered process automation framework.
# Ignore directories containing user credentials, local state, and settings.
# Learn more at https://abstra.io/docs
.abstra/

# Visual Studio Code
# Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore
# that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
# and can be added to the global gitignore or merged into this file. However, if you prefer,
# you could uncomment the following to ignore the entire vscode folder
# .vscode/

# Ruff stuff:
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

### Python Patch ###
# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
poetry.toml

# ruff
.ruff_cache/

# PyPI configuration file
.pypirc
# LSP config files
pyrightconfig.json

# Marimo
marimo/_static/
marimo/_lsp/
__marimo__/
### Terraform ###
# Local .terraform directories
**/.terraform/*

# Streamlit
.streamlit/secrets.toml
.DS_Store
.vscode/
# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version
# control as they are data points which are potentially sensitive and subject
# to change depending on the environment.
# *.tfvars
# *.tfvars.json

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

### Windows ###
# Windows thumbnail cache files
Thumbs.db
Thumbs.db:encryptable
ehthumbs.db
ehthumbs_vista.db

# Dump file
*.stackdump

# Folder config file
[Dd]esktop.ini

# Recycle Bin used on file shares
$RECYCLE.BIN/

# direnv
.envrc
.direnv/
# Windows Installer files
*.cab
*.msi
*.msix
*.msm
*.msp

# local profil.yml
# Windows shortcuts
*.lnk

# End of https://www.toptal.com/developers/gitignore/api/dbt,linux,macos,python,windows,terraform,dotenv

# Secrets and sensitive files
.secrets/
terraform-outputs.json

# Templated files
dbt/profiles.yml
prefect.yml

# raw data generated by seed script
raw_data/

# UML assets
dbt/assets/img/mart/core
dbt/assets/img/mart/core

# Presentation Assests
presentation/assets/
Loading