|
| 1 | +--- |
| 2 | +name: upgrade-python |
| 3 | +description: workflow to rebuild all packages against a newer python version |
| 4 | +--- |
| 5 | + |
| 6 | +Adds support for a new Python version by building all packages, identifying failures, and marking them with `python_versions` restrictions. |
| 7 | + |
| 8 | +## Usage |
| 9 | + |
| 10 | +``` |
| 11 | +/upgrade-python <full-version> |
| 12 | +``` |
| 13 | + |
| 14 | +Example: `/upgrade-python 3.14.3` |
| 15 | + |
| 16 | +The argument is the full Python version (e.g., `3.14.3`). Derive the major.minor (`3.14`) and cpython tag (`cp314`) from it. |
| 17 | + |
| 18 | +## Step 1: Modify files for single-version build mode |
| 19 | + |
| 20 | +Edit these files to build ONLY the new Python version (this speeds up CI dramatically): |
| 21 | + |
| 22 | +### `build.py` line 35 |
| 23 | +Change `PYTHONS` to only the new version: |
| 24 | +```python |
| 25 | +PYTHONS = ((3, 14),) # temporarily building only new version |
| 26 | +``` |
| 27 | + |
| 28 | +### `validate.py` line 19 |
| 29 | +Same change: |
| 30 | +```python |
| 31 | +PYTHONS = ((3, 14),) # temporarily building only new version |
| 32 | +``` |
| 33 | + |
| 34 | +### `docker/install-pythons` line 9 |
| 35 | +Change `VERSIONS` to only the new full version: |
| 36 | +```python |
| 37 | +VERSIONS = ("3.14.3",) # temporarily building only new version |
| 38 | +``` |
| 39 | + |
| 40 | +### `docker/Dockerfile` |
| 41 | + |
| 42 | +**Line 39** — keep only the new cpython PATH entry (the base image already provides python3.11 on PATH): |
| 43 | +```dockerfile |
| 44 | + PATH=/venv/bin:/opt/python/cp314-cp314/bin:$PATH \ |
| 45 | +``` |
| 46 | + |
| 47 | +**Line 51** — use `python3.11` directly (provided by the base image) instead of the installed cpython path: |
| 48 | +```dockerfile |
| 49 | + && python3.11 -m venv /venv \ |
| 50 | +``` |
| 51 | + |
| 52 | +### `.github/workflows/build.yml` |
| 53 | + |
| 54 | +The production workflow uses pre-built `:latest` images. During a Python upgrade, the Dockerfile has changes that aren't in the production image yet, so you must temporarily add an `image` job that builds and pushes the modified image, and wire the `linux` job to use it. |
| 55 | + |
| 56 | +**Add a temporary `image` job** before the `linux` job: |
| 57 | +```yaml |
| 58 | + image: |
| 59 | + strategy: |
| 60 | + matrix: |
| 61 | + include: |
| 62 | + - {arch: amd64, os: ubuntu-latest} |
| 63 | + - {arch: arm64, os: ubuntu-24.04-arm} |
| 64 | + runs-on: ${{ matrix.os }} |
| 65 | + permissions: |
| 66 | + packages: write |
| 67 | + steps: |
| 68 | + - uses: actions/checkout@v3 |
| 69 | + - run: docker login --username '${{ github.actor }}' --password-stdin ghcr.io <<< '${{ secrets.GITHUB_TOKEN }}' |
| 70 | + - run: | |
| 71 | + docker buildx build \ |
| 72 | + --cache-from ghcr.io/getsentry/pypi-manylinux-${{ matrix.arch }}-ci:latest \ |
| 73 | + --cache-to type=inline \ |
| 74 | + --platform linux/${{ matrix.arch }} \ |
| 75 | + --tag ghcr.io/getsentry/pypi-manylinux-${{ matrix.arch }}-ci:${{ github.sha }} \ |
| 76 | + ${{ github.ref == 'refs/heads/main' && format('--tag ghcr.io/getsentry/pypi-manylinux-{0}-ci:latest', matrix.arch) || '' }} \ |
| 77 | + --push \ |
| 78 | + docker |
| 79 | +``` |
| 80 | +
|
| 81 | +**Modify the `linux` job** to depend on `image` and use the SHA-tagged image: |
| 82 | +```yaml |
| 83 | + linux: |
| 84 | + needs: [image] |
| 85 | + ... |
| 86 | + container: ghcr.io/getsentry/pypi-manylinux-${{ matrix.arch }}-ci:${{ github.sha }} |
| 87 | +``` |
| 88 | + |
| 89 | +**macOS PATH entries** — keep only the new cpython PATH entry: |
| 90 | +```yaml |
| 91 | + - run: | |
| 92 | + echo "$PWD/pythons/cp314-cp314/bin" >> "$GITHUB_PATH" |
| 93 | + echo "$PWD/venv/bin" >> "$GITHUB_PATH" |
| 94 | +``` |
| 95 | + |
| 96 | +**Add `--upgrade-python` flag** to both linux and macos build commands: |
| 97 | +```yaml |
| 98 | + - run: python3 -um build --pypi-url https://pypi.devinfra.sentry.io --upgrade-python |
| 99 | +``` |
| 100 | + |
| 101 | +## Step 2: Commit and push |
| 102 | + |
| 103 | +Commit all changes with a message like "build: single-version mode for Python 3.14 upgrade" and push. |
| 104 | + |
| 105 | +## Step 3: Wait for CI, then download and parse logs |
| 106 | + |
| 107 | +Wait for CI to complete (it will likely fail — that's expected). |
| 108 | +Use `gh run watch` to avoid rate limits if polling. |
| 109 | + |
| 110 | +Download logs from each build job using the GitHub CLI: |
| 111 | + |
| 112 | +```bash |
| 113 | +# Get the workflow run |
| 114 | +gh run list --branch <current-branch> --limit 1 |
| 115 | +
|
| 116 | +# Get job IDs from the run |
| 117 | +gh run view <run-id> --json jobs --jq '.jobs[] | {name: .name, id: .databaseId, conclusion: .conclusion}' |
| 118 | +
|
| 119 | +# Download logs for each job |
| 120 | +gh api repos/getsentry/pypi/actions/jobs/<job-id>/logs > job-<name>.log |
| 121 | +``` |
| 122 | + |
| 123 | +Parse the logs to identify: |
| 124 | +- **Succeeded packages**: lines matching `=== <name>==<version>@<python>` that are NOT followed by `!!! FAILED:` or `!!! SKIPPED` |
| 125 | +- **Failed packages**: lines matching `!!! FAILED: <name>==<version>: <error message>` |
| 126 | +- **Skipped packages**: lines matching `!!! SKIPPED (newer version already failed): <name>==<version>` — these are older versions that were auto-skipped because a newer version of the same package already failed |
| 127 | + |
| 128 | +A package is considered failed if it failed or was skipped on ANY platform (linux-amd64, linux-arm64, macos). |
| 129 | + |
| 130 | +## Step 4: Update `packages.ini` |
| 131 | + |
| 132 | +In one pass: |
| 133 | + |
| 134 | +1. **Remove** all packages that **succeeded on all platforms** — delete their entire section (`[name==version]` header + all config lines). The `format-packages-ini` pre-commit hook uses `configparser` which strips `#` comments, so commenting out doesn't work. Just delete succeeded sections outright. They don't need to rebuild. |
| 135 | + |
| 136 | +2. **Add `python_versions = <MAJOR.MINOR`** to each **failed** package's section. Do NOT add `#` comments above — `configparser` strips them and can cause the `python_versions` line to be lost during formatting. |
| 137 | + |
| 138 | +3. **Do NOT modify** packages that already have a `python_versions` restriction that is stricter than or equal to the new version (e.g., if a package already has `python_versions = <3.13`, leave it alone). |
| 139 | + |
| 140 | +## Step 5: Write detailed failure summary to `PYTHON-MAJOR-MINOR-UPGRADE.md` |
| 141 | + |
| 142 | +After parsing logs, create a file named `PYTHON-MAJOR.MINOR-UPGRADE.md` (e.g., `PYTHON-3.14-UPGRADE.md`) in the repo root with a detailed summary of all packages that failed to build. This serves as a reference for fixing build issues. The file should contain: |
| 143 | + |
| 144 | +- A header with the Python version and date |
| 145 | +- A table or list of every failed package with: |
| 146 | + - Package name and version |
| 147 | + - Which platform(s) it failed on (linux-amd64, linux-arm64, macos, or all) |
| 148 | + - The error message / root cause extracted from the logs |
| 149 | + - A category for the failure (e.g., "Cython incompatibility", "pyo3 version too old", "missing C API", "setuptools/distutils issue", etc.) |
| 150 | +- A summary section grouping failures by category with counts, so we can prioritize which categories to tackle first |
| 151 | + |
| 152 | +Example structure: |
| 153 | +```markdown |
| 154 | +# Python 3.14 Upgrade — Build Failures |
| 155 | +
|
| 156 | +## Summary by category |
| 157 | +| Category | Count | Packages | |
| 158 | +|----------|-------|----------| |
| 159 | +| pyo3 too old | 5 | pkg1, pkg2, ... | |
| 160 | +| Cython incompatibility | 3 | pkg3, pkg4, ... | |
| 161 | +
|
| 162 | +## Detailed failures |
| 163 | +### pkg1==1.2.3 |
| 164 | +- **Platforms**: all |
| 165 | +- **Category**: pyo3 too old |
| 166 | +- **Error**: pyo3 0.22.2 only supports up to Python 3.13 |
| 167 | +``` |
| 168 | + |
| 169 | +## Step 6: Commit, push, repeat |
| 170 | + |
| 171 | +Commit with a message like "mark python 3.14 build failures in packages.ini" and push. |
| 172 | + |
| 173 | +Wait for CI again. If there are still failures, repeat steps 3-6 until CI is green. |
| 174 | + |
| 175 | +## Step 7: Restore deleted packages and verify |
| 176 | + |
| 177 | +After CI is green, restore all previously-succeeded packages that were deleted in step 4. These packages don't need to rebuild (their existing wheels are fine), but they must remain in `packages.ini` so future builds include them. |
| 178 | + |
| 179 | +Use a script to find sections present in the pre-deletion commit but missing from the current file: |
| 180 | + |
| 181 | +```python |
| 182 | +import configparser, subprocess |
| 183 | +
|
| 184 | +old_content = subprocess.check_output(['git', 'show', '<pre-deletion-commit>:packages.ini']).decode() |
| 185 | +old = configparser.RawConfigParser(strict=False) |
| 186 | +old.read_string(old_content) |
| 187 | +
|
| 188 | +with open('packages.ini') as f: |
| 189 | + cur = configparser.RawConfigParser(strict=False) |
| 190 | + cur.read_string(f.read()) |
| 191 | +
|
| 192 | +missing = set(old.sections()) - set(cur.sections()) |
| 193 | +# Exclude any packages intentionally not restored |
| 194 | +
|
| 195 | +# Append missing sections to packages.ini |
| 196 | +with open('packages.ini', 'a') as f: |
| 197 | + for section in sorted(missing): |
| 198 | + f.write(f'\n[{section}]\n') |
| 199 | + for k, v in old[section].items(): |
| 200 | + v = v.strip() |
| 201 | + if '\n' in v: |
| 202 | + f.write(f'{k} =\n') |
| 203 | + for part in v.split('\n'): |
| 204 | + if part.strip(): |
| 205 | + f.write(f' {part.strip()}\n') |
| 206 | + else: |
| 207 | + f.write(f'{k} = {v}\n') |
| 208 | +``` |
| 209 | + |
| 210 | +Then run `python3 -m format_ini packages.ini` to sort and format. The formatter handles ordering automatically. |
| 211 | + |
| 212 | +Commit and push. Verify CI passes — all packages should either download pre-built wheels or build successfully. |
| 213 | + |
| 214 | +## Step 8: Revert single-version mode |
| 215 | + |
| 216 | +After all packages are restored and CI passes, revert the single-version mode changes from step 1 to build all Python versions again: |
| 217 | + |
| 218 | +1. **`build.py`**: Change `PYTHONS` back to all versions (e.g., `((3, 11), (3, 12), (3, 13), (3, 14))`) |
| 219 | +2. **`validate.py`**: Same change to `PYTHONS` |
| 220 | +3. **`docker/install-pythons`**: Change `VERSIONS` back to all versions (e.g., `("3.11.14", "3.12.12", "3.13.12", "3.14.3")`) |
| 221 | +4. **`docker/Dockerfile`**: Restore all cpython paths in `PATH` env var |
| 222 | +5. **`.github/workflows/build.yml`**: |
| 223 | + - Remove the temporary `image` job entirely |
| 224 | + - Remove `needs: [image]` from the `linux` job |
| 225 | + - Change the `linux` container back to `:latest` tag (from `:${{ github.sha }}`) |
| 226 | + - Remove `--upgrade-python` flag from both linux and macos build commands |
| 227 | + - Restore all cpython PATH entries in the macos job |
| 228 | + |
| 229 | +Commit with a message like "revert single-version mode, build all python versions" and push. Verify CI passes with all Python versions building. |
| 230 | + |
| 231 | +## Important notes |
| 232 | + |
| 233 | +- The `--upgrade-python` flag in `build.py` enables continue-on-failure mode with a 10-minute timeout per package. Without it, builds fail on first error (normal behavior). |
| 234 | +- The `=== name==version@python`, `!!! FAILED: name==version: error`, and `!!! SKIPPED (newer version already failed): name==version` log lines are the markers used to parse results. |
| 235 | +- In `--upgrade-python` mode, packages are sorted newest-version-first within each name. If the newest version fails, all older versions are automatically skipped. |
| 236 | +- Do NOT try to comment out sections with `#` — the `format-packages-ini` pre-commit hook uses Python's `configparser` which strips all `#` comments. Instead, delete succeeded sections entirely. |
| 237 | +- The `format-packages-ini` hook also sorts and reformats `packages.ini`, so ordering is handled automatically. |
0 commit comments