install diff-filter-buildkite-plugin to only build when relevant files changed

[fatteneder]

Fixes #243

This adds a filter that skips (most of the) build and test steps in the CI pipeline if a PR only changed files that are irrelevant for Julia's functionality.
In particular, changes to any of these files won't trigger a build (glob patterns as seen from the root of the Julia repo):

• "*.md"
• ".*"
• "julia.spdx.json"
• "CITATION.*"
• "typos.toml"

Note that we always build for x86_64-linux-gnu in order to generate the docs.

The filter can be disabled on a PR basis by adding a label with the name ci-force-build to it.

The addition of more fine grained filters is left for future PRs.

[IanButterworth]

I believe NEWS.md and HISTORY.md are in the docs, so I think it needs at least the docs to be built for those?

[fatteneder]

I think I managed to implement a mechanism to override the filter using Github labels and the github-pr-labels-buildkite-plugin.
If someone could add a label with the name ci-force-build to this PR, then it should build everything again.

---

I think the basic functionality is now there.
Currently it contains lots of boilerplate code in various places, so I would like to refactor it.

• Is there a simple way to share the filter definition between all relevant *.yml files?
• Is it possible to run the diff-filter plugin once before all steps and just export the trigger to the remaining steps?

[giordano]

I don't know if this is what the ignore-list is for (it's quite short, so can't tell it's what I'm thinking of), but for changes which involve only any of

• .devcontainer
• .github (JuliaLang/julia#56468 shouldn't have run buildkite CI)
• .clang-format
• .clangd
• .git-blame-ignore-revs
• .gitignore
• .mailmap
• CITATION.bib
• CITATION.cff
• julia.spdx.json
• typos.toml

shouldn't run any CI here at all. After this, for docs-related changes I think we need probably to build for all platforms but not run any test suites apart from the doctests? Not sure if this is what #400 (comment) was suggesting.

[giordano]

I guess this should be moved to this organisation before merge.

[DilumAluthge]

• manually ignoring the filter for pre-releases and tags,
• ignore the filter when merging into master.

This can likely be simplified to: apply the filter iff the repo is JuliaLang/julia and it is a PR.

[fatteneder]

After this, for docs-related changes I think we need probably to build for all platforms but not run any test suites apart from the doctests? Not sure if this is what #400 (comment) was suggesting.

Following #400 (comment) we now always build x86_64-linux, and then use that to run deploy_docs.yml, build_pdf_docs.yml and doctest.yml.

[fatteneder]

This can likely be simplified to: apply the filter iff the repo is JuliaLang/julia and it is a PR.

Ok, I did not yet test how the plugin behaves once a PR is merged.

About juliaup and the +pr feature: Not exactly sure how that works, does that just grab any artifact that is cached in buildkite? If so, then with this PR here one can no longer grab a build from a "filtered" CI run, unless one adds the ci-force-build label.

[fatteneder]

I have not permissions on this repo/in this org. Can someone please add the ci-force-build label here for testing?

[DilumAluthge]

Closing and reopening a PR doesn't retrigger build kite unfortunately. I asked Bill kite Support about that a while back, and they told me that was is the intended behavior.

you'll need to go to the build kite build and click the rebuild button.

[inkydragon]

Bump.

I was wondering if it would be possible to add another layer/step to check if other signed/unsigned CI jobs are to be started.
Perhaps this would avoid some duplication.

[WebUI] -> CheckIfLaunchCI -> launch unsigned & signed_jobs

[fatteneder]

@inkydragon I don't understand your request. Could you expand a bit?