update stac, weaver, magpie, twitcher, cowbird - security fixes for EOL Python and http-related libraires#622
Open
fmigneault wants to merge 22 commits intomasterfrom
Open
update stac, weaver, magpie, twitcher, cowbird - security fixes for EOL Python and http-related libraires#622fmigneault wants to merge 22 commits intomasterfrom
fmigneault wants to merge 22 commits intomasterfrom
Conversation
…OL Python and http-related libraires
github-actions
bot
added
documentation
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3908/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca
|
Member
Author
|
run tests |
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3940/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-216.rdext.crim.ca
|
mishaschwartz
requested changes
Jan 8, 2026
Collaborator
mishaschwartz
left a comment
mishaschwartz
left a comment
There was a problem hiding this comment.
There's a change in the way that magpie creates cookies now that means that weaver cannot connect to magpie in the weaver/post-docker-compose script:
On line 150 of that script you currently have:
cookie_jar=$(echo "${cookie_jar}" | grep -v '# ' | grep -v -e '^$' | grep -v '_\.')where that last grep filters out lines with _. in them. This was supposed to deal with the fact that magpie was supplying two identical cookies for the domains:
#HttpOnly_.${BIRDHOUSE_FQDN_PUBLIC}#HttpOnly_${BIRDHOUSE_FQDN_PUBLIC}
Now it seems that magpie is only sending a cookie for the domain with the . (#HttpOnly_.${BIRDHOUSE_FQDN_PUBLIC}) which means that line 150 of weaver/post-docker-compose filters out the only provided cookie and then reports that it cannot connect to Magpie.
I'm not sure what the best solution is but we should either update weaver/post-docker-compose or we should investigate why magpie cookies have changed (was that intentional?)
Member
Author
|
@mishaschwartz I think this is caused by internal library changes (pyramid / urllib3) that behave slightly differently, notably when I guess the fix would be to consider the opposite |
Member
Author
|
@mishaschwartz @tlvu |
Collaborator
That would work. If you want to make it a bit more future-proof though you could just include all cookies from the jar. You'd just have to delimit the cookies with a
All IPs it looks like, not just on a dev server. |
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3943/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-216.rdext.crim.ca
|
mishaschwartz
requested changes
Jan 14, 2026
Collaborator
mishaschwartz
left a comment
mishaschwartz
left a comment
There was a problem hiding this comment.
I've added my suggested fixes for the logging in #634
mishaschwartz
approved these changes
Jan 16, 2026
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3969/Result ❌ FAILUREBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-216.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/607/NOTEBOOK TEST RESULTS |
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3973/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-216.rdext.crim.ca
|
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3974/Result ❌ FAILUREBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/611/NOTEBOOK TEST RESULTS |
tlvu
approved these changes
Jan 21, 2026
Member
Author
|
@tlvu @mishaschwartz |
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/4059/Result ❌ FAILUREBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/672/NOTEBOOK TEST RESULTS |
…to clarify them / make it easier to update
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/4060/Result ❌ FAILUREBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/673/NOTEBOOK TEST RESULTS |
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/4061/Result ❌ FAILUREBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : false PAVICS_HOST : https://host-140-91.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/674/NOTEBOOK TEST RESULTS |
…fer options to handle larger link header results from weaver
Member
Author
|
New proxy buffer options applied and working here: |
Member
Author
|
@tlvu @mishaschwartz |
Collaborator
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/4063/Result ✅ SUCCESSBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-133.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/676/NOTEBOOK TEST RESULTS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Update multiple components with corresponding updates of
urllib,requests, etc.At the same time, bump to Python 3.13 versions as applicable.
Changes
Non-breaking changes
STAC API: Security update, minor OpenAPI version reporting fixes, and
stac-fastapi/starlettecompatibility fix using 2.3.0__version__for OpenAPI response crim-ca/stac-app#69Cowbird: Security update to version 2.6.0
Magpie: Security update to version 4.3.0
Twitcher: Security update to version 0.11.0
Weaver: Security and dependency fix update using version 6.8.3
setuptoolsto resolve itspkg_resourcesdependency still required bypyramidcrim-ca/weaver#877Weaver: Update
post-docker-compose-upscript.This can happen depending on specific internal HTTP libraries versions of the services.
To retain backward/forward compatibility, all cookies returned from Magpie are chained in following
curlcommands.logutility to report operations produced by the script rather than custom "echo level".Weaver: Job Result Proxy Buffers
weavercan return a lot ofLinkheaders. This is done to provide job metadatareferences and provenance traceability details, but also for actual results locations that can vary in quantity
depending on the actual process execution.
Therefore, the Ngnix
proxy_buffer_sizeandproxy_buffersdirectives of theproxyservice must be added withsufficiently large values to avoid HTTP 502 errors when the response headers exceed the default buffer sizes.
The
WEAVER_PROXY_RESPONSE_BUFFER_SIZEandWEAVER_PROXY_RESPONSE_BUFFER_COUNTvariables are added to allowfurther customization as needed by the server. Their defaults are reasonable values to meet minimal requirements
by
weaver's metadataLinkand a few result outputs.Birdhouse: Allow
log <LEVEL> -n ...andlog <LEVEL> -p ...to generate log outputs without newline/prefixes.These options allow writing multiple log entries onto the same line for correct visual rendering of distinct
logcalls separated to allow some intermediate logic. The
logfunction invocations with these options respect thelog levels in order to make the messages consistent with enabled redirections and verbosity.
Breaking changes
CI Operations
birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false
Warning
To Do / Self Note