[Question]: GLIBC issue on arm64v8/amazonlinux:latest

[noel-ooh]

Product

Amazon Linux 2023

What is your question?

We have golang Lambda Functions built on AWS' arm64v8/amazonlinux:latest. Last week, we started seeing a runtime error on cold start saying GLIBCXX_3.4.30 not found. Despite reverting our Lambda changes to the last known successful deployment, it still gave the same error.

So, we looked into the container we use for building the Lambda, and saw there was an update to arm64v8/amazonlinux:latest . We then tried to pin our build container to the previous tagged version of that container (arm64v8/amazonlinux:2023.6.20250317.2) and it is now working again without the glibc error on cold start.

Given the only difference is in our build container (arm64v8/amazonlinux:latest vs arm64v8/amazonlinux:2023.6.20250317.2), I would like to think there is something in the latest updates that caused the glibc issue. Could this be a potential regression on the latest image?

Reproducible repo using CDK is available at https://github.com/noel-ooh/amazonlinux-glib-issue.

[ozbenh]

This is a side effect of our updating of the gcc core runtimes to accomodate gcc 14. This is not actually glibc but libstdc++ (GLIBCXX_... symbols refer to libstdc++). Due to an issue with how gcc is packaged, unfortunately, since the update, things built with the old gcc 11 end up inheriting from a symbol version of the new library (they shouldn't, it's a bug, though one shared by other Linux distros as well).

We are working on releasing a fixed gcc (11) which will enforce linking against its own library versions (even if you can still use the new ones at runtime), but whatever was linked with the current one will need to be rebuilt to drop the bogus dependency once that is out.

For now, you need to update your container image (or update libstdc++ and other gcc runtime libs in your container image).

[noel-ooh]

@ozbenh Thank you for the response. None of my attempts to update libstdc++ have fixed the issue in my case.

Would you be able to look at the git repo above with the reproduction and point out what is missing? Here's the Dockerfile.

Would you suggest we pin our build container to arm64v8/amazonlinux:2023.6.20250317.2 since it seems you are working on a fixed gcc?

[ozbenh]

It might be that you are hitting the issue before you get a chance to update libstdc++ ? Versions 14 of libstdc++ definitely provides the missing symbol version.

That said yes, you could pin. The fix isn't in the release that went out today but should be in the next one if all our testing passes.

[noel-ooh]

@ozbenh I can't seem to update libstdc++ to version 14. dnf install just gives me 11.5.0. Same as gcc version.

[ozbenh]

You are running an old container image, it's locked to the old version of the repository. Update to a newer release and you should get the new library.

[ozbenh]

(either update the container image itself or run dnf check-release-update to see what inline updates are available)

[ozbenh]

Actually the latter is only available if you install dnf-plugin-release-notification which is not on the container image by default. Remember that AL2023 releases are locked by default to the version of the repository they were released with, you need to explicitely move them to a new version

[noel-ooh]

You are running an old container image, it's locked to the old version of the repository. Update to a newer release and you should get the new library.

I was using arm64v8/amazonlinux:latest.

[ozbenh]

We have been having some delays with docker hub updates, you may have a better experience using ECR when such thing happen ? It's rare that we produce such an update that requires updated libraries or runtimes but not impossible