Building `linux/arm64` Image is Extremely Slow

[eval-exec]

linux/amd64 need 14m, but build linux/arm64 need 3h26m: https://github.com/nervosnetwork/ckb-docker-builder/actions/runs/7204742933/job/19628095845