feat(wanda): implement artifact extraction using docker cp

[andrew-anyscale]

Extract artifacts from built images using docker cp for cross-platform reliability

• Create container from image without starting it
• Copy each artifact using docker cp
• Clean up container when done
• Extraction runs for base target only. Skips on cache hit.

Example:

❯  PYTHON_VERSION=3.13 MANYLINUX_VERSION=260128.221a193 HOSTTYPE=aarch64 ARCH_SUFFIX=-aarch64 BUILDKITE_COMMIT=b5737cefc0 IS_LOCAL_BUILD=true /Users/andrew/devel/rayci-wanda-artifacts/_release/wanda-darwin-arm64 --artifacts_dir .whl/ ci/docker/ray-wheel.wanda.yaml

...

2026/02/04 09:10:39 extracting 1 artifact(s) from localhost:5000/rayci-work:ray-wheel-py3.13-aarch64
Successfully copied 72MB to /Users/andrew/devel/ray-local-wheel-build/.whl/ray-3.0.0.dev0-cp313-cp313-manylinux2014_aarch64.whl
2026/02/04 09:10:40 extracted 1 artifact(s) in 722ms:
2026/02/04 09:10:40   /Users/andrew/devel/ray-local-wheel-build/.whl/ray-3.0.0.dev0-cp313-cp313-manylinux2014_aarch64.whl

Topic: wanda-artifact-copy

Signed-off-by: andrew andrew@anyscale.com

[andrew-anyscale]

Reviews in this chain:
└#411 feat(wanda): implement artifact extraction using docker cp
　└#409 feat(wanda): add artifacts_dir CLI flag
　　└#410 feat(raycicmd): set RAYCI_ARTIFACTS_DIR for wanda steps

[andrew-anyscale]

#	head	base	diff	date	summary
0	a3b19fc3	70e0db32	diff	Feb 3 14:18 PM	11 files changed, 349 insertions(+), 2 deletions(-)
1	80ab6e0b	25c63e1c	rebase	Feb 3 14:24 PM	0 files changed
2	81385ecc	25c63e1c	diff	Feb 3 17:51 PM	2 files changed, 6 insertions(+), 6 deletions(-)
3	1e3c63b9	f1104ff7	diff	Feb 4 14:22 PM	8 files changed, 253 insertions(+), 30 deletions(-)
4	217cc4d4	313a1f83	diff	Feb 4 14:27 PM	0 files changed
5	34ea72c0	7a60b1e6	diff	Feb 4 15:12 PM	0 files changed
6	868f0c18	7a60b1e6	diff	Feb 4 15:29 PM	2 files changed, 6 deletions(-)
7	f210797c	7a60b1e6	diff	Feb 5 8:32 AM	1 file changed, 7 insertions(+)
8	f343d80f	928b35c4	diff	Feb 5 11:38 AM	1 file changed, 7 insertions(+), 3 deletions(-)
9	44d0999e	928b35c4	diff	Feb 5 11:59 AM	1 file changed, 8 insertions(+)
10	3eb89c41	5adb6641	diff	Feb 9 15:55 PM	7 files changed, 57 insertions(+), 226 deletions(-)
11	074fb9a2	da274a09	diff	Feb 10 7:25 AM	1 file changed, 11 insertions(+), 7 deletions(-)

[gemini-code-assist]

Summary of Changes

Hello @andrew-anyscale, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the wanda build system by enabling the extraction of specified artifacts directly from built Docker images. This new capability, implemented using docker cp, provides a robust and cross-platform compatible way to retrieve files or directories from containers. It ensures that artifacts are extracted efficiently and reliably, with options for handling missing optional artifacts and clear logging for performance insights.

Highlights

• Artifact Extraction Implementation: Introduced functionality to extract artifacts from built Docker images using docker cp.
• Cross-Platform Reliability: The chosen docker cp method ensures reliable artifact extraction across different operating systems, including Windows.
• Controlled Container Lifecycle: Artifacts are extracted by creating a temporary container from the image, copying the specified files, and then removing the container, all without starting the container.
• Root Spec Only Extraction: Artifact extraction is performed exclusively for the root specification, even when a cache hit occurs for the image build.
• Optional Artifact Handling: Support for optional artifacts has been added, allowing the system to log a warning instead of failing if an optional artifact is not found.
• Performance Monitoring: The duration of the artifact extraction process is now logged to facilitate performance monitoring.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• wanda/docker_cmd.go
  • Added createContainer function to create a Docker container without starting it.
  • Added copyFromContainer function to copy files/directories from a container to the host.
  • Added removeContainer function to remove a Docker container.
• wanda/docker_cmd_test.go
  • Imported the os package.
  • Added TestDockerCmdCopyFromContainer to verify copying a single file.
  • Added TestDockerCmdCopyFromContainer_directory to verify copying a directory.
  • Added TestDockerCmdCopyFromContainer_notFound to test error handling for non-existent files.
• wanda/forge.go
  • Imported the time package.
  • Modified the Build function to trigger artifact extraction for the root spec if ArtifactsDir is configured and artifacts are defined.
  • Implemented ExtractArtifacts function, which orchestrates the container creation, artifact copying via docker cp, and container removal, including logging and optional artifact handling.
  • Added resolveArtifactDst helper function for determining artifact destination paths.
  • Updated the Build function's documentation to mention artifact extraction.
• wanda/forge_config.go
  • Added ArtifactsDir field to the ForgeConfig struct to specify the destination for extracted artifacts.
• wanda/forge_test.go
  • Added TestForgeConfigArtifactsDir to verify the ArtifactsDir configuration.
  • Added TestBuild_WithArtifacts_exact to test the extraction of a specific artifact.
  • Added TestBuild_WithArtifacts_optional to test the handling of optional artifacts.
  • Added TestBuild_WithArtifacts_rootOnly to ensure artifacts are extracted only from the root spec.
  • Added TestBuild_WithArtifacts_cacheHit to confirm artifact extraction occurs even on a cache hit.
• wanda/testdata/Dockerfile.artifact
  • New Dockerfile used for testing artifact extraction, creating various test files.
• wanda/testdata/Dockerfile.artifact-top
  • New Dockerfile for testing artifact extraction in a multi-stage or dependent build scenario.
• wanda/testdata/artifact-dep-base.wanda.yaml
  • New Wanda YAML specification defining a base image with an artifact to be extracted.
• wanda/testdata/artifact-dep-top.wanda.yaml
  • New Wanda YAML specification defining a top-level image that depends on artifact-dep-base and has its own artifact.
• wanda/testdata/artifact-exact.wanda.yaml
  • New Wanda YAML specification for testing exact artifact extraction.
• wanda/testdata/artifact-optional.wanda.yaml
  • New Wanda YAML specification for testing optional artifact extraction.

Activity

• The pull request introduces a new feature for artifact copying.
• The author, andrew-anyscale, has implemented the core logic for docker cp operations and integrated it into the wanda build process.
• The PR is currently marked as a draft, indicating ongoing work or awaiting initial feedback.
• The changes are categorized under the wanda-artifact-copy topic and are related to wanda-artifact-spec.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to copy artifacts from built Docker images using docker cp. The implementation is straightforward, adding new docker command wrappers and integrating artifact extraction into the build process. The changes are well-tested, covering file and directory copying, optional artifacts, and behavior on cache hits.

My review focuses on improving code style and correctness in the tests and documentation. Specifically:

• Using filepath.Join for path construction in tests to improve portability and adhere to Go idioms.
• Correcting a misleading comment in wanda/forge.go.

I also noticed some code duplication in the new test functions in wanda/docker_cmd_test.go. While I haven't added a specific comment, you might consider refactoring the common setup (pulling image, creating container) into a helper function to make the tests more concise.

Overall, this is a solid contribution that adds valuable functionality.

[aslonnie]

• why we are not using crane this time?
• what will happen when wanda cache hits?

[andrew-anyscale]

why we are not using crane this time?

I didn't see any capabilities for copying out individual files/directories outside of exporting the file system. I can rework if there is an entrypoint I am missing. https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane.md

what will happen when wanda cache hits?

Extraction runs for root spec only, even on cache hit. This is so that CI doesn't have missing artifacts in the case of a cache hit.

[aslonnie]

Extraction runs for root spec only, even on cache hit. This is so that CI doesn't have missing artifacts in the case of a cache hit.

sorry what does "root spec" mean?

[andrew-anyscale]

Root spec (I'm realizing 'base target' is more appropriate) is the spec that wanda is running against. I.e. for wanda ci/docker/ray-wheel.wanda.yaml, Ray wheel would be the root spec

[aslonnie]

yeah, it is just the crane export. behind the scene it is implemented like this:

https://github.com/google/go-containerregistry/blob/93aa2732266afabbde66bd80d55563e339056432/pkg/crane/export.go#L27

docker cp is faster if the image is already locally available, but it is slower if it is a wanda remote cache hit. when wanda remote cache hits, the image building is just a image retag on the remote CR; it does not pull down the image. where docker create will pull down the image (effectively runs a docker pull) and extract it on disk first.

that said.. wheel building never cache hits... however, on the other hand, the wheel output image is also small to perform export (it only contains the wheels).

maybe we can 1. use docker cp always, and 2. just drop the artifact exporting when it is cache hit..

[aslonnie]

nit: maybe use heredoc / multi-line?

[aslonnie]

maybe just drop the unused ? the "unused" is treated as a "command" which is optional on create.

[andrew-anyscale]

When testing on the wheel container, not having any input command causes it to fail out on creation--

❯  docker create cr.ray.io/rayproject/ray-wheel-py3.13-aarch64:latest
Error response from daemon: no command specified

[aslonnie]

$ docker create ubuntu:focal
508194d4ed04d9af40683f40dc8596693e7136bbef01b2a778d75d06461d4795

I think it is because the image does not have a CMD, which then demands docker create to specify an explicit CMD.

nit: instead of "unused", maybe use true? with a comment explaining what it is and why?

[andrew-anyscale]

Done, and added comment--

// createContainer creates a container from an image without starting it.
// Returns the container ID.
func (c *dockerCmd) createContainer(image string) (string, error) {
	// "true" is a no-op command required for images without CMD/ENTRYPOINT.
	// The container is never started, so the command doesn't actually run.
	out, err := c.output("create", image, "true")
	if err != nil {
		return "", err
	}
	return strings.TrimSpace(string(out)), nil
}

[aslonnie]

why not use c.run here?

[aslonnie]

maybe create a helper as c.output() for capturing the output of a docker subcommand?

[aslonnie]

this can be quite expensive.. and it is for supporting glob I guess. if we are doing this anyways, I would rather we do crane export so that we do not require having a docker daemon.

which gets to my point:

• maybe just drop globing support?
• and only support copying out a directory (which is supported by docker cp)
• and we can change the docker build files to copy files into /opt/artifacts directory or something so that we get the required result.

[andrew-anyscale]

Makes sense. I'll go ahead and drop glob support here to avoid any forms of this type of iteration

[andrew-anyscale]

Pushed an update to remove glob support, and simplify these flows

[aslonnie]

maybe return []byte, and convert to string and apply TrimSpace by the caller? []byte is the more generic format, where string and TrimSpace are only for particular outputs.

[andrew-anyscale]

Excellent to know, thank you! Made the change