Use pywb to create single wacz files

[zhabiba24]

Really cool to have this pywb tool for creating archives programatically...

Currently I'm trying to use pywb to create a single wacz record that corresponds to a given URL. However it seems right now to create warc.gz files which correspond to an entire archiving session.

My ideal flow:

1. Make a GET request to a local running pywb server
2. Receive a single wacz file that corresponds that to that URL

So far I have go tthis working:

uv run wb-manager init test-cli-archive
uv run wayback --record --live -a --auto-interval 10
curl http://localhost:8080/test-cli-archive/record/<MY URL>

This then gives me this file:

collections/test-cli-archive/archive/rec-20260123155603209431-<machine id>.warc.gz

But this is a warc.gz file of everything in this archiving session, when in reality I just want a single wacz that corresponds to the URL I just archived.

Is there a way to achieve this workflow?

[zhabiba24]

Just realised I can do this with https://github.com/webrecorder/py-wacz feel free to close

[zhabiba24]

Hm actually doing that with a command like

uv run wacz create -o test.wacz collections/test-cli-archive/archive/rec-20260123155603209431-<machine id>.warc.gz

Gives me a workable wacz but it seems to be missing a lot of information? e.g. not all images on the page has loaded on simple not paywalled newspaper site.

I'm initiating the crawl by doing curl http://localhost:8080/test-cli-archive/record/<URL> is there a better way to do that so that I get more high quality output here in wacz format?

[ato]

The recording mode just records any requests made through it. curl doesn't request images so none are recorded. It's often used interactively with a user manually loading pages in a browser but you could, for example, run Chrome or Firefox in --headless mode against it.

However, I would instead recommend using browsertrix-crawler as it has builtin support for WACZ output, manages running the browser for you and will do things like automatically scrolling down the page to force everything to load.

[ikreymer]

@zhabiba24 yes, as Alex mentioned, browsertrix-crawler is the tool that you want - it'll be able to create WACZ files of pages for you automatically from command line. For manual WACZ creation, we recommend https://archiveweb.page

[zhabiba24]

Thank you!