Alternative build system to #59

[rossbar]

This draft PR follows from the discussion in #59. To test this build system:

pip install -r requirements.txt
make html

This is the standard Makefile you get from sphinx-quickstart. I incorporated @stefanv's fixes to make some of the examples run again, but there were some that haven't been fixed yet, e.g. adv{3, 4, 5}_*. Even though these notebooks encounter exceptions, they site is still generated and can still be viewed in the browser (e.g. firefox _build/html/index.html). This is really handy for reviewing and works will with common CI site preview workflows. The caching is now handled by jupyter_cache rather than being handled in a Makefile. It's not perfect (and doesn't handle dependencies anywhere near as nicely as a Makefile does) but should have the property that only notebooks that have been modified will be executed on a rebuild.

[rossbar]

I used a script to autoconvert all the .ipynb files to markdown-based notebook format, but this really gummed up the PR with diffs that aren't relevant for the discussion... I will back those changes out.

[rossbar]

I managed to figure out how to unwind my "convert all ipynb -> md w/ jupytext" commit and limit the conversions to only those files that are actually used in the site (so far). The toctrees in introduction.md and applications.md summarize which files have been converted.

[rossbar]

I'm partial to CircleCI for building sphinx sites because of the nice preview options but I didn't bother adding a whole new CI service in this PR. In the absence of a CI preview, here's what the site looks like rendered as of 8b7a95a : https://rossbar.github.io/skimage-tutorials/

[stefanv]

Hey, Ross, this looks great and is a huge improvement over what we have right now. I'm happy to merge, but there's a small issue with the "imageconverts" package not found.

[rossbar]

Hey, Ross, this looks great and is a huge improvement over what we have right now. I'm happy to merge, but there's a small issue with the "imageconverts" package not found.

Gah a silly mistake, it should be imagecodecs

[stefanv]

The build failed, but it's hard to see why. How about adding those logs as build artifacts?

[rossbar]

How about adding those logs as build artifacts?

One of the advantages of this build system is that failures in the notebook execution actually show up in the rendered site. The adv3, adv4, and adv5 notebooks all fail to execute to completion (this is expected, they didn't execute in #59 either) and you can see the failures in the rendered site. For example,

• adv3: https://rossbar.github.io/skimage-tutorials/lectures/adv3_panorama-stitching.html#feature-detection-and-matching
• adv4: https://rossbar.github.io/skimage-tutorials/lectures/adv4_warping.html#here-s-a-template-solution
• adv5: https://rossbar.github.io/skimage-tutorials/lectures/adv5_pores.html

It looks like there was also a cell execution timeout in three_dimensional_image_processing. myst-nb throws a cell execution timeout error cells that take longer than 30 seconds to execute (the timeout period is configurable).

I've found that CircleCI is really a great fit for this workflow because you can upload the _build/html folder to get both the execution logs (which aren't as useful as they could be) as well as the site for previewing, even when the execution fails.

[stefanv]

@rossbar Would you like to add CircleCI now or do that in another PR? I'm happy with either.

[alexdesiqueira]

Hey @stefanv @rossbar,
I think we forgot this 🙂