Hello, Prebuilders

[zeke]

Found about this org via hypermodules/hyperamp#194 (comment)

I don't really know anything about C++, nan, n-api, node-pre-gyp, or any of the other jazz that goes into native node modules. I am, however, willing to help in an organized effort to improve the native module experience for end-users of node, such that they are scarcely aware of their very existence.

So... how do we get to native module utopia?

[zeke]

@juliangruber @ralphtheninja @mafintosh @ungoldman @bcomnes

[ralphtheninja]

@zeke

Hi! And thanks for spearheading this! ❤️

Some Background

Currently there are several different approaches as how to handle prebuilt binaries, node-pre-gyp and prebuild (I think there are others as well)

I believe @mafintosh started working on prebuild to have a simple(r) way of making prebuilt binaries. If you take a look at the node-pre-gyp README, it's rather massive and there's a lot of information to deal with in order to setup a system for your native module.

For this purpose I think simplicity is a key feature. It should be easy to set up and use.

We have used prebuild in leveldown for a while now and it has been working really well. Together with prebuild-ci, building and publishing your binaries is deferred to whatever ci system you're running, e.g. Travis, AppVeyor.

leveldown does this by invoking prebuild-ci during the test phase. prebuild-ci basically checks the version in package.json and compares it with the previous commit to see if anything should be done. If so, prebuild runs and uploads the binaries to github (uses ghreleases to create a new release), see leveldown releases.

Whenever the user installs a module that depends on a native module with prebuilt binaries, it can use prebuild-install to find the binary needed for the current system (based on os, node version etc). Some lore; prebuild used to do this, but this was refactored out to prebuild-install to avoid having a massive dependency just for downloading binaries. leveldown does that here. So if prebuild-install fails, it falls back to node-gyp building the binary.

[ralphtheninja]

cc @mathiask88 @lgeiger @piranna

[ralphtheninja]

@zeke Some projects using this setup https://www.npmjs.com/browse/depended/prebuild

Also, @mafintosh has been working on prebuildify which if I understand correctly is an attempt to actually bundle all prebuilts inside the npm tar ball. So whenever you download something from npm, you have all prebuilts already and just need to unpack them. This is cool because it's super fast.

Check out sodium-native for an example how this approach works.

I think this is the way we want to go, i.e. prebuildify will become the new prebuild (as renamed to prebuild and new major version).

@mafintosh Please correct me if I'm wrong here

[ralphtheninja]

Please let me know if anyone wants to be invited to the organization. I also think that everyone should be owner since it enables people and removes useless friction :)

[bcomnes]

This good info. Ty

[ungoldman]

🖖 I would be interested in helping. The model I'm most familiar with is electron-prebuilt, which is fantastic for the sake of avoiding building from source on every install.

Does tarballing every target distro and including that on npm mean incurring a significant download size increase for all users?

Is this preferable to a postinstall script which is sometimes gross and side-effecty?

Happy to experiment and come up with a good strategy, or to help recruit other projects into adopting prebuild. leveldb, node-sass, sqlite seem like big communities that would benefit from prebuilds a great deal. I look forward to a day when I don't see an install fail because of a weird node-gyp race condition.

[ralphtheninja]

Does tarballing every target distro and including that on npm mean incurring a significant download size increase for all users?

At first glance it might look significant. If you take a look at the binaries for leveldown@1.7.2, there are roughly 50 files and each is around 200kb, so it's like 10Mb (just rough calculations here).

It's worth noting here that npm caches modules, so in theory it should be really fast for consecutive downloads. Although we trade off some disk space for it.

But there's ways to reduce this total size (@mafintosh can pitch in on details here), if need to be. If you're writing a simple c++ layer yourself it might not be worth it, but in the sodium-native case there's huge gain by building libsodium as a shared lib and linking with it (as opposed to what leveldown does, which always compile all c++ code, both the c++ side for leveldown and the c++ leveldb lib by google).

[mathiask88]

Hey, nice to see some action regarding the org :)
@juliangruber created an org month ago https://github.com/prebuild but there was no further progress. See prebuild/prebuild#158 Edit: Sry, @piranna created the org and @juliangruber moved his prebuild-ci ;)

@ralphtheninja there are even more projects using prebuild https://www.npmjs.com/browse/depended/prebuild does not list the projects where prebuild is a devDependency. Try https://www.npmjs.com/browse/depended/prebuild-install instead.

[piranna]

I'm using prebuild and prebuild-install for NodeOS prebuild images and also have done some useful tools like buildDependencies to download dependencies needed to build the module only if a prebuild image is not available, and buho to check for new versions of upstream C/C++ code on a CI server and generate the new versiones images.

[mathiask88]

Happy to experiment and come up with a good strategy, or to help recruit other projects into adopting prebuild. leveldb, node-sass, sqlite seem like big communities that would benefit from prebuilds a great deal. I look forward to a day when I don't see an install fail because of a weird node-gyp race condition.

@ungoldman I created a list of packages that depend on nan sorted by download count per months. Many of the big projects already have some kind of prebuild handling e.g. node-sass does the same prebuild and prebuild-install does but with an own implementation.

[zeke]

I created a list of packages that depend on nan sorted by download count

Ha! Me too.

https://github.com/nice-registry/native-modules#readme

[mathiask88]

Nice and I see your list is missing scoped packages (~400) as well. I didn't find a way to bring the npm rest api to list these. And you check the devDependencies 👍

[zeke]

list is missing scoped packages

Ah thanks for noticing. I'm collecting scoped packages but was not rendering them in the README. The skimdb.npmjs.com service doesn't serve up scoped packages, but the new-ish replicate.npmjs.com service does. My package-stream module and the underlying changes-stream make it pretty easy to consume metadata from that new endpoint.

Working on an updated list now..

[lgeiger]

Thanks for getting the discussion started!

We've been using prebuild and prebuild-install in zeromq for some time now and it greatly reduced the number of installation issues we receive. 💯
We're using zeromq heavily in an Electron context with electron-builder and even inside an Atom package. As long as people omit the --build-from-source flag the experience has been pretty good so far (apart from the need to rebuild when working with Electron). prebuildify sounds like a good solution if one doesn't mind the extra package size.

A complaint we sometimes get is that prebuild-install doesn't behave well for people behind a proxy (I'm not behind a proxy so I can't easily reproduce).

Has anybody already tried N-API and will it be able to work across Node and Electron?