Add comparison to loaders.gl GeoArrowLoader

[alexkreidler]

loaders.gl has a GeoArrowLoader which supports all the GeoArrow types and loads GeoArrow into a ColumnarTable which can be passed to deck.gl.

It seems like @geoarrow/deck.gl-layers is higher-level and implements some boilerplate arrow accessors and does some
geometric computations for common layer types, but my understanding here may be off.

I'm curious if @geoarrow/deck.gl-layers is different from GeoArrowLoader specifically on these points from the README:
copying binary buffers directly from an Arrow Table to the GPU using deck.gl's binary data interface, and doing multi-threaded polygon triangulation

It would be awesome to add a short paragraph comparison to the README or docs.

[kylebarron]

👋

It's a good question, and we should have better documentation about it.

I think the tl;dr is that @geoarrow/deck.gl-layers is focused on the absolute fastest rendering from GeoArrow, with as minimal data transformation as possible. While the GeoArrowLoader is an adapter to an existing legacy binary interface that has more overhead.

deck.gl/loaders.gl has had a "binary features" concept for a while. I was actually part of that effort in 2020, see visgl/loaders.gl#685, visgl/loaders.gl#690, visgl/loaders.gl#703.

GeoArrow was just starting early discussions at the same time in 2020, and GeoArrow was not mature enough to use. So deck.gl started building tooling around its own custom format. This means that several deck.gl layers support the binary features type.

But GeoArrow has progressed over time and is now in much broader use than deck.gl's own custom format. And to me, it's very valuable that GeoArrow is a language-independent spec, so that the binary data format can be exactly the same in multiple languages. It means we can do wild stuff like direct copies of point and linestring data from Python inside WebAssembly, straight to the GPU, because the Arrow binary layout is the same in Python memory as it is in JS.

So I want to build tooling around GeoArrow.

GeoArrowLoader is an adapter interface that will parse GeoArrow into either GeoJSON or their own custom interface. If you're parsing GeoArrow to binary features, there's likely to be some overhead, but not a lot. If you're parsing GeoArrow to GeoJSON, there's a LOT of overhead, and the entire point of GeoArrow is to remove all that overhead, since deck.gl just takes that GeoJSON and repacks into binary buffers that essentially match GeoArrow anyways.

I'm not sure on the status of which layers accept binary features. Last I knew, the MVTLayer supported binary features but not all layers supported it. And even when layers do support it, like the GeoJSONLayer, I'm pretty sure there's more overhead to rendering binary features.

• @geoarrow/deck.gl-layers will always use worker threads for polygon triangulation unless the worker fetch failed or was explicitly turned off. It looks like loaders.gl code can do polygon triangulation off the main thread, but I believe that is only the case when the loader is already off the main thread. That means that the GeoArrowLoader might not be doing the triangulation off the main thread unless that loader is already in a worker (I'm not sure whether it is).
• GeoArrowLoader doesn't support separated GeoArrow coordinates, it only supports interleaved coordinates (for the record @geoarrow/deck.gl-layers also doesn't support separated coordinates yet)

Hopefully this helps

[alexkreidler]

Thank you, that's a really insightful explanation with lots to dive into! I really appreciate you writing it!