Exposing DuckDB Attached Catalogs as Arrow Flight (airport-go based)

[realdatadriven]

Hi 👋

I’ve been experimenting with Arrow Flight + DuckDB in a small project called duck-flight:
👉 https://github.com/realdatadriven/duck-flight

It’s built on top of airport-go and exposes a single Arrow Flight catalog, where schemas are defined via YAML.
Each schema comes from different DuckDB attached catalog, and automatically discovers, tables, and fields (that are converted to arrow schemas) based on the config.

I’ve implemented the core ideas in Go with Copilot's help, but I'm still a low-level Go programmer. This is mainly an experiment driven by my interest in Arrow Flight and DuckDB, and I’d really appreciate feedback on:

• Architecture / design choices
• Go and Arrow Flight best practices
• Pitfalls or improvements

If this overlaps with your interests, I’d be happy to iterate or collaborate.

Thanks for airport-go — it’s been a great foundation 🙏

[VGSML]

Hi, thank you for using this package. I’ll take a closer look.

At first glance, it seems that you can use DuckDB’s Arrow functionality provided by duckdb-go to retrieve query results directly as Arrow data (arrow.QueryContext returns an arrow.RecordBatch). To inspect Arrow types, there is currently no dedicated API, but you can run a query like FROM <table_name> LIMIT 0 and obtain the Arrow schema from the returned record batch.

I think this approach can significantly simplify things and help avoid manual conversions. I'm going to start a discussion in the duckdb-go repository about adding a method to retrieve an ArrowSchema for tables directly.

I’m also planning to implement a similar SQL interface in Hugr, primarily focused on data ingestion.

[realdatadriven]

Thanks a lot for taking the time

Using DuckDB’s Arrow functionality to directly work with arrow.QueryContext and Using LIMIT 0 to retrieve the Arrow schema is a brilliant, and would simplify the implementation a lot.

Really appreciate the suggestion.

Thanks again!