🌪 0.52.1

Changes

🐛 Bug Fixes

• Dispatch each I/O request (#4464) @gatesn

🌪 0.52.0

Changes

✨ Features

• Make footer and segment cache sizes configurable in Datafusion (#4491) @AdamGS
• feat: VortexDataset.get_fragments yields one fragment per split (#4386) @danking
• Change evaluations to take MaskFuture (#4482) @gatesn
• Refactor ObjectStoreWriter (#4475) @AdamGS
• Feature: add to_canonical to ArrayBuilder (#4473) @connortsui20
• Feature: Add append_option method to every builder (#4455) @connortsui20
• Feature: Add eq_with_nullability_subset and eq_with_nullability_superset and validate extend_from_array (#4445) @connortsui20

🐛 Bug Fixes

• fix[vortex-array]: don't use input list field type in preferred conversion (#4481) @asubiotto
• Remove caching the array on flat layout reader, rely on the segment cache instead (#4459) @onursatici
• fix[array]: fill_null with nullable scalar must return nullable array (#4472) @joseph-isaacs
• Fix: CanonicalVTable::to_canonical and apped_to_builder doesn't return errors (#4441) @robert3005
• fix[layout]: remove println (#4454) @joseph-isaacs
• test[mask] cannot convert nullable bool array to mask (#4429) @joseph-isaacs

🧰 Maintenance

23 changes

• chore(deps): update rust crate zip to v4.6.1 (#4490) @renovate[bot]
• chore: rudimentary Python benchmarks & support more Polars types (#4442) @danking
• chore: #[inline] fns vortex-dtype (#4486) @0ax1
• chore(deps): update rust crate clap to v4.5.47 (#4488) @renovate[bot]
• chore: info -> debug logging (#4484) @a10y
• chore: #[inline] fns vortex-scan (#4483) @0ax1
• chore: #[inline] fns vortex-array (#4477) @0ax1
• chore: #[inline] fns vortex-buffer (#4479) @0ax1
• chore: #[inline] fns vortex-mask (#4478) @0ax1
• chore(deps): update codspeedhq/action action to v4 (#4471) @renovate[bot]
• feat[encodings/dict]: allow different nullability of codes and values in dict (#3736) @joseph-isaacs
• chore(deps): update rust crate uuid to v1.18.1 (#4467) @renovate[bot]
• Chore: Clean up PrimitiveBuilder API (#4457) @connortsui20
• fix(deps): update all patch updates to v1.0.174 (patch) (#4465) @renovate[bot]
• chore(deps): Update tracing-subscriber (#4463) @blaginin
• Chore: ArrayBuilder refactor (#4452) @connortsui20
• chore(deps): update rust crate cc to v1.2.35 (#4456) @renovate[bot]
• Chore: Make all ArrayBuilders consistent (#4450) @connortsui20
• chore(deps): update rust crate bindgen to v0.72.1 (#4451) @renovate[bot]
• chore(deps): update rust crate zip to v4.6.0 (#4446) @renovate[bot]
• chore(deps): update rust crate tracing-subscriber to v0.3.20 (#4439) @renovate[bot]
• chore: GetItem to apply parent nullability (#4425) @blaginin
• chore(deps): update rust crate hashbrown to 0.16.0 (#4432) @renovate[bot]

🌪 0.51.0

Changes

✨ Features

• feat: expr -> operator conversion and use in layouts (#4388) @joseph-isaacs
• Feature: Add FixedSizeListBuilder (#4428) @connortsui20
• Add RepeatedScan in vortex-data python package (#4426) @delta003
• feat[fastlanes]: bitpacking can handle offset (#4382) @joseph-isaacs
• feat: prepared scan: accelerate successive range reads to a file (#4399) @danking
• Feature: Add optional git hooks scripts (#4414) @connortsui20
• Feature: Add FixedSizeListArray and FixedSizeListVTable (#4405) @connortsui20
• feat: teach Patches to fast-path search sorted on primitive array (#4416) @danking
• feat: teach DashMap to use a faster hasher (#4417) @danking
• feature: ValidityVTable functions are non fallible (#4412) @robert3005
• feat: teach patches a fast path for size one slice (#4415) @danking
• feat: Add BitOrd and Concat for Mask (#4407) @blaginin

🚀 Performance

• feat: expr -> operator conversion and use in layouts (#4388) @joseph-isaacs
• Replace Flume with Kanal (#4401) @AdamGS

🐛 Bug Fixes

• fix: When coercing scalars make sure values don't exceed maximum value (#4381) @robert3005
• fix: make cargo test work by hiding Python code from Rust (#4431) @danking
• Fix: ListArray validates offsets min/max against max offset value (#4430) @robert3005
• fix[vortex-array]: propagate preferred type conversion to nested fields (#4390) @asubiotto

🧰 Maintenance

15 changes

• Chore: Validate Scalar::new for null and clean up DecimalScalar (#4435) @connortsui20
• fix(deps): update all patch updates to v1.0.173 (patch) (#4437) @renovate[bot]
• chore: better cast errors (#4436) @blaginin
• fix(deps): update all patch updates to v1.0.172 (patch) (#4421) @renovate[bot]
• chore: more README citations (#4434) @lwwmanning
• chore(deps): update plugin com.palantir.consistent-versions to v3.2.0 (#4433) @renovate[bot]
• chore[duckdb]: lifetime to object cache (#4409) @joseph-isaacs
• chore(deps): update plugin com.gradleup.shadow to v9.1.0 (#4423) @renovate[bot]
• Chore: fix valid expect (#4422) @connortsui20
• chore: remove gradle inferred processors and update gradle jvm links (#4396) @robert3005
• Chore: Rename TimeUnit variants to be more descriptive (#4410) @connortsui20
• Chore: Rename as_struct_opt to as_struct_fields_opt (#4411) @connortsui20
• Chore: rename bad todos (#4413) @connortsui20
• chore[bench]: enable duckdb metadata caching for parquet (#4375) @joseph-isaacs
• chore: Unify slice methods to use Range arguments (#4391) @blaginin

🌪 0.50.0

Changes

✨ Features

• Feature: FixedSizeList infrastructure (#4385) @connortsui20
• Pretty tree display for Vortex expressions (#4373) @AdamGS
• Feature: Add FixedSizeList support in vortex-scalar (#4380) @connortsui20
• Feature: Add DType::FixedSizeList variant (#4371) @connortsui20

🚀 Performance

• performance: faster utf8 validation by using simd instructions (#4347) @robert3005

🐛 Bug Fixes

• fix: ubuntu uses dash which lacks pipefail (#4392) @danking
• fix: Add downcasts for dict codes and runend ends (#4384) @robert3005
• fix: don't recursively dictionary-encode dictionary codes (#4352) @asubiotto
• fix[vortex-datafusion]: add filename context on read error (#4369) @asubiotto
• fix: cleanup old to_record_batch_reader (#4368) @a10y

🧰 Maintenance

6 changes

• chore: prepare all before publish (#4397) @danking
• chore(deps): update plugin com.palantir.consistent-versions to v3 (#4394) @renovate[bot]
• fix(deps): update all patch updates to v1.0.170 (patch) (#4387) @renovate[bot]
• Remove old redundent display function in the fuzzer (#4377) @AdamGS
• chore(deps): update rust crate clap to v4.5.46 (#4376) @renovate[bot]
• chore(deps): update rust crate mimalloc to v0.1.48 (#4370) @renovate[bot]

🌪 0.49.0

Changes

✨ Features

• feat: show statpopgen benchmarks (#4359) @danking
• feat: statistical and population genetics benchmark queries and dataset (#4175) @danking

🐛 Bug Fixes

• fix: Always use setSafe when writing values from spark (#4367) @robert3005
• fix: python integration docs (#4363) @danking
• chore: revert to 20.04, enforce glibc version == 2.31 (#4354) @danking
• Revert "Re-open DuckDB context between iterations" (#4355) @lwwmanning

🧰 Maintenance

15 changes

• chore: exclude vortex-error from coverage report (#4366) @lwwmanning
• fix(deps): update all patch updates to v1.0.169 (patch) (#4365) @renovate[bot]
• Re-open DuckDB context between iterations (take 2) (#4362) @lwwmanning
• chore(deps): update rust crate pyo3 to 0.25.0 (#4364) @renovate[bot]
• chore(deps): update rust crate noodles-bgzf to 0.43.0 (#4360) @renovate[bot]
• chore(deps): update rust crate noodles-vcf to 0.81.0 (#4361) @renovate[bot]
• chore(deps): lock file maintenance (#4349) @renovate[bot]
• chore: Inline arrow-pyarrow crate to avoid upstream pyo3 version pin (#4342) @robert3005
• chore: update renovate.json (#4357) @lwwmanning
• chore(deps): update rust crate regex to v1.11.2 (#4348) @renovate[bot]
• chore: shard codspeed benchmarks (#4353) @lwwmanning
• chore(deps): update ubuntu docker tag to v24 (#4345) @renovate[bot]
• chore: add PyO3 environment variables (#4350) @connortsui20
• chore(deps): update datafusion and friends to v49.0.2 (patch) (#4351) @renovate[bot]
• Re-open DuckDB context between iterations (#4346) @gatesn

🌪 0.48.0

Changes

✨ Features

• feat: teach Vortex how to implement a Ray Datasource (#4339) @danking
• feat: teach VortexDataset get_fragments (#4340) @danking
• feat: vortex-python/check.sh (#4338) @danking
• feat: teach VortexDataset to filter & document vortex.expr.* (#4336) @danking

🐛 Bug Fixes

• fix: Remove ScalarValue::is_instance_of (#4341) @robert3005

🧰 Maintenance

4 changes

• fix: Java builds with GLIBC 2.31+ compatibility (#4331) @a10y
• chore(deps): update rust crate url to v2.5.7 (#4343) @renovate[bot]
• chore(deps): update actions/upload-pages-artifact action to v4 (#4329) @renovate[bot]
• fix: Buffer Clone bound (#4337) @a10y

🌪 0.47.0

Changes

✨ Features

• feat[vortex-array]: add stepped pipeline execution (#4312) @joseph-isaacs
• feat[dtype]: ptype visitor (#4307) @joseph-isaacs

🚀 Performance

• feat[vortex-array]: add stepped pipeline execution (#4312) @joseph-isaacs

🐛 Bug Fixes

• fix(java): Spark writer properly handles string and array data (#4335) @a10y
• fix: Spawn datafusion write tasks (#4316) @AdamGS
• fix: vortex-cxx examples macos build (#4308) @0ax1
• fix: Remove conversion from usize to scalar (#4304) @robert3005
• fix: downscale FSST offsets (#4301) @a10y
• feat[vortex-array]: support converting to timestamp arrow arrays with tz (#4285) @asubiotto

🧰 Maintenance

28 changes

• chore: pyright (#4289) @danking
• Clean up match statements and misc (#4332) @connortsui20
• chore: vortex-ffi testing & build improvements (#4218) @lwwmanning
• chore[bench]: have --skip-generate option and only create iterators for tpch when needed (#4328) @joseph-isaacs
• Add local .claude folder to .gitignore (#4325) @connortsui20
• chore: configure codecov comments to be more useful (#4326) @lwwmanning
• chore(deps): update all patch updates (patch) (#4323) @renovate[bot]
• chore: return lvalue ref for cxx's scanbuilder functions (#4306) @XinyuZeng
• chore(deps): update rust crate zip to v4.5.0 (#4320) @renovate[bot]
• Add the missing copyright profile file (#4318) @gatesn
• chore: Don't prune uv caches in CI (#4314) @robert3005
• chore: vortex-scalar cleanup (#4313) @connortsui20
• chore(deps): update actions/setup-java action to v5 (#4305) @renovate[bot]
• chore: Make downscale_integer_array a method on PrimitiveArray (#4303) @robert3005
• chore(deps): update rust crate url to v2.5.6 (#4309) @renovate[bot]
• chore(deps): update spiraldb/actions action to v0.15.0 (#4311) @renovate[bot]
• chore(deps): update rust crate zip to v4.4.0 (#4310) @renovate[bot]
• Chore: Add list of list tests (#4298) @connortsui20
• chore: move field_names tests to field_names.rs (#4300) @connortsui20
• Add test coverage badge to README (#4299) @AdamGS
• Report coverage to codecov (#4296) @AdamGS
• chore(deps): update rust crate inventory to v0.3.21 (#4297) @renovate[bot]
• chore(deps): update rust crate taffy to v0.9.1 (#4295) @renovate[bot]
• Improve test coverage for conversion between DataFusion and Vortex objects (#4294) @AdamGS
• chore(deps): update rust crate thiserror to v2.0.16 (#4293) @renovate[bot]
• chore(deps): update rust crate tempfile to v3.21.0 (#4292) @renovate[bot]
• chore: DType documentation (#4291) @connortsui20
• chore(deps): update rust crate cfg-if to v1.0.3 (#4290) @renovate[bot]

🌪 0.46.1

Changes

🧰 Maintenance

• fix: timeout crates publish after 1hr (#4288) @a10y

🌪 0.46.0

Changes

✨ Features

• feat: ArrayOperations infallible, eager validation + new_unchecked (#4177) @a10y
• feat: teach VortexDataset how to limit the batch_size (#4275) @danking
• chore[vortex-duckdb]: duckdb explain custom display (#4272) @joseph-isaacs

🚀 Performance

• Add more non copying stats operations; replace stats replace with inherit_from (#4279) @robert3005

🐛 Bug Fixes

• Fix: converting from duckdb_string_t requires looking at length (#4277) @robert3005
• fix[duckdb]: set max cardinality in duckdb (#4270) @joseph-isaacs

🧰 Maintenance

14 changes

• chore: infinite publish timeout (#4286) @a10y
• chore(deps): update polarsignals/gh-actions-ps-profiling action to v0.6.0 (#4284) @renovate[bot]
• chore: monad vs. monoid (#4283) @connortsui20
• Add Display impl for FieldNames (#4282) @AdamGS
• chore(deps): update rust crate serde_json to v1.0.143 (#4280) @renovate[bot]
• Chore: Cleanup in take (#4278) @connortsui20
• docs: Add documentation to take function (#4274) @connortsui20
• chore: hide aliases from docs (#4273) @danking
• chore: Add rust-analyzer to the rust-toolchain.tom file (#4271) @connortsui20
• Enable TPC-DS benchmarks for datafusion (#4269) @AdamGS
• Remove old dead test code (#4265) @AdamGS
• chore(deps): update datafusion and friends to v49.0.1 (patch) (#4267) @renovate[bot]
• chore(deps): lock file maintenance (#4264) @renovate[bot]
• chore(deps): update rust crate thiserror to v2.0.15 (#4262) @renovate[bot]

🌪 0.45.0

Changes

✨ Features

• feat[spark]: write support in VortexDataSourceV2 (#4246) @lwwmanning
• feat: implement Display for StructFields (#4255) @blaginin
• feat: Support more datafusion features (#4180) @AdamGS
• Support converting TimestampTZ to and from duckdb (#4226) @robert3005
• Use perfetto tracing (#4250) @gatesn
• feat: Remove 'parquet' feature and some other dependency bloat (#4251) @AdamGS
• feat: Make FieldNames comparison more ergonomic (#4239) @AdamGS
• feat(c++): support passing filter and projection expression in scan (#4224) @XinyuZeng
• feat[expr]: add back the expression folder, implemented over Node (#4230) @joseph-isaacs
• feature: Add TryFrom impls for owned scalars to values (#4234) @AdamGS
• docs: add javadoc to docs.vortex.dev (#4229) @a10y
• feat: Expose various expr transformations directly instead of through module (#4212) @AdamGS
• feat[duckdb]: support more operators (#4170) @joseph-isaacs
• feat: teach Vortex how to export list arrays to duckdb (#4169) @danking
• feat(python): Write PyArrow types directly to Vortex (#4161) @a10y
• Inline the logic of the filter layout reader (#4089) @gatesn
• Support exporting Arrow RunArray to Vortex (#4162) @robert3005
• feature: Support converting Arrow DictArray into vortex (#4157) @robert3005
• refactor the LayoutStrategy (#4149) @a10y

🚀 Performance

• perf: Use sum compute function (#4207) @viirya

🐛 Bug Fixes

20 changes

• fix: Validity export using shifted u64 copy (#4244) @robert3005
• fix[bench]: better bench input size (#4243) @joseph-isaacs
• fix[bench-site]: Compression Size group missing charts (#4236) @lwwmanning
• Fix: Return currect field names in SelectExpr::as_include (#4232) @AdamGS
• fix: javadoc workflow (#4231) @a10y
• fix: Correctly convert arrow temporal arrays with timezone (#4227) @robert3005
• fix: runend decompress benchmarks (#4225) @0ax1
• fix: Avoid running projection evaluation for all false masks (#4209) @robert3005
• fix: small resource leak in vortex-io, plus test improvements (#4201) @lwwmanning
• fix: race condition in vortex-layout, plus test improvements (#4198) @lwwmanning
• fix[bench-website]: add tpc-ds to bench website (#4199) @joseph-isaacs
• shared waker in struct writer (#4193) @onursatici
• fix: duckdb dyn expr race on filter constant (#4168) @0ax1
• fix[scan]: don't convert filter to cnf just extract conjunct list (#4174) @joseph-isaacs
• fix: Fix some minor typos and remove the "single-file" feature from clickbench (#4182) @AdamGS
• fix: incorrect decimal scalar nbytes (#4176) @lwwmanning
• fix(pyvortex): encoding-specific dispatch (#4171) @a10y
• fix[duckdb]: validity ref u64 not u8 (#4172) @joseph-isaacs
• ScalarValue operations must go via Scalar (#4152) @robert3005
• Only coerce scalar values when necessary (#4135) @robert3005

🧰 Maintenance

55 changes

• chore: Fix all rustdoc warnings and enforce no warning in CI (#4261) @AdamGS
• fix(deps): update dependency org.junit.jupiter:junit-jupiter to v5.13.4 (#4260) @renovate[bot]
• chore[vortex-expr]: folder docs (#4238) @joseph-isaacs
• fix(deps): update dependency ch.qos.logback:logback-classic to v1.5.18 (#4259) @renovate[bot]
• fix(deps): update slf4j monorepo to v2.0.17 (patch) (#4258) @renovate[bot]
• fix(deps): update dependency org.junit.jupiter:junit-jupiter to v5.10.5 (#4257) @renovate[bot]
• chore: Add docs to vortex-expr (#4253) @AdamGS
• chore(deps): update rust crate cc to v1.2.33 (#4256) @renovate[bot]
• chore[fastlanes]: remove FoR unsigned transmute (#4252) @joseph-isaacs
• chore(deps): update plugin com.gradleup.shadow to v9.0.2 (#4249) @renovate[bot]
• fix(deps): update protobuf monorepo to v4.32.0 (minor) (#4248) @renovate[bot]
• chore(deps): update rust crate arbitrary to v1.4.2 (#4247) @renovate[bot]
• chore(deps): update rust crate async-trait to v0.1.89 (#4245) @renovate[bot]
• chore[vortex-array]: facilitate third-party ToArrowKernel implementations (#4237) @asubiotto
• chore: Allow coveralls step to fail in CI without failing the job (#4240) @AdamGS
• chore: speed up the benchmark page loading (#4235) @a10y
• docs: fill in missing javadoc (#4233) @a10y
• chore: expand runend decompress benchmarks (#4228) @0ax1
• fix(deps): update dependency org.immutables:value to v2.11.3 (#4222) @renovate[bot]
• chore(deps): update rust crate rayon to v1.11.0 (#4223) @renovate[bot]
• chore(deps): update all patch updates (patch) (#4211) @renovate[bot]
• chore: sequential consistency in vortex-scan (#4217) @lwwmanning
• chore: add vortex-ffi example to coverage (#4210) @lwwmanning
• chore: improve test coverage in vortex-scan (#4206) @lwwmanning
• chore: Fix random-access series colours and delete dead code from benchmarking website (#4213) @robert3005
• chore: improve vortex-dtype test coverage (#4204) @lwwmanning
• chore(deps): update all patch updates (patch) (#4205) @renovate[bot]
• chore(deps): update rust crate clap to v4.5.44 (#4202) @renovate[bot]
• Improve BoolArray::from_indices (#4200) @viirya
• chore[bench]: enable tpcds on label and merge (#4196) @joseph-isaacs
• chore(deps): update actions/checkout action to v5 (#4195) @renovate[bot]
• chore[bench]: enable tpcds on label and merge (#4187) @joseph-isaacs
• Remove for whom argument from SegmentSource (#4190) @gatesn
• chore(deps): update rust crate uuid to v1.18.0 (#4192) @renovate[bot]
• chore(deps): update rust crate glob to v0.3.3 (#4189) @renovate[bot]
• chore(deps): lock file maintenance (#4186) @renovate[bot]
• chore(deps): update rust crate sysinfo to 0.37.0 (#4185) @renovate[bot]
• fix(deps): update all patch updates to v1.0.168 (patch) (#4184) @renovate[bot]
• fix(deps): update all patch updates to v1.0.166 (patch) (#4183) @renovate[bot]
• chore(deps): update plugin com.gradleup.shadow to v9.0.1 (#4181) @renovate[bot]
• chore: improve vortex-mask testing (#4178) @lwwmanning
• chore(deps): update rust crate rustversion to v1.0.22 (#4179) @renovate[bot]
• build: Use Create cc to link to libc++ (#4173) @ssmccoy
• chore[bench-website]: add back tpc-ds to query_bench (#4155) @joseph-isaacs
• chore: clippy allow unreachable function (#4167) @0ax1
• chore(deps): update rust crate cc to v1.2.32 (#4166) @renovate[bot]
• fix(deps): update rust crate cxx to v1.0.165 (#4164) @renovate[bot]
• chore(deps): update all patch updates (patch) (#4163) @renovate[bot]
• chore: Bump msrv to 1.89 (#4159) @AdamGS
• chore: add CastKernel implementations for remaining array encodings (#4131) @lwwmanning
• chore: improve vortex-scalar tests, reorganize module structure (#4145) @lwwmanning
• chore: check out ddb version in ext build based on env var (#4153) @0ax1
• chore(deps): update rust crate taffy to 0.9.0 (#4154) @renovate[bot]
• chore(deps): update plugin com.gradleup.shadow to v9 (#4150) @renovate[bot]
• fix(deps): update all patch updates to v1.0.164 (patch) (#4151) @renovate[bot]