Construct from already packed data

[RagnarGrootKoerkamp]

It would be great if for alphabet size 2 and 4, it would be possible to construct a TextWithRankSupport directly from already-packed &[usize] data.
Currently I'm manually converting a bitmask to a Vec<u8> of 0 and 1, and then passing that, which is wasteful both for time and memory usage.

[feldroop]

I already use something similar, for alphabets of size 16:

genedex/src/text_with_rank_support/condensed.rs

Line 59 in 1c275aa

fn construct_from_maybe_slice_compressed_text<S: SliceCompression>(

I could probably generalize it without too much effort. However, I was actually planning on removing the whole TextWithRankSupport part from the public API, so I can be more messy and experimental with it. Now it seems like it might be useful after all, so I'd be interested in what you're using it for 😄

Edit: It might actually be a bit more complicated, if I want to handle input slices with different integer widths, due to be endianness issues.

[RagnarGrootKoerkamp]

I dont really care about he exact API, but its important that the input is exactly 1 or 2-bit packed, so that there's no memory overhead in constructing the data structure. Ideally you could even take a Vec and then construct everything without taking more than the final amount of memory.
(But i dont even do this myself yet so not super critical.) But long term it'd be nice to construct things on say 40GB of data with 64GB of ram.

[feldroop]

Okay, I see. I didn't optimize the memory usage here yet, because I use the normal libsais for suffix array construction and therefore, the memory bottleneck is the suffix array. But I already saw you use a special library for that. I will add it to the roadmap, but it will probably not be the top priority 👍

[RagnarGrootKoerkamp]

Ah that's a fair point 😅
I've just been benchmarking on as large random inputs as I can get to work, but of course that skips the construction process.

Still for pre-built indices it's nice if they can be used on smaller machines.