R/W ops can get out of sync

[szarnyasg]

Moving a discussion on Slack here:

• read ops & write ops are sent to different threads, read threads are a pool, write threads are long lived threads that are pinned to op streams
• so reads and/or writes can outpace one another if the compression ratio is set too ambitiously…
• passing all ops via a pool is a no-go, because it’s too slow
• but what might make sense is to use those long lived threads for all ops
• and assign reads to those long lived threads before even starting, so there is no need to feed ops to threads via a queue. rather, each thread would get an iterator that it simply exhausts

[jackwaudby]

As mentioned in Slack, generating the complete workload upfront and distributing these across long-lived threads feels a good idea and would simplify the code and runtime complexity.

My only concern is how we would generate the input parameters for the short reads which are currently fed by the results of complex reads at runtime.

[jackwaudby]

Relevant section from the docs:
For each complex read instance, a sequence of short reads is planned. There are two types of short read sequences: Person centric and Message centric. Depending on the type of the complex read, one of them is chosen. Each sequence consists of a set of short reads which are issued in a row. The issue time assigned to each short read in the sequence is determined at run time, and is based on the completion time of the complex read it depends on. The substitution parameters for short reads are taken from the results of previously executed complex reads and short reads. Once a short read sequence is issued (and provided that sufficient substitution parameters exist), there is a probability that another short read sequence is issued. This probability decreases for each new sequence issued.

[jackwaudby]

AFAIK the idea behind the pool of complex read results being used to fed short reads is to simulate the behavior of running some complex queries then exploring the area around/looking up the results.