Configurable poll frequency

[sat]

Current default behaviour is to stay within 4 req/s second per shard. Kinesis limit per shard is 5 req/s.

https://github.com/pipelinedb/pipeline_kinesis/blob/master/kinesis_consumer.cpp#L417

If another process is reading from the stream (eg. lambda), pipeline_kinesis will exhaust the capacity of the shard.

A configurable poll frequency could enable this use case:

eg.

pipeline_kinesis.consume_begin ( endpoint text, stream text, relation text, format text := ‘text’, delimiter text := E’\t’, quote text := NULL, escape text := NULL, batchsize int := 1000, parallelism int := 1, start_offset int := NULL, poll_freq real := 0.25)