Job queue iteration error handling

[adam-fowler]

Currently when we call next() on the JobQueue iterator it can throw an error. This error is propagated further up the call stack and causes the ServiceLifecycle to instigate graceful shutdown of the application. This could be seen as a little aggressive.

Examples of errors that can be thrown include

• Database connection failure
• Internal structure of data is not as expected. eg Postgres table not setup correctly

We could set it up so only the JobQueueHandler service is killed, but that could mean it goes unnoticed as the rest of the application is still running. Or we could ignore these errors and continue to try accessing job queue. This could mean hitting a database service repeatedly until it responds which is not very good.

We could improve the situation by defining some driver neutral errors that the job queue understands eg underlying storage connection error, structural error and have the different drivers attempt to transform their errors into something the job queue handler would recognise. eg connection errors could have an exponential backoff, structural errors cause the application to shutdown

[adam-fowler]

I have a PR #77 which retries on failed driver calls. Given Postgres already does this internally I am wondering if this code should be made the responsibility of the driver instead.