Job queue changes

Hummingbird.docc/Jobs/JobQueueDriver.md

@@ -17,20 +17,17 @@ Defines the requirements for job queue implementation.
 
 ### Lifecycle
 
-- ``onInit()``
+- ``waitUntilReady()``
 - ``stop()``
 - ``shutdownGracefully()``
 
 ### Jobs
 
+- ``registerJob(_:)``
 - ``push(_:options:)``
 - ``finished(jobID:)``
 - ``failed(jobID:error:)``
-
-### Metadata
-
-- ``getMetadata(_:)``
-- ``setMetadata(key:value:)``
+- ``retry(_:jobRequest:options:)``
 
 ### Implementations
 

Hummingbird.docc/Jobs/Jobs.md

@@ -14,18 +14,28 @@ A Job consists of a payload and an execute method to run the job. `Jobs` provide
 
 ### Jobs
 
+- ``JobName``
 - ``JobDefinition``
 - ``JobParameters``
 - ``JobExecutionContext``
 
 ### Queues
 
 - ``JobQueue``
+- ``JobQueueProcessor``
+- ``JobQueueProtocol``
 - ``JobQueueOptions``
+- ``JobQueueProcessorOptions``
 - ``JobQueueDriver``
 - ``MemoryQueue``
 - ``JobOptionsProtocol``
 
+### Optional features
+
+- ``CancellableJobQueue``
+- ``ResumableJobQueue``
+- ``JobMetadataDriver``
+
 ### Scheduler
 
 - ``JobSchedule``
@@ -37,7 +47,9 @@ A Job consists of a payload and an execute method to run the job. `Jobs` provide
 - ``MetricsJobMiddleware``
 - ``TracingJobMiddleware``
 - ``JobMiddlewareBuilder``
-- ``JobQueueContext``
+- ``JobPushQueueContext``
+- ``JobPopQueueContext``
+- ``JobCompletedQueueContext``
 
 ### Error
 
@@ -51,6 +63,13 @@ A Job consists of a payload and an execute method to run the job. `Jobs` provide
 - ``JobQueueResult``
 - ``JobRegistry``
 - ``JobRequest``
+- ``JobRetryOptions``
+
+### Retry
+
+- ``JobRetryStrategy``
+- ``ExponentialJitterJobRetryStrategy``
+- ``NoRetryJobRetryStrategy``
 
 ## See Also
 

Hummingbird.docc/Jobs/JobsGuide.md

@@ -12,17 +12,15 @@ A Job consists of a payload and an execute method to run the job. Swift Jobs pro
 
 ### Setting up a Job queue
 
-Before you can start adding or processing jobs you need to setup a Jobs queue to push jobs onto. Below we create a job queue stored in local memory that will process four jobs concurrently.
+Before you can start adding or processing jobs you need to setup a Jobs queue to push jobs onto. Below we create a job queue stored in local memory.
 
 ```swift
-let jobQueue = JobQueue(.memory, numWorkers: 4, logger: logger)
+let jobQueue = JobQueue(.memory, logger: logger)
 ```
 
 ### Creating a Job
 
-Before you can start running jobs you need to define a job. A job definition requires an identifier for the job, the job parameters and the function that runs the job. 
-
-We use a struct conforming to ``Jobs/JobParameters`` to define the job parameters and identifier.
+Creating a job requires an identifier, the parameters for the job and the function that runs the job. We use a struct conforming to ``Jobs/JobParameters`` to define the job parameters and identifier.
 
 ```swift
 struct SendEmailJobParameters: JobParameters {
@@ -42,7 +40,7 @@ jobQueue.registerJob(parameters: SendEmailJobParameters.self) { parameters, cont
 }
 ```
 
-Now your job is ready to create. Jobs can be queued up using the function `push` on `JobQueue`.
+Now your job is ready to create. Jobs can be queued up using the function ``Jobs/JobQueue/push(_:options:)`` from `JobQueue`.
 
 ```swift
 let job = SendEmailJobParameters(
@@ -53,13 +51,29 @@ let job = SendEmailJobParameters(
 jobQueue.push(job)
 ```
 
+Alternatively you can create a job using a ``Jobs/JobName``. This associates a type with a name, but that type can be used multiple times with different job names.
+
+```swift
+let printStringJob = JobName<String>("Print String")
+jobQueue.registerJob(printStringJob) { parameters, context in
+    print(parameters)
+}
+```
+
+You then queue your job for execution using ``Jobs/JobQueue/push(_:parameters:options:)``
+
+```swift
+jobQueue.push(printStringJob, parameters: "Testing,testing,1,2,3")
+```
+
 ### Processing Jobs
 
-When you create a `JobQueue` the `numWorkers` parameter indicates how many jobs you want serviced concurrently by the job queue. If you want to activate these workers you need to add the job queue to your `ServiceGroup`.
+To start processing jobs on your queue you need a ``Jobs/JobQueueProcessor``. You can create the job processor for a job queue by calling ``Jobs/JobQueue/processor(options:)``. The options passed in when creating your `JobQueueProcessor` includes the parameter `numWorkers` which indicates how many jobs you want to run concurrently. If you want to activate the `JobQueueProcessor` you can call the ``Jobs/JobQueueProcessor/run()`` method but it is preferable to use [Swift Service Lifecycle](https://github.com/swift-server/swift-service-lifecycle) to manage the running of the processor to ensure clean shutdown when your application is shutdown.
 
 ```swift
+let jobProcessor = jobQueue.processor(options: .init(numWorkers: 8))
 let serviceGroup = ServiceGroup(
-    services: [server, jobQueue],
+    services: [server, jobProcessor],
     configuration: .init(gracefulShutdownSignals: [.sigterm, .sigint]),
     logger: logger
 )
@@ -68,7 +82,7 @@ try await serviceGroup.run()
 Or it can be added to the array of services that `Application` manages
 ```swift
 let app = Application(...)
-app.addServices(jobQueue)
+app.addServices(jobProcessor)
 ```
 If you want to process jobs on a separate server you will need to use a job queue driver that saves to some external storage eg ``JobsRedis/RedisJobQueue`` or ``JobsPostgres/PostgresJobQueue``.
 
@@ -118,13 +132,6 @@ Setting it to `.all` will schedule a job for every trigger point it missed eg if
 jobSchedule.addJob(TestJobParameters(), schedule: .hourly(minute: 30), accuracy: .all)
 ```
 
-## Topics
-
-### Reference
-
-- ``JobsPostgres``
-- ``JobsRedis``
-
 ## See Also
 
 - ``Jobs/JobParameters``

Hummingbird.docc/JobsPostgres/JobsPostgres.md

@@ -35,7 +35,6 @@ let jobQueue = JobQueue(
         ),
         logger: logger
     ), 
-    numWorkers: 4, 
     logger: logger
 )
 ```
@@ -107,6 +106,57 @@ try await jobQueue.pause(jobID: jobID)
 try await jobQueue.resume(jobID: jobID)
 ```
 
+### Job retention
+
+The queue has options to retain jobs once it has finished with them depending on status. By default the queue will retain failed jobs and drop cancelled or completed jobs, but these decisions are configurable.
+
+```swift
+let jobQueue = JobQueue(
+    .postgres(
+        client: postgresClient,
+        migrations: postgresMigrations,
+        configuration: .init(
+            queueName: "MyJobQueue", 
+            retentionPolicy: .init(
+                completedJobs: .retain, 
+                failedJobs: .retain, 
+                cancelledJobs: .doNotRetain
+            )
+        )
+    ),
+    logger: logger
+)
+```
+
+### Job queue cleanup
+
+If you do opt to retain jobs after processing you will probably eventually want to clean them up. The Postgres queue provides a method `cleanup` which allows you to remove or attempt to re-run jobs based on what state they are in. You should be careful not to do anything to pending or processing jobs while the job queue is being processed as it might confuse the job processor.
+
+```swift
+jobQueue.queue.cleanup(
+    pendingJobs: .doNothing,
+    processingJobs: .doNothing,
+    completedJobs: .remove(maxAge: .seconds(7*24*60*60)),
+    failedJobs: .rerun,
+    cancelledJobs: .remove, 
+)
+```
+
+#### Scheduling cleanup
+
+Given this is a job you will probably want to do regularly the queue also provides a job you can use in conjunction with the `JobScheduler` that will do the cleanup for you. 
+
+```swift
+var jobSchedule = JobSchedule()
+jobSchedule.addJob(
+    jobQueue.queue.cleanupJob,
+    parameters: .init(completedJobs: .remove, failedJobs: .rerun, cancelledJobs: .remove),
+    schedule: .weekly(day: .sunday)
+)
+```
+
+You can find out more about the Job scheduler in the Jobs guide <doc:JobsGuide#Job-Scheduler>
+
 ## Topics
 
 ### Job Queue

Hummingbird.docc/JobsRedis/JobsRedis.md

@@ -31,7 +31,6 @@ let jobQueue = JobQueue(
             pollTime: .milliseconds(50)
         )
     ),
-    numWorkers: 10,
     logger: logger
 )
 let serviceGroup = ServiceGroup(
@@ -104,6 +103,56 @@ try await jobQueue.pause(jobID: jobID)
 try await jobQueue.resume(jobID: jobID)
 ```
 
+### Job retention
+
+The queue has options to retain jobs once it has finished with them depending on status. By default the queue will retain failed jobs and drop cancelled or completed jobs, but these decisions are configurable.
+
+```swift
+let jobQueue = JobQueue(
+    .redis(
+        redisService.pool, 
+        configuration: .init(
+            queueKey: "MyJobQueue", 
+            retentionPolicy: .init(
+                completedJobs: .retain, 
+                failedJobs: .retain, 
+                cancelledJobs: .doNotRetain
+            )
+        )
+    ),
+    logger: logger
+)
+```
+
+### Job queue cleanup
+
+If you do opt to retain jobs after processing you will probably eventually want to clean them up. The Redis queue provides a method `cleanup` which allows you to remove or attempt to re-run jobs based on what state they are in. You should be careful not to do anything to pending or processing jobs while the job queue is being processed as it might confuse the job processor.
+
+```swift
+jobQueue.queue.cleanup(
+    pendingJobs: .doNothing,
+    processingJobs: .doNothing,
+    completedJobs: .remove(maxAge: .seconds(7*24*60*60)),
+    failedJobs: .rerun,
+    cancelledJobs: .remove, 
+)
+```
+
+#### Scheduling cleanup
+
+Given this is a job you will probably want to do regularly the queue also provides a job you can use in conjunction with the `JobScheduler` that will do the cleanup for you. 
+
+```swift
+var jobSchedule = JobSchedule()
+jobSchedule.addJob(
+    jobQueue.queue.cleanupJob,
+    parameters: .init(completedJobs: .remove, failedJobs: .rerun, cancelledJobs: .remove),
+    schedule: .weekly(day: .sunday)
+)
+```
+
+You can find out more about the Job scheduler in the Jobs guide <doc:JobsGuide#Job-Scheduler>
+
 ## Topics
 
 ### Job Queue

Package.swift

@@ -1,11 +1,11 @@
-// swift-tools-version: 5.10
+// swift-tools-version: 6.0
 // The swift-tools-version declares the minimum version of Swift required to build this package.
 
 import PackageDescription
 
 let package = Package(
     name: "hummingbird-docs",
-    platforms: [.macOS(.v14)],
+    platforms: [.macOS(.v15)],
     products: [
         // Products define the executables and libraries a package produces, and make them visible to other packages.
         .library(
@@ -18,13 +18,13 @@ let package = Package(
         .package(url: "https://github.com/hummingbird-project/hummingbird-auth.git", from: "2.0.0"),
         .package(url: "https://github.com/hummingbird-project/hummingbird-compression.git", from: "2.0.0"),
         .package(url: "https://github.com/hummingbird-project/hummingbird-fluent.git", from: "2.0.0"),
-        .package(url: "https://github.com/hummingbird-project/swift-jobs.git", from: "1.0.0-beta.8"),
-        .package(url: "https://github.com/hummingbird-project/swift-jobs-postgres.git", from: "1.0.0-beta.2"),
-        .package(url: "https://github.com/hummingbird-project/swift-jobs-redis.git", revision: "bdb42b1ca7499818bb3798f1005880ff77da5ab2"),
+        .package(url: "https://github.com/hummingbird-project/swift-jobs.git", from: "1.0.0-rc"),
+        .package(url: "https://github.com/hummingbird-project/swift-jobs-postgres.git", from: "1.0.0-rc"),
+        .package(url: "https://github.com/hummingbird-project/swift-jobs-redis.git", from: "1.0.0-rc"),
         .package(url: "https://github.com/hummingbird-project/hummingbird-lambda.git", from: "2.0.0-rc.3"),
         .package(url: "https://github.com/hummingbird-project/swift-mustache.git", from: "2.0.0"),
-        .package(url: "https://github.com/hummingbird-project/hummingbird-postgres.git", branch: "0.8.0"),
-        .package(url: "https://github.com/hummingbird-project/postgres-migrations.git", from: "0.1.0"),
+        .package(url: "https://github.com/hummingbird-project/hummingbird-postgres.git", from: "1.0.0-rc"),
+        .package(url: "https://github.com/hummingbird-project/postgres-migrations.git", from: "1.0.0-rc"),
         .package(url: "https://github.com/hummingbird-project/hummingbird-redis.git", from: "2.0.0"),
         .package(url: "https://github.com/hummingbird-project/hummingbird-websocket.git", from: "2.2.0"),
         .package(url: "https://github.com/hummingbird-project/swift-websocket.git", from: "1.2.0"),