Publisher bottleneck

[lordent]

django-outbox-pattern/django_outbox_pattern/management/commands/publish.py

Line 47 in ea419ef

published = self.published_class.objects.filter(

Such an implementation looks like a bottleneck

published = self.published_class.objects.filter(
    status=StatusChoice.SCHEDULE, expires_at__gte=timezone.now()
)
for message in published:
    try:
        attempts = self.producer.send(message)
    except ExceededSendAttemptsException as exc:
        logger.exception(exc)
        message.retry = exc.attempts
        message.status = StatusChoice.FAILED
        message.expires_at = timezone.now() + timedelta(15)
        self.stdout.write(f"Message no published with body:\n{message.body}")
    else:
        message.retry = attempts
        message.status = StatusChoice.SUCCEEDED
        self.stdout.write(f"Message published with body:\n{message.body}")
    finally:
        message.save()

because new messages can arrive faster than once a second

def _waiting():
    sleep(1)

while the solution does not allow itself to be scaled in any way; you cannot simply launch another Publisher, otherwise the message will be sent twice

[ricardochaves]

Hi @lordent ,
We still don't have a fix date. Any collaboration is welcome.