Persist events that are skipped due to disabled destinations

[laurenzlong]

Is your feature request related to a problem? Please describe.
Currently events are basically just gone if a destination is disabled (I see that there was a discussion in #152), this is not ideal for a few reasons:

• impossible to replay events once the destination is re-enabled
• very hard to debug what happened when customer complains about missing events, I was able to eventually find a log line that said "skipping disabled destination" after hours of digging, but the event doesn't show up in the portal or in the API

Describe the solution you'd like
When a destination is disabled, the event should still be:

• Persisted in the events table but should have "Failed" status
• Be viewable in the portal & via API
• Be retryable (via portal or API) once the destination is re-enabled. I don't think it should auto retry since there might be quite a pile up, but manual retry should be supported.

Describe alternatives you've considered
Current state

Additional context
N/A

[alexbouchardd]

@laurenzlong I'd like to understand this request a bit better. Our intention for "disabling" a destination was to stop occuring cost associated with that destination, usually because it's not working. That's how pretty much every webhook provider works, once you get disabled, it's your problem.

Do you have an alternative use case for "disabling" a destination or do you have a different opinion on the cost tradeoff here?

In the case where it's the later, then I'd advocate for disabling the auto-disable for destinations!

[laurenzlong]

Hey @alexbouchardd we've configured auto-disabling for a destination that repeated has failures (usually due to credential issues), once the customer is aware of this and fixes their credentials, we'd like to be able to replay the messages that they've missed. To be clear, I do want the publish to stop when a destination has repeated failures, but I do still want traceability and the ability to replay later. Alternatively, the publish event API endpoint can return an error when a destination is disabled, currently it accepts the request and then silently drops it with no traces of the event having ever existed. This is rather confusing behavior.

[alexbouchardd]

On the second point, 100% we need to improve of the feedback on the publishing.

For persisting events it's definitely something worth considering, it opens up a whole can of worms of replayability however given right now the retry is really more ment as a one-off ad-hoc thing rather then large bulk retry. I'm not entirely sure if that should be in scope for Outpost but it's worth debatting. I'll leave this issue open and we'll come back to it when more feedback comes in!

[laurenzlong]

Hey @alexbouchardd yes I agree that persisting the events is a larger decision & has implications. In the meantime, can the publish event API return an error when the destination is disabled?

[alexbouchardd]

Yes scoping this. cc @alexluong

[obazoud]

Being able to replay events once a destination is re-enabled is definitely a feature I’d like to have, rather than losing those events.

Events should be always persisted (with a retention policy), then it will be easy to replay events as long as the destination is enabled. Currently, retries are one by one. We should introduce a batch API that allows retrying 10 events at a time to limit the load caused by a large-scale replay (and maybe add a rate limit on this api).

In the meantime, can the publish event API return an error when the destination is disabled?

How would this work when events are published via SQS ?

[alexbouchardd]

@laurenzlong Does #669 look like a viable solution here? The idea is that if destination_ids is [] then it implies the event hasn't been persisted. It's not an "error" per say but it does give you the necessary information.