Exception in .CustomAction results in losing the message

[aherba-krk]

When using wolverine with rabbitmq, throwing an exception within .CustomAction, like this:

opts.Policies.OnAnyException().CustomAction(async (runtime, lifecycle, exception) =>
        {
            throw new Exception("Exception in custom action");
        }, "customAction");

results in wolverine losing the message, as if it was processed successfully.

I think it's due to this catch:
https://github.com/JasperFx/wolverine/blob/main/src/Wolverine/Runtime/HandlerPipeline.cs#L91

[jeremydmiller]

@aherba-krk What do you mean the message is lost? Just that it's discarded? What are you actually trying to do with your custom action?

And this is Wolverine working as intended, so I'm moving this off to a discussion

[aherba-krk]

What i mean by 'lost' is that I got to this point because handler had failed, so obviously message had not been processed successfully, but then, the thing that was supposed to handle that failure also fails. And yet, rabbitmq queue is being sent an ACK, like nothing happened, so the message is not there anymore. It's not on its original queue, it's not on DLQ. So like you said - message is discarded. To my mind, we're left with an exception, and no way of telling what that message contained.

As for what i am actually doing with my custom action, I'm adding headers with some exception data, and reroute the message to the DLQ. So far it's been working for me, but I've been wondering what would happen if i get some exception in this custom action. I was quite surprised to find out that message is just being discarded, which I really cannot allow to happen in my system.

[jeremydmiller]

We never really considered your use case at all. Why are you using custom DLQ mechanisms? Can we instead just enhance the built in Rabbit MQ mechanism in Wolverine which does have some level of retry resiliency instead of relying on bespoke code?

[jeremydmiller]

Or have your custom action requeue the message on exceptions. I think you're going to have to apply some try/catch logic yourself.

[jeremydmiller]

Also see this: #401.

Can we play that story in core Wolverine so you don't have to be using the custom action at all?