QUIC network implementation thrashes sessions

[raulk]

I did a quick review of the code for the QUIC network implementation. It seems to be creating a QUIC session and dropping it for every incoming packet. Is this intentional? If yes, why?

Of course thrashing sessions will penalise performance – and a 3x slowdown when compared to UDP is not even bad in that circumstance.

To run fair UDP vs QUIC comparisons, this aspect should be fixed.

(BTW – apologies in advance if I misread the code – I did a very quick pass)

Relevant code: https://github.com/ConsenSys/handel/blob/master/network/quic/net.go#L127

CC @marten-seemann

[bkolad]

@raulk

Of course thrashing sessions will penalise performance – and a 3x slowdown when compared to UDP is not even bad in that circumstance.

This and the default quic encryption. We run experiments on t2.nano instances which are one core machine. Taking this 2 factors into account, I think 3x slowdown is acceptable so I was fine with the raw QUIC performance in this particular case.

It seems to be creating a QUIC session and dropping it for every incoming packet. Is this intentional? If yes, why?

We want to scale to thousands of nodes (we managed to run experiments with 4000 handel nodes),
initially I discarded the possibility of managing thousands of open connections as every handel node contacts only O(log(N)) random peers on every round (so even if caching thousands of QUIC connections is possible it would be very artificial for handel case) we hope the 0-RTT feature handshake can help us here once implemented.

To run fair UDP vs QUIC comparisons, this aspect should be fixed.

I agree we reported 3x slowdown but I don't think this is an issue, the slowdown is justified.
The aim was to check what will happen if we switch from UDP to QUIC and QUIC was not good fit for handel (initially the reason why we wanted to do this exercise was to check the impact of 0-RTT handshake feature which is not available yet)

[marten-seemann]

By just using a single stream in a connection seems you're giving up on all the benefits of QUIC stream multiplexing. If you really just need a single stream (at a time), you might be better of using TCP/TLS 1.3.

Why are you using a bufio.Reader in https://github.com/ConsenSys/handel/blob/a51c5874b326f27df3899ff056a5ae44ac81198d/network/quic/net.go#L120? Calls to quic.Stream.Read should be cheap.

Furthermore, you don't need to drain anything in https://github.com/ConsenSys/handel/blob/a51c5874b326f27df3899ff056a5ae44ac81198d/network/quic/net.go#L122-L126
You can just close the session. The issue you linked is 1. a h2quic, not a quic issue and 2. was fixed.

[bkolad]

@marten-seemann, I think we will stick to UDP as we don't need transport reliability anyway.
Thanks for the hints, I will apply your suggestions.

[raulk]

@bkolad would be fantastic to apply @marten-seemann's suggestions and report back numbers, even if it's at a smaller scale (please don't deploy 500 AWS instances for this) ;-)

[bkolad]

@raulk I will give it a shot tmr.

[bkolad]

I run the stress tests one more time (with suggested changes):
500 t2.nano instances
UDP ~ 0.4s
QUIC with AcceptCookie:nil: ~ 1.6s
QUIC with AcceptCookie -> true 1.3s

I would like to stress one more time that above numbers does NOT imply that QUIC has poor performance. Contrary taking into account additional factors:

• lack of session caching in handel (which means handshake on every packet)
• all the additional quic features (authentication, encryption etc)
• very low end machines we use for tests (one CPU which is already busy with BLS verification)

i would say this is what I would expect and it actually looks good.

On yesterday's ETH2.0 call I recommended to investigate QUIC more as a protocol for all peer communication.
thanks @raulk