supported HTTP/2 by chobits · Pull Request #278 · chobits/ngx_http_proxy_connect_module

chobits · 2023-07-08T05:58:41Z

try to fix: #25

RFC: https://httpwg.org/specs/rfc7540.html#CONNECT

WIP: not ready now

…s to be changed RFC: https://httpwg.org/specs/rfc7540.html#CONNECT The location where the code needs to be changed is as follows: 1. client read before data proxying: check and process HTTP/2 header ":authority" 2. client write before data proxying: Once backend connection is successfully established, the proxy sends a HEADERS frame containing a 2xx series status code to the client, as defined in 3. client read/write during data proxying: read/write DATA frames from/to client

…connect_module into for_pull_request

chobits · 2023-07-08T06:09:09Z

phrase from RFC as following

Client sends HEADERS frame (:method = CONNECT, :authority: <host>[:<port>]) to the proxy server.
The proxy server establishes a connection with backend server, then it sends HEADERS frame with 2xx status.
The proxy server reads data from backend/client, then transmits the received data to client/backend.
- if proxy server reads data from backend, then assemble it into DATA frame and sent it to client.
- if proxy server reads DATA frame from client, then send the payload of DATA frame to backend .

…tream

chobits · 2023-07-16T12:11:20Z

HINT: How upstream reads request body from http/2 DATA frame:

If we dont wanna modify the http/2 logic in the nginx core, we need to use the logic of upstream reading request body.

ngx_http_v2_state_read_data
 -> ngx_http_v2_process_request_body(r, pos) // pos --> h2c head buf
   ngx_memcopy pos -> r->request_body->buf->last
                 data saved in r->request->body->buf->[pos, last]
   -> ngx_http_v2_filter_request_body(r)
         cl->b <= r->request->body->buf
         r->request_body->busy/free <-- ngx_chain_update_chains --- cl->b
   -> rb->post_handler(r);
       -> ngx_http_upstream_init
          -> ngx_http_upstream_init_request

              if (r->request_body) {
                  u->request_bufs = r->request_body->bufs;
              }

            -> ngx_http_upstream_connect
             -> ngx_http_upstream_send_request
              -> ngx_http_upstream_send_request_body(r, u, do_write);
                out <- u->request_bufs
                r->request_body->bufs <- NULL
                for ( ;; ) {
                     if (do_write) {
                         rc = ngx_output_chain(&u->output, out);    // send to upstream fd
                     }
                     ...
                }

…ream via http/2 protocol

…ding 200 OK ngx_http_v2_send_output_queue() will do this for us.

chobits · 2023-07-25T12:42:24Z

source notes for http/2 request body reading logic( DATA frame receiving logic):


------
http/2 connection handler：
    rev->handler = ngx_http_v2_read_handler;
                      -> ngx_http_v2_state_head
                        -> ngx_http_v2_state_data
    c->write->handler = ngx_http_v2_write_handler;

------
ngx_http_v2_state_read_data
 -> ngx_http_v2_process_request_body(r, pos) // pos --> h2c head buf
   -> ngx_memcopy(pos -> r->request_body->buf->last)
      // data saved in r->request->body->buf->[pos, last]
   -> ngx_http_v2_filter_request_body(r)
         cl->b <= r->request->body->buf
         r->request_body->busy/free <-- ngx_chain_update_chains --- cl->b
   -> rb->post_handler(r);              // TODO: is it ngx_http_init_upstream always?
       -> ngx_http_upstream_init
          -> ngx_http_upstream_init_request

              if (r->request_body) {
                  u->request_bufs = r->request_body->bufs;
              }

            -> ngx_http_upstream_connect
             -> ngx_http_upstream_send_request
              -> ngx_http_upstream_send_request_body(r, u, do_write);
                out <- u->request_bufs
                r->request_body->bufs <- NULL
                for ( ;; ) {
                    if (do_write) {
                        rc = ngx_output_chain(&u->output, out);    // send to upstream fd
                    }

                    if (r->reading_body) {
                        /* read client request body */
                        rc = ngx_http_read_unbuffered_request_body(r);
                              -> ngx_http_v2_read_unbuffered_request_body(r);
                                -> ngx_http_v2_process_request_body
                                  -> ngx_http_v2_filter_request_body
                    }
                }

chobits · 2023-07-31T07:19:28Z

If proxy_connect does not connect to backend server and DATA frame from client has already been sent to nginx, the logic backtrace is as following:

header frame + data frame ==> nginx
handle header frame , connecting to backend but waiting establishment of this connection
handle DATA frame ---. following bt log

(gdb)
1190        return ngx_http_v2_state_complete(h2c, pos, end);
(gdb) bt
#0  ngx_http_v2_state_read_data (h2c=h2c@entry=0x562d049be0d0, pos=0x562d049d1920 "hello HTTP/1.1\r\nHost: test.com\r\n\r\n", pos@entry=0x562d049d18f9 "GET /hello HTTP/1.1\r\nHost: test.com\r\n\r\nhello HTTP/1.1\r\nHost: test.com\r\n\r\n", end=end@entry=0x562d049d1920 "hello HTTP/1.1\r\nHost: test.com\r\n\r\n") at src/http/v2/ngx_http_v2.c:1190
#1  0x0000562d038190c7 in ngx_http_v2_state_data (h2c=0x562d049be0d0, pos=0x562d049d18f9 "GET /hello HTTP/1.1\r\nHost: test.com\r\n\r\nhello HTTP/1.1\r\nHost: test.com\r\n\r\n", end=0x562d049d1920 "hello HTTP/1.1\r\nHost: test.com\r\n\r\n") at src/http/v2/ngx_http_v2.c:1087
#2  0x0000562d038175c6 in ngx_http_v2_state_head (h2c=0x562d049be0d0, pos=0x562d049d18f9 "GET /hello HTTP/1.1\r\nHost: test.com\r\n\r\nhello HTTP/1.1\r\nHost: test.com\r\n\r\n", end=0x562d049d1920 "hello HTTP/1.1\r\nHost: test.com\r\n\r\n") at src/http/v2/ngx_http_v2.c:947
#3  0x0000562d038196cb in ngx_http_v2_read_handler (rev=0x7f98954e4250) at src/http/v2/ngx_http_v2.c:435
#4  0x0000562d037c2221 in ngx_epoll_process_events (cycle=0x562d048a4c80, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:901
#5  0x0000562d037b654e in ngx_process_events_and_timers (cycle=cycle@entry=0x562d048a4c80) at src/event/ngx_event.c:248
#6  0x0000562d037bfd8f in ngx_worker_process_cycle (cycle=0x562d048a4c80, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:721
#7  0x0000562d037be1c3 in ngx_spawn_process (cycle=cycle@entry=0x562d048a4c80, proc=proc@entry=0x562d037bfc52 <ngx_worker_process_cycle>, data=data@entry=0x0, name=name@entry=0x562d038c96ea "worker process", respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:199
#8  0x0000562d037bf4c1 in ngx_start_worker_processes (cycle=cycle@entry=0x562d048a4c80, n=1, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:344
#9  0x0000562d037c06b0 in ngx_master_process_cycle (cycle=0x562d048a4c80) at src/os/unix/ngx_process_cycle.c:130
#10 0x0000562d037942ed in main (argc=1, argv=<optimized out>) at src/core/nginx.c:383

debug log

2023/07/31 15:14:22 [debug] 522#0: *16 event timer add: 17: 10000:22052976
2023/07/31 15:14:22 [debug] 522#0: *16 http finalize request: -4, "?" a:1, c:2
2023/07/31 15:14:22 [debug] 522#0: *16 http request count:2 blk:0
2023/07/31 15:14:22 [debug] 522#0: *16 http2 frame complete pos:0000562D049D191B end:0000562D049D191B
2023/07/31 15:14:22 [debug] 522#0: *16 http2 read handler
2023/07/31 15:14:22 [debug] 522#0: *16 SSL_read: 48
2023/07/31 15:14:22 [debug] 522#0: *16 SSL_read: -1
2023/07/31 15:14:22 [debug] 522#0: *16 SSL_get_error: 2
2023/07/31 15:14:22 [debug] 522#0: *16 http2 frame type:0 f:1 l:39 sid:1
2023/07/31 15:14:22 [debug] 522#0: *16 http2 DATA frame
2023/07/31 15:14:22 [debug] 522#0: *16 malloc: 0000562D04A15910:65536
2023/07/31 15:14:22 [debug] 522#0: *16 http2 frame complete pos:0000562D049D1920 end:0000562D049D1920

chobits · 2023-07-31T08:59:22Z

for http2 & upstream:

read upstream and send data to downstream: ngx_http_upstream_process_non_buffered_request
read downstream and send data to upstream: ngx_http_upstream_read_request_handler->ngx_http_upstream_send_request

chobits · 2023-08-02T10:27:26Z

Unfortunately, there is no low-level IO operation API of c->recv()/send() provided in nginx's HTTPv2 implementation. So if we want to support http/2 CONNECT tunnel, we need to use request_body API for reading DATA frames from http/2 request and output_filter for sending DATA frames through the http/2 CONNECT tunnel.

This makes the implementation of supporting HTTP/2 in this module more complex.

doublex · 2023-12-19T13:05:13Z

Great feature!!
Thanks for your efforts!

doublex · 2023-12-22T12:28:29Z

Question:
Is the hard part downstream or upstream?
Would it be easier if only the downstream-connection is http2 (so eavesdroppers think it is http2)?
E.g.:
Client -> http2:connect -> nginx:chobits -> http11 -> website

chobits · 2023-12-25T02:36:25Z

Question: Is the hard part downstream or upstream? Would it be easier if only the downstream-connection is http2 (so eavesdroppers think it is http2)? E.g.: Client -> http2:connect -> nginx:chobits -> http11 -> website

This pr is under development, so some part of the logic is not completed. so it could not support the HTTP2 now.

chobits added 3 commits July 8, 2023 12:01

supported http/2: return 400 while failing to parse connect host

672721a

Merge branch 'for_pull_request' of github.com:chobits/ngx_http_proxy_…

c58c08a

…connect_module into for_pull_request

chobits changed the title ~~supported HTTP/2~~ [WIP] supported HTTP/2 Jul 8, 2023

chobits added 5 commits July 14, 2023 20:55

supported http/2 (part3): process connect host and port from HTTP/2 s…

9a7255d

…tream

supported http/2 (part3.1): added debug log and testing for port

4fc823e

supported http/2 (part4): added template for test case

d6bf313

supported http/2 (part5): sent "200" status for tunnel establishment

b71ebfa

supported http/2 (part5.1): completed logic of sending status 200 OK

2ed77da

chobits added 3 commits July 21, 2023 19:48

supported http/2 (part5.2): completed logic of sending data to downst…

bdae33b

…ream via http/2 protocol

supported http/2 (part5.3): removed logic of handling event after sen…

0ae1f3f

…ding 200 OK ngx_http_v2_send_output_queue() will do this for us.

supported http/2 (part5.4): added request body reading logic

357a2d4

supported http/2 (part5.5): refactor IO operations to buf/chain API

37c42ba

chobits marked this pull request as draft September 4, 2023 03:18

chobits changed the title ~~[WIP] supported HTTP/2~~ supported HTTP/2 Sep 4, 2023

chobits mentioned this pull request Dec 11, 2023

quic protocol #292

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

supported HTTP/2#278

supported HTTP/2#278
chobits wants to merge 12 commits intomasterfrom
for_pull_request

chobits commented Jul 8, 2023 •

edited

Loading

Uh oh!

chobits commented Jul 8, 2023 •

edited

Loading

Uh oh!

chobits commented Jul 16, 2023 •

edited

Loading

Uh oh!

chobits commented Jul 25, 2023

Uh oh!

chobits commented Jul 31, 2023 •

edited

Loading

Uh oh!

chobits commented Jul 31, 2023 •

edited

Loading

Uh oh!

chobits commented Aug 2, 2023 •

edited

Loading

Uh oh!

doublex commented Dec 19, 2023

Uh oh!

doublex commented Dec 22, 2023

Uh oh!

chobits commented Dec 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

chobits commented Jul 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chobits commented Jul 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chobits commented Jul 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chobits commented Jul 25, 2023

Uh oh!

chobits commented Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chobits commented Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chobits commented Aug 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doublex commented Dec 19, 2023

Uh oh!

doublex commented Dec 22, 2023

Uh oh!

chobits commented Dec 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chobits commented Jul 8, 2023 •

edited

Loading

chobits commented Jul 8, 2023 •

edited

Loading

chobits commented Jul 16, 2023 •

edited

Loading

chobits commented Jul 31, 2023 •

edited

Loading

chobits commented Jul 31, 2023 •

edited

Loading

chobits commented Aug 2, 2023 •

edited

Loading