How to Properly Handle Connection Closures in TCP Proxies
There are fewer scenarios in which just shutting down a TCP connection to read and write and using a simplex connection is a good idea than shutting down a TCP connection directly, but generic TCP proxies need to take this part of the scenario into account as well.
contexts
While looking at old code today, I found a rough implementation of a core function for TCP proxies that received theEOF After that, you can just brute-force close both TCP connections.
func ConnCat(uConn, rConn ) {
wg := {}
(2)
go func() {
defer ()
(uConn, rConn)
()
()
}()
go func() {
defer ()
(rConn, uConn)
()
()
}()
()
}
The general scenario is not perceived as a problem, but as a proxy, it should only transmit client/server behavior, and redundant actions should not occur, such as the client closing the write, the proxy only needs to pass the close to the server.
Connection closed
invocationsclose Closing a connection is common practice, and there is a relatedshutdown System Call.shutdown together withclose Compared to the ability to more finely control the reading and writing of connections, but not responsible for thefd resources are released, in other words, whether or not the call to theshutdown， close It all ends up needing to be called.
insofar asshutdown Description of the second parameter
- SHUT_RD The connection closes the read and still allows the write to continue.
- SHUT_WR The connection closes for writing and can still continue to read; and it sends aFIN Package.
- SHUT_RDWR The connection reads and writes are closed; and aFIN Package.
For the higher-level application, you only need to focus on the result of the read and receive theFIN(a.k.a.EOF) Although it is not possible to determine whether the opposite end is turned off for reads and writes or only writes, subsequent processing is not affected.
Process subsequent logic based on read data
- Determine if the read data is as expected to close the connection or write the data.
- Write data to the connection, if it fails, just close the connection, if it doesn't fail, the current connection is in simplex mode.
- Close connection
Example of connection closure read/write in Go
The test code demonstrates that two TCP connections are closed for reading (writing) and then for writing (reading) again.
func TestTCPClose(t *) {
lis, err := ("tcp", &{IP: ("127.0.0.1"), Port: 12345})
if err != nil {
(err)
}
var (
conn0 *
conn1 *
acceptErr error
)
acceptDoneCh := make(chan struct{})
go func() {
conn0, acceptErr = ()
close(acceptDoneCh)
}()
conn1, err = ("tcp", nil, ().(*))
if err != nil {
(err)
}
<-acceptDoneCh
if acceptErr != nil {
(acceptErr)
}
wg := {}
(2)
go func() {
([]byte("hello"))
( * 1)
()
b := make([]byte, 1024)
(b)
()
}()
go func() {
b := make([]byte, 1024)
(b)
()
( * 2)
([]byte("test"))
()
}()
()
()
()
}
A tcpdump packet capture also shows that CloseWrite sends aFIN contract (to or for)
17:21:09.877056 IP 127.0.0.1.44158 > 127.0.0.1.12345: Flags [S], seq 4257116181, win 65495, options [mss 65495,sackOK,TS val 3165750919 ecr 0,nop,wscale 7], length 0
17:21:09.877069 IP 127.0.0.1.12345 > 127.0.0.1.44158: Flags [S.], seq 188514168, ack 4257116182, win 65483, options [mss 65495,sackOK,TS val 3165750919 ecr 3165750919,nop,wscale 7], length 0
17:21:09.877081 IP 127.0.0.1.44158 > 127.0.0.1.12345: Flags [.], ack 1, win 512, options [nop,nop,TS val 3165750919 ecr 3165750919], length 0
17:21:09.877211 IP 127.0.0.1.44158 > 127.0.0.1.12345: Flags [P.], seq 1:6, ack 1, win 512, options [nop,nop,TS val 3165750920 ecr 3165750919], length 5
17:21:09.877219 IP 127.0.0.1.12345 > 127.0.0.1.44158: Flags [.], ack 6, win 512, options [nop,nop,TS val 3165750920 ecr 3165750920], length 0
17:21:10.878149 IP 127.0.0.1.44158 > 127.0.0.1.12345: Flags [F.], seq 6, ack 1, win 512, options [nop,nop,TS val 3165751920 ecr 3165750920], length 0
17:21:10.920263 IP 127.0.0.1.12345 > 127.0.0.1.44158: Flags [.], ack 7, win 512, options [nop,nop,TS val 3165751963 ecr 3165751920], length 0
17:21:11.877430 IP 127.0.0.1.12345 > 127.0.0.1.44158: Flags [P.], seq 1:5, ack 7, win 512, options [nop,nop,TS val 3165752920 ecr 3165751920], length 4
17:21:11.877460 IP 127.0.0.1.44158 > 127.0.0.1.12345: Flags [.], ack 5, win 512, options [nop,nop,TS val 3165752920 ecr 3165752920], length 0
17:21:11.882928 IP 127.0.0.1.12345 > 127.0.0.1.44158: Flags [F.], seq 5, ack 7, win 512, options [nop,nop,TS val 3165752925 ecr 3165752920], length 0
17:21:11.882957 IP 127.0.0.1.44158 > 127.0.0.1.12345: Flags [.], ack 6, win 512, options [nop,nop,TS val 3165752925 ecr 3165752925], length 0
analyze
A diagram of a fully established TCP connection is shown below, with each line representing a simplex connection.
┌────────┐ R W ┌────────┐ R W ┌────────┐
│ │ ◄─────────────── │ │ ◄─────────────── │ │
│ Client │ UConn │ Proxy │ RConn │ Server │
│ | ───────────────► │ │ ───────────────► │ │
└────────┘ W R └────────┘ W R └────────┘
For the Proxy, packets from one connection need to be passed to another connection, and packets received are forwarded and read to theEOF then close the other connection for writes (you can also close this connection for reads with one more system call)
The entire shutdown process is initiated by the Client (and the Server as well) and is a drum roll:
- Client closes the write side of the UConn connection (or the read side if subsequent data writes report an error).
- Proxy receives UConn'sEOFThe write side of the RConn connection is closed.
- Server receives RConn'sEOFThe write side of the RConn connection is closed.
- Proxy receives RConn'sEOFThe write side of the UConn connection is closed.
- All simplex connections are closed and the connection broker is complete
Core Realization
takedocker-proxy implementation was modified to additionally support active exit logic.
-
()This line of code can be left out, alreadyEOF, no more data will appear for this connection. - Read or write failure scenarios are all included in the
in the process and ignores error handling to minimize the interaction of the two agent processes.
func ConnCat(ctx , client *, backend *) {
var wg
broker := func(to, from *) {
(to, from)
()
()
()
}
(2)
go broker(client, backend)
go broker(backend, client)
finish := make(chan struct{})
go func() {
()
close(finish)
}()
select {
case <-():
case <-finish:
}
()
()
<-finish
}
consultation
- /moby/moby/blob/master/cmd/docker-proxy/tcp_proxy_linux.go#L27, docker-proxy's tcp proxy implementation