深入分析broken pipe和connetion reset by peer

这两个异常在网络编程里应该是比较常见的,之前对于他们的理解只知道是对端关闭了连接,但什么情况下会报Broken pipe,什么情况下会报Connetion reset by peer是一无所知的,今天专门就这个问题做一个调研,下面记录下整个调研过程。

TCP的连接的建立与终止

在讲具体原因之前,有必要说一下TCP的三次握手与四次挥手,下面用一张图来说明(图是网上找的)。

tcp_hank

上面三条是连接的建立,也就是说的三次挥手;下面四条是正常的连接终止,也就是四次挥手,关于为什么建立连接需要三次交互,而终止连接需要四次交互,这里不做过多解释,可以查询相关信息。正常关闭连接是发送FIN,还有一种异常关闭是发送RST,今天要讨论的就和这个有关系。

异常模拟

下面写了一段NIO的代码,Server端代码如下,代码很简单,看不懂的可以稍微补一点Nio的知识:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.util.Iterator;
import java.util.concurrent.TimeUnit;

public class {
public static void main(String[] args) throws IOException, InterruptedException {
ServerSocketChannel serverSocketChannel = ServerSocketChannel.open();
serverSocketChannel.socket().bind(new InetSocketAddress(9000));
serverSocketChannel.configureBlocking(false);
Selector selector = Selector.open();
serverSocketChannel.register(selector, SelectionKey.OP_ACCEPT);

while (true) {
int selected = selector.select();
if (selected > 0) {
Iterator<SelectionKey> iterator = selector.selectedKeys().iterator();
while (iterator.hasNext()) {
SelectionKey selectionKey = iterator.next();
iterator.remove();
if (selectionKey.isAcceptable()) {
System.err.println("Acceptable");
SocketChannel socketChannel = serverSocketChannel.accept();
socketChannel.configureBlocking(false);
socketChannel.register(selector, SelectionKey.OP_READ);
} else if (selectionKey.isReadable()) {
System.err.println("Readable");
SocketChannel socketChannel = (SocketChannel) selectionKey.channel();
ByteBuffer buffer = ByteBuffer.allocate(128);
socketChannel.read(buffer);
System.out.println("接收来自客户端的数据:" + new String(buffer.array()));
selectionKey.interestOps(SelectionKey.OP_WRITE);
} else if (selectionKey.isWritable()) {
System.err.println("Writable");
TimeUnit.SECONDS.sleep(3);
SocketChannel channel = (SocketChannel) selectionKey.channel();
String content = "向客户端发送数据 : " + System.currentTimeMillis();
ByteBuffer bufferOne = ByteBuffer.wrap(content.getBytes());
channel.write(bufferOne);
System.err.println("发送bufferOne");
TimeUnit.SECONDS.sleep(3);
ByteBuffer bufferTwo = ByteBuffer.wrap(content.getBytes());
channel.write(bufferTwo); //@2
System.err.println("发送bufferTwo");
selectionKey.interestOps(SelectionKey.OP_READ);
}
}
}
}
}
}

大家可以看到,在接收到客户端的数据以后,首先会往客户端写数据,在@1代码处,然后会再次往客户端写数据,在@2代码处。下面再看一下客户端的代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.SocketChannel;
import java.util.Iterator;

public class NioClient {
public static void main(String[] args) throws IOException {
SocketChannel socketChannel = SocketChannel.open();

socketChannel.configureBlocking(false);
Selector selector = Selector.open();
socketChannel.register(selector, SelectionKey.OP_CONNECT);
socketChannel.connect(new InetSocketAddress("192.168.2.60", 9000));
//socketChannel.socket().setSoLinger(true,0); //@3

while (true) {
int select = selector.select();
if (select > 0) {
Iterator<SelectionKey> iterator = selector.selectedKeys().iterator();
while (iterator.hasNext()) {
SelectionKey selectionKey = iterator.next();
if (selectionKey.isConnectable()) {
System.err.println("Connectable");
SocketChannel clientChannel = (SocketChannel) selectionKey.channel();
clientChannel.finishConnect();
selectionKey.interestOps(SelectionKey.OP_WRITE);
} else if (selectionKey.isReadable()) {
System.out.println("Readable");
SocketChannel channel = (SocketChannel) selectionKey.channel();
ByteBuffer buffer = ByteBuffer.allocate(128);
channel.read(buffer);
selectionKey.interestOps(SelectionKey.OP_WRITE);
System.out.println("收到服务端数据" + new String(buffer.array()));
} else if (selectionKey.isWritable()) {
SocketChannel clientChannel = (SocketChannel) selectionKey.channel();
String str = "qiwoo mobile";
ByteBuffer buffer = ByteBuffer.wrap(str.getBytes());
clientChannel.write(buffer);
selectionKey.interestOps(SelectionKey.OP_READ);
System.out.println("向服务端发送数据" + new String(buffer.array()));
clientChannel.close(); //@4
}
iterator.remove();
}
}
}
}
}

当客户端建立完连接往服务端发送数据后,会调close()方法,关闭连接,在代码@4处。我们先启动NioServer,然后再启动NioClient。会发现NioServer会报如下异常:

1
2
3
4
5
6
7
8
9
10
11
12
13
[[email protected] tmp]# java NioServer
Acceptable
Readable
接收来自客户端的数据:qiwoo mobile
Writable
发送bufferOne
Exception in thread "main" java.io.IOException: 断开的管道
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at NioServer.main(NioServer.java:48)

而此时,我们用tcpdump抓包看的情况如下:

1
2
3
4
5
6
7
8
9
17:35:38.819801 IP 192.168.2.11.33962 > 192.168.2.60.cslistener: Flags [S], seq 3601143964, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
17:35:38.820008 IP 192.168.2.60.cslistener > 192.168.2.11.33962: Flags [S.], seq 3163373872, ack 3601143965, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
17:35:38.820072 IP 192.168.2.11.33962 > 192.168.2.60.cslistener: Flags [.], ack 1, win 29, length 0
17:35:38.821801 IP 192.168.2.11.33962 > 192.168.2.60.cslistener: Flags [P.], seq 1:13, ack 1, win 29, length 12
17:35:38.821958 IP 192.168.2.60.cslistener > 192.168.2.11.33962: Flags [.], ack 13, win 58, length 0
17:35:38.822097 IP 192.168.2.11.33962 > 192.168.2.60.cslistener: Flags [F.], seq 13, ack 1, win 29, length 0
17:35:38.861551 IP 192.168.2.60.cslistener > 192.168.2.11.33962: Flags [.], ack 14, win 58, length 0
17:35:41.825209 IP 192.168.2.60.cslistener > 192.168.2.11.33962: Flags [P.], seq 1:41, ack 14, win 58, length 40
17:35:41.825223 IP 192.168.2.11.33962 > 192.168.2.60.cslistener: Flags [R], seq 3601143978, win 0, length 0

结合上面信息,我们可以看到,在NioClient调用close()方法后,NioClient会往NioServer发送一个FIN包,而NioServer还继续往NioClient写数据,此时会收到NioClient返回的RST;接着NioServer继续往NioClient写数据,这个时候,NioServer就报了Broken pipe;

现在我们将NioClient代码处@3的注释打开,然后执行同样的操作,会发现NioServer报如下异常:

1
2
3
4
5
6
7
8
9
10
11
12
[[email protected] tmp]# java NioServer
Acceptable
Readable
接收来自客户端的数据:qiwoo mobile
Writable
Exception in thread "main" java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at NioServer.main(NioServer.java:44)

而此时,我们用tcpdump抓包看的情况如下:

1
2
3
4
5
6
17:37:12.234530 IP 192.168.2.11.34818 > 192.168.2.60.cslistener: Flags [S], seq 2138229963, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
17:37:12.234728 IP 192.168.2.60.cslistener > 192.168.2.11.34818: Flags [S.], seq 3174262802, ack 2138229964, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
17:37:12.234740 IP 192.168.2.11.34818 > 192.168.2.60.cslistener: Flags [.], ack 1, win 29, length 0
17:37:12.295824 IP 192.168.2.11.34818 > 192.168.2.60.cslistener: Flags [P.], seq 1:13, ack 1, win 29, length 12
17:37:12.296007 IP 192.168.2.60.cslistener > 192.168.2.11.34818: Flags [.], ack 13, win 58, length 0
17:37:12.296074 IP 192.168.2.11.34818 > 192.168.2.60.cslistener: Flags [R.], seq 13, ack 1, win 29, length 0

结合上面信息,我们可以看到,在NioClient调用close()方法后,因为配置了soLinger选项,也即,NioClient会往NioServer发送一个RST包,而NioServer还继续往NioClient写数据,这个时候,NioServer就报了Broken pipe。

结论

从上面的两个例子可以看出,对于被动关闭方来说,也就是上述例子的NioServer,有两种情况:

被FIN关闭

第一次write得到了RST响应,但不会抛异常,第二次write,就会报Broken pipe(针对Linux系统)

被RST关闭

无论读写都会报Connection reset by peer

RST的场景

什么场景下会发送RST呢,网上总结的也很多,这里就不在赘述了。

参考文章