[asio] What happens when socket receive buffer overflows?
Hello List, I have a network application written with asio that uses TCP. The message exchange between client and server is done using a protocol that puts a prefix on each message indicating the number its bytes number. This way, at one side a prefix is read, and the number of bytes indicated in the prefix is read as a message, then it reads another prefix and so on. While testing the application I was having some errors related to reading a prefix that would indicate a strange number of bytes incoming, a number that was far greater than the max size possible of message in my protocol. After assuring that the bytes on the stream were not being merged, i started suspecting about socket buffer size. So i reduced the socket buffer size to 1K, and the application crashed with a lot less network load than before. At the moment, i am almost sure that the problem is the socket receive buffer which overflows and the bytes are discarded, creating a "gap" in the byte stream which causes an inconsistent state on the application. Does that make sense? What should happen when the socket receive buffer overflows? It silently discards the bytes or what? Thanx for your attention, -- Matheus Araújo Aguiar Computer Scientist matheus.pit@gmail.com
Not sure if anyone replied yet or not ...
... protocol that puts a prefix on each message indicating the number its bytes number.
Yes, a fairly common design (I've done a variation on it at least a dozen times).
... errors related to reading a prefix that would indicate a strange number of bytes incoming, a number that was far greater than the max size possible of message ...
Also pretty common problem. I'd bet a major body part there's a bug in the sending (or receiving) side of the app. Something has dropped some bytes somewhere, or lost track of where it is in the "message framing".
... So i reduced the socket buffer size to 1K, and the application crashed with a lot less network load than before. ... At the moment, i am almost sure that the problem is the socket receive buffer which overflows and the bytes are discarded, creating a "gap" in the byte stream which causes an inconsistent state on the application.
As mentioned, you're probably on the mark as to why you're crashing. But not as to the underlying cause. You're focusing on an OS or TCP stack problem, rather than your code. The TCP protocol has "flow control" built in, and if the receiving side fills up or is congested, it will tell the sending side to quit sending. If packets get dropped while the flow control is "kicking in", the packets will (eventually) get re-transmitted. (On a side note, this type of congestion can be a system-level problem, with certain use cases.)
... What should happen when the socket receive buffer overflows? It silently discards the bytes or what?
TCP never "silently discards" bytes (UDP will, obviously). There's a bug in your code (including the way you're using Asio). Asio streambufs or iostreams can leave characters in the buffer, and your logic must account for these. If you're still having the problem, reduce your code to the smallest example that will reproduce the crash, and post it to the mailing list (there's also an Asio mailing list you can use). Cliff
Thanks for your reply Cliff, On Fri, Apr 9, 2010 at 2:50 PM, Cliff Green <cliffg@codewrangler.net> wrote:
Not sure if anyone replied yet or not ...
... protocol that puts a prefix on each message indicating the number its bytes number.
Yes, a fairly common design (I've done a variation on it at least a dozen times).
... errors related to reading a prefix that would indicate a strange number of bytes incoming, a number that was far greater than the max size possible of message ...
Also pretty common problem. I'd bet a major body part there's a bug in the sending (or receiving) side of the app. Something has dropped some bytes somewhere, or lost track of where it is in the "message framing".
... So i reduced the socket buffer size to 1K, and the application crashed with a lot less network load than before. ... At the moment, i am almost sure that the problem is the socket receive buffer which overflows and the bytes are discarded, creating a "gap" in the byte stream which causes an inconsistent state on the application.
As mentioned, you're probably on the mark as to why you're crashing. But not as to the underlying cause.
You're focusing on an OS or TCP stack problem, rather than your code. The TCP protocol has "flow control" built in, and if the receiving side fills up or is congested, it will tell the sending side to quit sending. If packets get dropped while the flow control is "kicking in", the packets will (eventually) get re-transmitted. (On a side note, this type of congestion can be a system-level problem, with certain use cases.)
Probably you are right. Believing that the problem was with the buffer, i modified the way i read from the socket, in order to make it faster, and the problem is not occurring anymore.
... What should happen when the socket receive buffer overflows? It silently discards the bytes or what?
TCP never "silently discards" bytes (UDP will, obviously). There's a bug in your code (including the way you're using Asio).
Asio streambufs or iostreams can leave characters in the buffer, and your logic must account for these.
If you're still having the problem, reduce your code to the smallest example that will reproduce the crash, and post it to the mailing list (there's also an Asio mailing list you can use).
Cliff
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- Matheus Araújo Aguiar Computer Scientist matheus.pit@gmail.com
2010/4/7 Matheus Araújo Aguiar <matheus.pit@gmail.com>:
While testing the application I was having some errors related to reading a prefix that would indicate a strange number of bytes incoming, a number that was far greater than the max size possible of message in my protocol.
It sounds like you have made the problem go away. FWIW, on the send side of a TCP connection, you may do 4 individual writes of 64 bytes each, but on the receive side you may only see 256 bytes of data available on the socket. TCP is a stream protocol, so at the application level, there is no inherent concept of packets or messages. TCP *uses* segments, which are packets on "the wire", but gives you a stream at the application level when you read data from the socket. So there is no correlation between the number of writes on the send side, and the number of reads on the receive side. It only guarantees that you will receive the bytes in the same order they were sent... If you receive them at all. ;-) You must design you socket reader with this in mind. Good luck! :-) Jon
On Fri, Apr 9, 2010 at 5:52 PM, Jonathan Franklin < franklin.jonathan@gmail.com> wrote:
2010/4/7 Matheus Araújo Aguiar <matheus.pit@gmail.com>:
While testing the application I was having some errors related to reading a prefix that would indicate a strange number of bytes incoming, a number that was far greater than the max size possible of message in my protocol.
It sounds like you have made the problem go away.
Apparently, yes!
FWIW, on the send side of a TCP connection, you may do 4 individual writes of 64 bytes each, but on the receive side you may only see 256 bytes of data available on the socket.
TCP is a stream protocol, so
at the application level, there is no inherent concept of packets or messages. TCP *uses* segments, which are packets on "the wire", but gives you a stream at the application level when you read data from the socket. So there is no correlation between the number of writes on the send side, and the number of reads on the receive side. It only guarantees that you will receive the bytes in the same order they were sent... If you receive them at all. ;-)
You must design you socket reader with this in mind. Good luck! :-)
Thanks for the tip. Actually i was already thinking like that. When i refer to messages in my post, i want to mean "number of bytes expected before reading another size of bytes expected" ..
Jon _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
best regards, -- Matheus Araújo Aguiar Computer Scientist matheus.pit@gmail.com
participants (3)
-
Cliff Green -
Jonathan Franklin -
Matheus Araújo Aguiar