Possible bug in RTPInterface::sendDataOverTCP

[Sergey Lvov via live-devel]

Hallo everybody!

I discovered strange disconnections when I used streaming over TCP.
I recompiled live555 library with -DDEBUG and -DDEBUG_SEND and saw some diagnostic:

sendRTPorRTCPPacketOverTCP: 1448 bytes over channel 0 (socket 7)
sendDataOverTCP: resending 795-byte send (blocking)
sendDataOverTCP: blocking send() failed (delivering -1 bytes out of 795); closing socket 7
SocketDescriptor(socket 7)::deregisterRTPInterface(channel 255)
sendRTPorRTCPPacketOverTCP: failed! (errno 11)
RTSPClientConnection[0x8e80978]::handleRequestBytes() read 4 new bytes:$
RTSPClientConnection[0x8e80978]::handleRequestBytes() read 52 new bytes:?
schedule(5.170436->1411036332.468457)
RTSPClientConnection[0x8e7baf0]::handleRequestBytes() read 212 new bytes:GET_PARAMETER rtsp://192.168.0.35:8554/archive?record=541697a20c8ac43f&sessionId=35/ RTSP/1.0
CSeq: 21349
User-Agent: LibVLC/2.2.0-pre4-20140908-0202 (LIVE555 Streaming Media v2014.07.25)
Session: CED66A9C

So, errno 11 - it's EAGAIN, and that's very strange for socket in blocking mode.

However, I found topic: stackoverflow.com/questions/735249/blocking-socket-returns-eagain
And I understood that is quite possible.

I tried to fix the problem by this way:

- sendResult = send(socketNum, (char const*)(&data[numBytesSentSoFar]), numBytesRemainingToSend, 0/*flags*/);
+
+ do {
+ sendResult = send(socketNum, (char const*)(&data[numBytesSentSoFar]), numBytesRemainingToSend, 0/*flags*/);
+ } while(sendResult == -1 && envir().getErrno() == EAGAIN);


And it works now!

Could you possibly investigate this problem?

Thank you for you work!

Best regards,
Sergey.

[Ross Finlayson]

Post by Sergey Lvov via live-devel
- sendResult = send(socketNum, (char const*)(&data[numBytesSentSoFar]), numBytesRemainingToSend, 0/*flags*/);
+
+ do {
+ sendResult = send(socketNum, (char const*)(&data[numBytesSentSoFar]), numBytesRemainingToSend, 0/*flags*/);
+ } while(sendResult == -1 && envir().getErrno() == EAGAIN);

No, you can't do this (and there's not a 'problem' that needs fixing)! If the TCP connection gets blocked permanently (e.g., because the receiving client has stopped running, or has a cable disconnected), then you can't just sit in a loop, attempting to send() over and over again. That would starve out all other server activity.

There's no 'bug' or 'problem' here. The "EAGAIN" error occurs when the sending OS's TCP buffer is full - which occurs if the stream's bitrate exceeds (at least temporarily) the capacity of the TCP connection. When this happens, the only solution is to discard outgoing data. Our code does so by ensuring that if/when data gets discarded, the discarded data will be a complete RTP (or RTCP) packet - equivalent to the loss of a packet if you were streaming over UDP.

Some people seem to think that streaming over TCP is a 'magic wand' that will avoid all data loss, regardless of the bitrate of your stream and the capacity of your network. But that's impossible. If your stream's bitrate exceeds the capacity of your network, you *will* lose data. The only way to prevent this is either to transmit a slower stream, or use a faster network.

Yet again, let me remind everyone that streaming over TCP is something that you should be doing *only* if you are behind a firewall that blocks UDP packets

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

[Sergey Lvov via live-devel]

Thank you very much for detailed explanation.
Probably increasing of the send buffer (SO_SNDBUF) can help, it's value depends on estimatedBitrate from RTPSink:

if (rtpSink != NULL && rtpSink->estimatedBitrate() > 0) streamBitrate = rtpSink->estimatedBitrate();

if (rtpGroupsock != NULL) {
// Try to use a big send buffer for RTP - at least 0.1 second of
// specified bandwidth and at least 50 KB
unsigned rtpBufSize = streamBitrate * 25 / 2; // 1 kbps * 0.1 s = 12.5 bytes
if (rtpBufSize < 50 * 1024) rtpBufSize = 50 * 1024;
increaseSendBufferTo(envir(), rtpGroupsock->socketNum(), rtpBufSize);
}

Post by Ross Finlayson

Post by Sergey Lvov via live-devel
- sendResult = send(socketNum, (char const*)(&data[numBytesSentSoFar]), numBytesRemainingToSend, 0/*flags*/);
+
+ do {
+ sendResult = send(socketNum, (char const*)(&data[numBytesSentSoFar]), numBytesRemainingToSend, 0/*flags*/);
+ } while(sendResult == -1 && envir().getErrno() == EAGAIN);

No, you can't do this (and there's not a 'problem' that needs fixing)! If the TCP connection gets blocked permanently (e.g., because the receiving client has stopped running, or has a cable disconnected), then you can't just sit in a loop, attempting to send() over and over again. That would starve out all other server activity.
There's no 'bug' or 'problem' here. The "EAGAIN" error occurs when the sending OS's TCP buffer is full - which occurs if the stream's bitrate exceeds (at least temporarily) the capacity of the TCP connection. When this happens, the only solution is to discard outgoing data. Our code does so by ensuring that if/when data gets discarded, the discarded data will be a complete RTP (or RTCP) packet - equivalent to the loss of a packet if you were streaming over UDP.
Some people seem to think that streaming over TCP is a 'magic wand' that will avoid all data loss, regardless of the bitrate of your stream and the capacity of your network. But that's impossible. If your stream's bitrate exceeds the capacity of your network, you *will* lose data. The only way to prevent this is either to transmit a slower stream, or use a faster network.
Yet again, let me remind everyone that streaming over TCP is something that you should be doing *only* if you are behind a firewall that blocks UDP packets
Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

[Ross Finlayson]

Post by Sergey Lvov via live-devel
Probably increasing of the send buffer (SO_SNDBUF) can help

Yes, you can call "increaseSendBufferTo()" in your application. However, you'll still get data loss if your stream's bitrate exceeds the capacity of your TCP connection (which is *not* the same as the nominal bitrate of your network). You can never avoid that.


Ross Finlayson
Live Networks, Inc.
http://www.live555.com/