Slow connection problem with RTP over RTSP/TCP

[m***@orangemail.sk]

Hello,

I have tested LIVE555 and I found one problem. Clients connecting to live server over slow network have problems receiving data with RTSP TCP encapsulation turned on. If bitrate of the video is bigger, than available network bandwidth, Live server discards packed randomly. This keeps network busy sending data, but client receives few or none whole
frames.

I found problem in function sendRTPOverTCP in file RTPInterface.cpp. If send() function is unable to send data, because OS buffer is full, it simply discards rest of the data. This is not very good solution, because it waste bandwidth and breaks frames. On some video sources and for some type of application it is possible to reduce framerate, so you can watch video on slow network. It is better to send whole frame, then
skip some frames and again send whole frame.

My solution for this problem is to try to repeat send for the same data, if there was no space for them in OS buffers. I tried to determine, if it is possible to send data with function select() and then send them or schedule this data for sending in another time. This solution is valid only with sources, that can change framerate, like web camera, it does not help with file sources that much, except it seems to be more stable for them on slow networks.

I found out that this solution also helps with really big frames, which were not send sometime even on LAN.

I do not know how to make proper patch file, so you can test it with live.2009.11.12.tar.gz. I attached some diff file generated by svn, with my changes. Maybe someone will find this information useful.

Martin

Index: .liveMediaincludeRTPInterface.hh
===================================================================
--- .liveMediaincludeRTPInterface.hh
+++ .liveMediaincludeRTPInterface.hh
@@ -63,7 +63,7 @@

void setClientSession(void* clientSession){rtspClientSession = clientSession;}

- void sendPacket(unsigned char* packet, unsigned packetSize);
+ int sendPacket(unsigned char* packet, unsigned packetSize);
void startNetworkReading(TaskScheduler::BackgroundHandlerProc*
handlerProc);
bool handleRead(unsigned char* buffer, unsigned bufferMaxSize,
Index: .liveMediaincludeMultiFramedRTPSink.hh
===================================================================
--- .liveMediaincludeMultiFramedRTPSink.hh
+++ .liveMediaincludeMultiFramedRTPSink.hh
@@ -99,6 +99,7 @@
void sendPacketIfNecessary();
static void sendNext(void* firstArg);
friend void sendNext(void*);
+ static void sendThisAgain(void* firstArg);

static void afterGettingFrame(void* clientData,
unsigned numBytesRead, unsigned numTruncatedBytes,
Index: .liveMediaRTPInterface.cpp
===================================================================
--- .liveMediaRTPInterface.cpp
+++ .liveMediaRTPInterface.cpp
@@ -34,7 +34,7 @@
// Helper routines and data structures, used to implement
// sendingreceiving RTPRTCP over a TCP socket:

-static void sendRTPOverTCP(unsigned char* packet, unsigned packetSize,
+static int sendRTPOverTCP(unsigned char* packet, unsigned packetSize,
int socketNum, unsigned char streamChannelId);

// Reading RTP-over-TCP is implemented using two levels of hash tables.
@@ -166,16 +166,18 @@
}
}

-void RTPInterface::sendPacket(unsigned char* packet, unsigned packetSize) {
+int RTPInterface::sendPacket(unsigned char* packet, unsigned packetSize) {
// Normal case: Send as a UDP packet:
fGS->output(envir(), fGS->ttl(), packet, packetSize);
+ int iSend = packetSize;

// Also, send over each of our TCP sockets:
for (tcpStreamRecord* streams = fTCPStreams; streams != NULL;
streams = streams->fNext) {
- sendRTPOverTCP(packet, packetSize,
+ iSend = sendRTPOverTCP(packet, packetSize,
streams->fStreamSocketNum, streams->fStreamChannelId);
}
+ return iSend;
}

void RTPInterface
@@ -256,7 +258,7 @@

////////// Helper Functions - Implementation ////////

-void sendRTPOverTCP(unsigned char* packet, unsigned packetSize,
+int sendRTPOverTCP(unsigned char* packet, unsigned packetSize,
int socketNum, unsigned char streamChannelId) {
#ifdef DEBUG
fprintf(stderr, "sendRTPOverTCP: %d bytes over channel %d (socket %d)n",
@@ -265,6 +267,21 @@
// Send RTP over TCP, using the encoding defined in
// RFC 2326, section 10.12:
do {
+
+ //This code is trying to check if we are able to send data over TCP
+ //in some case we cannot send data, because winows buffers are full of our previous data
+ fd_set wrts;
+ struct timeval timeout;
+ timeout.tv_sec = 0;
+ timeout.tv_usec = 1;
+ FD_ZERO(&wrts);
+ FD_SET(socketNum,&wrts);
+ select(NULL, NULL, &wrts, NULL, &timeout);
+ if (!FD_ISSET(socketNum, &wrts))
+ {
+ return 0;
+ }
+
char const dollar = '$';
if (send(socketNum, &dollar, 1, 0) != 1) break;
if (send(socketNum, (char*)&streamChannelId, 1, 0) != 1) break;
@@ -280,13 +297,17 @@
fprintf(stderr, "sendRTPOverTCP: completedn"); fflush(stderr);
#endif

- return;
+ if (packetSize != 0)
+ return packetSize;
+ else
+ return 1;
} while (0);

RTPOverTCP_OK = false; // HACK #####
#ifdef DEBUG
fprintf(stderr, "sendRTPOverTCP: failed!n"); fflush(stderr);
#endif
+ return -1;
}

SocketDescriptor::SocketDescriptor(UsageEnvironment& env, int socketNum)
Index: .liveMediaMultiFramedRTPSink.cpp
===================================================================
--- .liveMediaMultiFramedRTPSink.cpp
+++ .liveMediaMultiFramedRTPSink.cpp
@@ -357,7 +357,12 @@
#ifdef TEST_LOSS
if ((our_random()%10) != 0) // simulate 10% packet loss #####
#endif
- fRTPInterface.sendPacket(fOutBuf->packet(), fOutBuf->curPacketSize());
+ if (fRTPInterface.sendPacket(fOutBuf->packet(), fOutBuf->curPacketSize()) == 0)
+ {
+ //if I was unable to send data through TCP, try to send them later
+ nextTask() = envir().taskScheduler().scheduleDelayedTask (100, (TaskFunc*)sendThisAgain, this);
+ return;
+ }
++fPacketCount;
fTotalOctetCount += fOutBuf->curPacketSize();
fOctetCount += fOutBuf->curPacketSize()
@@ -411,6 +416,12 @@
sink->buildAndSendPacket(false);
}

+// The following is called after TCP was unable to send data
+void MultiFramedRTPSink::sendThisAgain(void* firstArg) {
+ MultiFramedRTPSink* sink = (MultiFramedRTPSink*)firstArg;
+ sink->sendPacketIfNecessary();
+}
+
void MultiFramedRTPSink::ourHandleClosure(void* clientData) {
MultiFramedRTPSink* sink = (MultiFramedRTPSink*)clientData;
// There are no frames left, but we may have a partially built packet

[Ross Finlayson]

No, this is nonsense. TCP is intended to be a reliable byte-stream
protocol; it's the job of the operating system's TCP implementation -
not application-level code (such as LIVE555) - to provide reliable
delivery. If, however, your stream's bitrate is too large for your
network, then you're inevitably going to get packet loss (usually
because an OS buffer in the sender OS will overflow). There's
nothing you can do to avoid this. If your stream's bitrate is really
too large for your network, then you should not be sending it (and
you should certainly not be sending it over TCP).

If, however, your stream's *average* bitrate is within the capacity
of your network, you can reduce the probability of your server OS's
TCP implementation losing data by increasing its OS socket buffer.
Note that we provide a function "increaseSendBufferTo()" that does
this. Our RTSP server implementation - by default - sets this buffer
size (for each server and stream socket) to 50 kBytes (search for
"increaseSendBufferTo" in the code). However, you could use a larger
value this (e.g., by subclassing "RTSPServer").
--

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

[Matt Schuckmann]

Ross, I have to disagree with you a little here.
Shouldn't it be the servers responsibility to provide the best possible
experience to the client given the network conditions, the content
delivery method and the content requested by the client.
And if the server can detect that the network is to slow or congested to
send all of the requested data (by detecting that the network buffers in
the OS are full) make an attempt to intelligently send enough data to
provide a working, all be it degraded, experience to the client rather
than just randomly dropping parts of frames so that nothing may work.

I haven't looked at his patch so I can't say if it's good or bad but I
do know we ended up doing something similar for our branch of the
RTSPServer because we can not guarantee all the conditions our product
will be used in and we wanted to the best we could to provide a working
system as much as possible.

Matt S.


Ross Finlayson wrote:
> No, this is nonsense. TCP is intended to be a reliable byte-stream
> protocol; it's the job of the operating system's TCP implementation -
> not application-level code (such as LIVE555) - to provide reliable
> delivery. If, however, your stream's bitrate is too large for your
> network, then you're inevitably going to get packet loss (usually
> because an OS buffer in the sender OS will overflow). There's nothing
> you can do to avoid this. If your stream's bitrate is really too
> large for your network, then you should not be sending it (and you
> should certainly not be sending it over TCP).
>
> If, however, your stream's *average* bitrate is within the capacity of
> your network, you can reduce the probability of your server OS's TCP
> implementation losing data by increasing its OS socket buffer. Note
> that we provide a function "increaseSendBufferTo()" that does this.
> Our RTSP server implementation - by default - sets this buffer size
> (for each server and stream socket) to 50 kBytes (search for
> "increaseSendBufferTo" in the code). However, you could use a larger
> value this (e.g., by subclassing "RTSPServer").

[HQ Wang]

Hi all,



I am tring to forward the H.264 stream from a H.264 RTSP server to VLC by live555. Since I don't care the type of UDP payload, I don't want to unpack&repack the H.264 stream over RTP. Can I forward the UDP datagram directly to VLC?



Just made an attemption, it couldn't play on VLC -- the Input Demuxed kept increasing, but the decoded blocks and display frames were always 0.



Thanks in advance for your help,

Wilson

_________________________________________________________________
Hotmail: Trusted email with Microsoft's powerful SPAM protection.
http://clk.atdmt.com/GBL/go/177141664/direct/01/
http://clk.atdmt.com/GBL/go/177141664/direct/01/

[m***@orangemail.sk]

>No, this is nonsense. TCP is intended to be a reliable
>byte-stream protocol; it's the job of the operating system's
>TCP implementation - not application-level code (such as
>LIVE555) - to provide reliable delivery.

Yes it is true, but problem is that LIVE is loosing the data, not the OS. LIVE is using non blocking sockets. If you are unable to send data, send() returns with number of data send and you discard the rest. That implementation would work for blocking sockets, because you are guarantee to really send the data, but not here.

This generate traffic on the network, but as packet are discarded randomly, you must discard many incomplete frames on the client, so much of the network traffic is wasted.

>If, however, your stream's bitrate is too large for your
>network, then you're inevitably going to get packet loss
>(usually because an OS buffer in the sender OS will
>overflow). There's nothing you can do to avoid this. If your
>stream's bitrate is really too large for your network, then you
>should not be sending it (and you should certainly not be
>sending it over TCP).

In some cases you can change bitrate of the video by changing framerate. So you can save the day even if original setup was unsuitable for network bandwidth. This is true if you are compressing video on the fly and do not care, that much about framerate.

>If, however, your stream's *average* bitrate is within the
>capacity of your network, you can reduce the probability of
>your server OS's

I have very little knowledge about portability, but I can think, you can support it at least for some OS, by #ifdef.

>Note that we provide a function "increaseSendBufferTo()" that
>does this. Our RTSP server implementation - by default -
>sets this buffer size (for each server and stream socket) to
>50 kBytes (search for "increaseSendBufferTo" in the code).
>However, you could use a larger value this (e.g., by
>subclassing "RTSPServer").

"Free size" of the OS buffer is dependent also on the TCP window size on the server and on the client, so you have no power over "free size" in OS buffer.

Martin

>Ross Finlayson
>Live Networks, Inc.
>http://www.live555.com/
>_______________________________________________
>live-devel mailing list
>live-***@lists.live555.com
>http://lists.live555.com/mailman/listinfo/live-devel

[Ross Finlayson]

>And if the server can detect that the network is to slow or
>congested to send all of the requested data (by detecting that the
>network buffers in the OS are full) make an attempt to intelligently
>send enough data to provide a working, all be it degraded,
>experience to the client rather than just randomly dropping parts of
>frames so that nothing may work.

Yes, at some point it would be nice to support the IETF's RTP/AVPF
profile, and an associated feedback-based retransmission (and/or
media rescaling) scheme. But this, of course, is on top of UDP.
Trying to implement a loss-mitigation scheme over TCP is retarded.
--

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

[Jeremy Noring]

On Thu, Nov 19, 2009 at 2:31 AM, Ross Finlayson <***@live555.com>wrote:

> And if the server can detect that the network is to slow or congested to
>> send all of the requested data (by detecting that the network buffers in the
>> OS are full) make an attempt to intelligently send enough data to provide a
>> working, all be it degraded, experience to the client rather than just
>> randomly dropping parts of frames so that nothing may work.
>>
>
> Yes, at some point it would be nice to support the IETF's RTP/AVPF profile,
> and an associated feedback-based retransmission (and/or media rescaling)
> scheme. But this, of course, is on top of UDP. Trying to implement a
> loss-mitigation scheme over TCP is retarded.


I agree.

Furthermore, basically the patch implements a "retransmit" on the RTP stack
level, which is wholly incompatible with any RTP specification I know of.
RTP uses application level framing, which means some of the behaviors are
specifically not defined, and should be implemented by the application
level. If my video stream is live, "retransmitting" is likely a waste of
time because the data is already obsolete. And mostly likely I can only
start a stream on keyframe boundaries, so retransmitting some intermediate
frame is also a waste of time if there's already been previous loss. (hence
"application level framing"--only the calling application knows what sort of
loss mitigation plan it should implement)

Question: does Live555 provide a feedback mechanisms for loss, jitter, etc.
(basically some way of getting the RTCP stats) so I can tweak the encoder in
real-time to fit network conditions?

[Ross Finlayson]

>Question: does Live555 provide a feedback mechanisms for loss,
>jitter, etc. (basically some way of getting the RTCP stats) so I can
>tweak the encoder in real-time to fit network conditions?

Yes - note the classes "RTPTransmissionStatsDB" and
"RTPTransmissionStats", defined in "liveMedia/include/RTPSink.hh".

There's no application code that demonstrates how to use this, but
similar classes (for 'reception stats') are used by the "openRTSP"
demo client (see "testProgs/playCommon.cpp"). If you search for
"RTPReceptionStats" in that file, you should be able to figure out
how to access the "RTPTransmissionStats" in your server code.
--

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/