scaling live555 rtsp proxy to hundreds of viewers.

[Pete Pulliam]

I have an implementation of an rtsp proxy server based on the
live555ProxyServer that ships with the live555 source. When connecting
only a couple of viewers to a single proxied stream, things look great.
I'm hoping to get a few hundred viewers per stream though. What I'm seeing
is that there is a large drop off in quality and kbps as I add more
viewers. I see this drop off both with the custom proxy I've written and
the stock proxy that ships with the source.

While this is happening, the CPU and memory are only lightly used, and the
NIC is not that busy.

I'm trying to use this to shield a transcoder that provides an RTSP origin
(as well as several other formats). When I run the same test directly
against the origin I don't see a drop off in quality like this for viewers
less than 50 (and haven't tested that for more than 50 viewers). Hitting
the origin directly looks great.

Is there perhaps something I should have tuned in the proxy, or could tune
with the Linux box this is running on that is causing this drop off?
Advice wanted to improve the scaling of single stream => many viewer.

The kbps drop off looks like:
# ​
of clients
​ , ​
bitrate (kbps)
2 977.688820
3 976.666311
4 936.494160
5 940.096328
6 944.486316
7 955.723431
8 945.044076
9 944.803396
10 930.465372
11 925.247045
12 931.066158
13 713.205068
14 486.331708
15 492.473134
16 485.362249
17 483.711453
18 490.636454
19 485.919289
20 487.862608
21 489.587224

Pete

[Ross Finlayson]

Problems like this are often caused by running into an OS-imposed limit on the number of open files (i.e., sockets) that a process can have open at a time.
http://www.live555.com/liveMedia/faq.html#scalability
Sometimes this limit tends to be a 'soft limit'; if you approach it, the OS's performance may suffer, but not fail completely.

So, I would recommend checking whether you can increase this limit in your OS.


Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

[Jeff Shanab]

Are you making a copy for each connected viewer?
The system I worked on just over a year ago could stream around 400 streams
but never was it 400 of 1 stream, it was 5 or 10 of 100-200 sources.
Even then I used a buffer pool and a shared pointer so when the last
unicast client was sent the packet, it returned itself to the pool.

Another thing was my architecture, which may be important. I was RTSP in
from many sources and serving thru my own http protocol basterdization. In
this I ran "Processors" of a settable number of sources. Each Processor was
it's own usage environment and ran in a thread. This allowed the OS to
parallelize the I/O to the buffers to the ethernet interface. IF the event
Queue design, whichdoes have the best overall thruoughput, gets too deep,
(spread across 2 many sources) then latency can go up and slow overall
thruput to multiple clients.

I was able to get to the limit of the gigibit network.
I usually ran from 10 to 64 sources per "processor"
If the cameras had small frames like D1 it was better to run 64 per
processor. (FD_SET on windows is slower beyond 64 sockets)
If they were high resolution cameras and had big nal frames, it was
faster to break it into 6 processors of 10 and let the OS utilize the cores.
(Threads tend to stay on a single core)

Post by Pete Pulliam
I have an implementation of an rtsp proxy server based on the
live555ProxyServer that ships with the live555 source. When connecting
only a couple of viewers to a single proxied stream, things look great.
I'm hoping to get a few hundred viewers per stream though. What I'm seeing
is that there is a large drop off in quality and kbps as I add more
viewers. I see this drop off both with the custom proxy I've written and
the stock proxy that ships with the source.
While this is happening, the CPU and memory are only lightly used, and the
NIC is not that busy.
I'm trying to use this to shield a transcoder that provides an RTSP origin
(as well as several other formats). When I run the same test directly
against the origin I don't see a drop off in quality like this for viewers
less than 50 (and haven't tested that for more than 50 viewers). Hitting
the origin directly looks great.
Is there perhaps something I should have tuned in the proxy, or could tune
with the Linux box this is running on that is causing this drop off?
Advice wanted to improve the scaling of single stream => many viewer.
# ​
of clients
​ , ​
bitrate (kbps)
2 977.688820
3 976.666311
4 936.494160
5 940.096328
6 944.486316
7 955.723431
8 945.044076
9 944.803396
10 930.465372
11 925.247045
12 931.066158
13 713.205068
14 486.331708
15 492.473134
16 485.362249
17 483.711453
18 490.636454
19 485.919289
20 487.862608
21 489.587224
Pete
The information in this message may be confidential. It is intended
solely for
the addressee(s). If you are not the intended recipient, any disclosure,
copying or distribution of the message, or any action or omission taken by
you
in reliance on it, is prohibited and may be unlawful. Please immediately
contact the sender if you have received this message in error.
_______________________________________________
live-devel mailing list
http://lists.live555.com/mailman/listinfo/live-devel