Discussion:
RTSP proxy not noticing server restart
Jonathan Brady
2014-09-22 13:40:55 UTC
Permalink
Hello,

It is possible to restart an origin server quickly enough that a live
555 proxy server does not notice.

As a test this can be done by modifying one of the live 555 test
programs (I was using testH264VideoStreamer) to enable
SO_REUSEADDR/SO_REUSEPORT, then the process could be quickly killed and
restarted without waiting for the socket to timeout. Other examples of
this being embedded systems that reboot quickly.

When the origin server is restarted the proxy server simply stops
receiving data, the proxy sends DESCRIBE requests but they succeed
because of the fast restart so the proxy never notices. Disconnecting
all clients from the proxy server results in a PAUSE command being sent
to the origin server, the proxy receives back an invalid session id, as
it does with a PLAY command when another client connects, however these
errors are ignored.

Once this has happened the only fix is to restart the proxy server, or
stop the origin server long enough for the live 555 proxy to notice.

Is it possible to notice this situation and rectify it?

Regards,
Jonathan
Ross Finlayson
2014-09-22 14:42:01 UTC
Permalink
It is possible to restart an origin server quickly enough that a live 555 proxy server does not notice.
As a test this can be done by modifying one of the live 555 test programs (I was using testH264VideoStreamer) to enable SO_REUSEADDR/SO_REUSEPORT, then the process could be quickly killed and restarted without waiting for the socket to timeout.
And that's precisely why you shouldn't make such a change to the server's code :-)

But anyway, the proxy server sends periodic RTSP "OPTIONS" commands to the 'back-end' server, to test whether its still alive. If any of these "OPTIONS" commands fails (or if the RTSP (TCP) connection to the back-end server fails), then the proxy server will notice this, and establish a new connection (with a new "DESCRIBE" command) to the back-end server.

You should be able to see this by running the proxy server with the -V (uppercase "V") option.

But (getting back to the first point) if your server is able to restart in such a way that a previously-set-up TCP connection can get misinterpreted as still being valid, then that's a serious security flaw in your server OS; that's the real bug that you should be fixing.


Ross Finlayson
Live Networks, Inc.
http://www.live555.com/
Jonathan Brady
2014-09-22 15:17:40 UTC
Permalink
Post by Ross Finlayson
Post by Jonathan Brady
It is possible to restart an origin server quickly enough that a live
555 proxy server does not notice.
As a test this can be done by modifying one of the live 555 test
programs (I was using testH264VideoStreamer) to enable
SO_REUSEADDR/SO_REUSEPORT, then the process could be quickly killed
and restarted without waiting for the socket to timeout.
And that's precisely why you shouldn't make such a change to the
server's code :-)
Its not that I'm making a change to the server code. It was actually a
third party embedded server that is capable of restarting fast enough
that the proxy server never notices. The change to the server code
suggested is just a quick way of reproducing the problem on the same
machine.
Post by Ross Finlayson
But anyway, the proxy server sends periodic RTSP "OPTIONS" commands to
the 'back-end' server, to test whether its still alive. If any of
these "OPTIONS" commands fails (or if the RTSP (TCP) connection to the
back-end server fails), then the proxy server will notice this, and
establish a new connection (with a new "DESCRIBE" command) to the
back-end server.
You should be able to see this by running the proxy server with the -V
(uppercase "V") option.
But (getting back to the first point) if your server is able to
restart in such a way that a previously-set-up TCP connection can get
misinterpreted as still being valid, then that's a serious security
flaw in your server OS; that's the real bug that you should be fixing.
I can indeed see this running the proxy server with -V. It is not the
case of a TCP connection being misinterpreted as valid. It is the case
of when the proxy server retries an OPTIONS command, it sees the TCP
connection has gone away, reconnects and receives a valid response, it
doesn't realise this is because the server has restarted. Perhaps
rather than using OPTIONS something that sends a session id should be
sent and error responses checked for?

Sending request: PLAY rtsp://127.0.0.1:8554/testStream/ RTSP/1.0
CSeq: 4
User-Agent: ProxyRTSPClient (LIVE555 Streaming Media v2014.09.11)
Session: 29A64580


Received 186 new bytes of response data.
Received a complete PLAY response:
RTSP/1.0 200 OK
CSeq: 4
Date: Mon, Sep 22 2014 15:07:50 GMT
Range: npt=29.767-
Session: 29A64580
RTP-Info:
url=rtsp://127.0.0.1:8554/testStream/track1;seq=24127;rtptime=32393928

==================
I restart the origin server here
==================


Opening connection to 127.0.0.1, port 8554...
...remote connection opened
Sending request: OPTIONS rtsp://127.0.0.1:8554/testStream/ RTSP/1.0
CSeq: 6
User-Agent: ProxyRTSPClient (LIVE555 Streaming Media v2014.09.11)
Session: 29A64580


Received 152 new bytes of response data.
Received a complete OPTIONS response:
RTSP/1.0 200 OK
CSeq: 6
Date: Mon, Sep 22 2014 15:08:47 GMT
Public: OPTIONS, DESCRIBE, SETUP, TEARDOWN, PLAY, PAUSE, GET_PARAMETER,
SET_PARAMETER

=============
I close my client here
=============

ProxyServerMediaSubsession["H264"]::closeStreamSource()
Sending request: PAUSE rtsp://127.0.0.1:8554/testStream/ RTSP/1.0
CSeq: 7
User-Agent: ProxyRTSPClient (LIVE555 Streaming Media v2014.09.11)
Session: 29A64580


Received 80 new bytes of response data.
Received a complete PAUSE response:
RTSP/1.0 454 Session Not Found
CSeq: 7
Date: Mon, Sep 22 2014 15:08:54 GMT


Regards,
Jonathan
Ross Finlayson
2014-09-22 19:06:00 UTC
Permalink
I can indeed see this running the proxy server with -V. It is not the case of a TCP connection being misinterpreted as valid. It is the case of when the proxy server retries an OPTIONS command, it sees the TCP connection has gone away, reconnects and receives a valid response, it doesn't realise this is because the server has restarted.
OK, now I understand what's going on.

The problem here is that the proxy server can't tell that the server has restarted - merely from the fact that it needed to reopen the TCP connection in order to send an "OPTIONS" command. It's perfectly valid (although admittedly unusual) for a server to close the RTSP (TCP) connection while the RTP (UDP) stream is ongoing. In this case, the server's client (in this case, the proxy server) would need to reopen the RTSP (TCP) connection in order to send the next RTSP command (in this case, "OPTIONS"). Our code does this just fine. The problem is that - because this next RTSP command ("OPTIONS") completes without error, the proxy server can't tell that the server has restarted.
Perhaps rather than using OPTIONS something that sends a session id should be sent and error responses checked for?
The "OPTIONS" command sent by the proxy server *does* contain a session id (when a session has been established - i.e., when there's at least one front-end client). Unfortunately, the server (after restarting) didn't see this incoming "OPTIONS" command as being erroneous - because it didn't look at the "Session:" id.

I've now installed a new version (2014.09.22) of the "LIVE555 Streaming Media" code that changes the way in which the RTSP server handles incoming "OPTIONS" commands that contain a "Session:" header. If the specified session id refers to a session that does not exist, the server now responds with a "Session not found" error. (If the session id is valid, or is not present at all in the "OPTIONS" request, then the request is handled as normal.)

If you rebuild your 'back-end' server using this new version of the code, then the proxy server will now be able to better handle the case when it restarts.


Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

Loading...