RTSP is Real Time Streaming Protocol, as defined in RFC 2326. There's more information at http://www.rtsp.org/
RTSP can use either TCP or UDP and by default uses port 554. It is also commonly tunneled through HTTP, for example by QuickTime, and so will also be found on port 80.
l7-filter uses the rtsp pattern for normal RTSP and the http-rtsp pattern for RTSP tunneled through HTTP. These patterns have been tested and are believed to work well.