Some send() errors are not treated as fatal but the connection gets shut
down regardless. When this happens, librtmp may send an FCunpublish
message which various services interpret as an "end of stream" message
and disable features like "disconnect protection". Instead, let's
explicitly close the socket so that the remote end is aware that this is
an unclean disconnect.
Replaces the usage of SO_SNDTIMEO with TCP_USER_TIMEOUT on Linux.
- Noted as more effective than SO_SNDTIMEO by multiple sources.
- Drops TCP connection on timeout instead of returning send() errors,
meaning more predictable reconnect timing and reconnect.
- Timeout period actually reflected in practice. SO_SNDTIMEO takes
double the timeout time than actually requested to time out on
unix, whereas WinSock actually waits the specified time.
When trying to update to mbedTLS 3.2.1, I ran into failures with RTMPS
output. After consulting an mbedTLS implementation example, I determined
that we were not setting up the SSL/TLS context config in the correct
order, causing the connection to fail. Performing the setup in the
recommended order fixes that.
This commit fixes what is arguably a long-winded series of previous
commits that have possibly caused just as many problems as they have
fixed. I'll spare the details, but basically, there's no reason that
any of the RTMP object should ever be used across socket sessions.
This provides a slight enhancement by removing the `RTMP_Init` call
in `rtmp_stream_create()`, since it effectively just initializes TLS
just for `try_connect` to deinitialize it before it is even used.
This also fixes the current `SO_RCVTIMEO` timeout functionality by
making sure that `RTMP_Reset` is called last.
This fixes a bug where the RTMP send thread can deadlock upon the
underlying TCP connection being broken. By introducing a send timeout,
this allows the thread to unblock and give up, triggering a reconnect
as normal. The correct solution to this problem would be to rewrite
librtmp with asynchronous IO, but that seems like something unlikely
to happen.
**Before**:
- Start stream in OBS
- Use tool (pfSense) to invalidate connection state
- OBS bitrate drops to 0
- Output does not respond to stop signals, and hangs for an undefined
amount of time (usually multiple minutes) before finally giving up
**After**:
- Start stream in OBS
- Use tool (pfSense) to invalidate connection state
- OBS bitrate drops to 0
- Output sits in blocked state for maximum of 8 seconds, then cleans
up and triggers the reconnect logic
Bug is caused by the internal connection variables not being reset on
reconnect, leading OBS to both be unable to parse valid packets from and
send valid packets to the remote end. This commit splits RTMP_Init off
into a new RTMP_Reset function, which resets these internal variables
without re-initing the rest of the library. The original RTMP_Init
calls the new function, perfectly preserving the old behaviour while
adding a new reset function to address the issue with.
Fixesobsproject/obs-studio#2865
Certain RTMP status descriptions can contain stream keys. We don't want
to log those normally, so log the description for unhandled status codes
as RTMP_LOGDEBUG. If someone needs to debug an RTMP server's return
messages, they can compile OBS with the RTMP Log Level set to
RTMP_LOGDEBUG.
Don't assume @setDataFrame when using an info packet. I'm guessing
librtmp did this because the only time this packet type was normally
used was when you use onMetaData.
RTMP authentication requires the curStreamIdx and nStreams be set
between the disconnect / reconnect that occurs in PublisherAuth. Now
that there's no good place to clean them up, zero the whole rtmp->Link
before starting a new stream.
Every addStream call would increment this counter. After merging the
mbedTLS fixes, we no longer have extraneous RTMP_Init calls which were
masking the prescence of this bug. This caused every stream after the
first stream to have the wrong channel index, and eventually OBS would
crash due to an out of bounds write if the counter reached
RTMP_MAX_STREAMS.
The report in https://github.com/obsproject/obs-studio/issues/2350
identified the issue as being caused by mbedtls not following symbolic
links, but it turns out the issue was the mbedtls_x509_crt_parse_path
return value which was already fixed in 4d89123c. So these changes are
no longer necessary.
This code is very old and seems to be non-functional in its current
state. The TLS support is also complicated to maintain across multiple
deprecated mbedtls functions.
Though this should now be very rare, it's more helpful than "Failed to
connect to server". Other TLS error codes are now also stored for future
use instead of copying them on a case by case basis.
Per mbedtls documentation, "If you share a context between threads, you
need to call these functions only from the main thread, at the beginning
and end of the context's lifetime.". OBS violated this since librtmp
uses a global context and it was allocated and freed in different
threads such as the auto config test.
This commit attaches the mbedtls context to an RTMP structure so there
is no more global state. It also fixes a rare double-free crash that
could occur if RTMP_TLS_Free was called twice (this happened in rare
situations such as the auto config running followed by a mode change
from Advanced to Simple).
mbedtls_x509_crt_parse_path returns a positive number if it partially
succeeds and a negative number on complete failure. This changes the
positive result to no longer error and prevent TLS connections (OBS
verifies all endpoints so having no CA chain prevents TLS).
mbedtls_md5_starts(), mbedtls_md5_update() and mbedtls_md5_finish()
have been marked deprecated since mbedtls version 2.7 and may go
away in the future.
These function have been superseded by versions with a return
value. They are suffixed with "_ret". We do not actually check
return values.
Also the header "mbedtls/net.h" has been superseded by
"mbedtls/net_sockets.h".
Some security layer libraries code path used by the rtmp output had a
not used variable and not used param on HMAC_finish macro that was
triggering warnings during compilation.
On 64bit systems, this check will always evaluate to false due to
SIZE_MAX type and triggers a compiler warning.
This both makes it clearer that its only needed on 32bit system and
clear the compiler warning.
This diff adds mbedTLS support to the obs-outputs plugin. PolarSSL and
mbedTLS have grown so different between 2015-or-so when libRTMP was
written, and now it's no longer feasible to just use the USE_POLARSSL
flag.
This commit adds a WITH_RTMPS tri-state CMake variable (auto/on/off),
set to "Auto" by default. "Auto" will use RTMPS if mbedTLS is found,
otherwise will disable RTMPS. "On" will make it require mbedTLS,
otherwise fails configuration, and "Off" disables RTMPS support
altogether.
Closesobsproject/obs-studio#1360
