When a scene is duplicated the filters on the scene were not copied to
the new scene. This causes that a temporary copy of a scene renders
differently in the program than in the preview when using studio mode.
(Note: This commit also modifies coreaudio-encoder, win-capture, and
win-mf modules)
This reduces logging to the user's log file. Most of the things
specified are not useful for examining log files, and make reading log
files more painful.
The things that are useful to log should be up to the front-end to
implement. The core and core plugins should have minimal mandatory
logging.
The active_refs and show_refs variable would only increment/decrement
their children if their values were 1 and 0, which means that in the
case of scenes within scenes, sub-sources of scenes within scenes would
end up having the wrong ref values.
When using GPU conversion for 4:2:0 frames on async video sources, it
would create a texture bigger than necessary and try to copy too much
data from the frame, resulting in a crash.
When a transition is a sub-source of another source, it would not call
the transition's active source enum function, meaning that any sources
the transition had would not increment their active/showing refs (it
would only be called when activating the transition directly before).
That would result in negative/invalid active/showing refs on its
sub-sources, causing them to become permanently active/inactive and/or
permanently showing/hidden.
(Note: this commit also modifies the obs-filters and test-input modules)
Changes the obs_source_process_filter_begin return type so that it
returns true/false to indicate that filter processing should or should
not continue (for example if the filter is bypassed or if there's some
other sort of issue that causes the filtering to fail)
The source shouldn't be inserted into obs->data.first_audio_source until it's
fully initialized, or other threads will access source->control and
dereference an uninitialized pointer.
This patch fixes a specific crash where if the user named a filter the
same name as an input source that already existed in the system, scene
item loading code could find the filter with the same name instead of
the source, and mistakenly use it as the scene item's source directly.
This would cause a crash when trying to render that filter as a regular
source.
Marking filters as private is a temporary and simple workaround to the
solution. Filters are currently not meant to be found via the main
enumeration/search functions, which is a design flaw (lack of
consistency). In future major API revisions of libobs, filters should
be reworked to act as sources, with the sources they filter as
sub-sources ideally.
Additionally, the concept of "private context objects" and "primary
lists of context objects" in the back-end should probably also be
removed, allowing the font-end (or optional separate API layers) to
control all primary lists of obs context objects. These minor issues
that occur ultimately stem from API design flaws which need to be
corrected.
This crash happened when a filter was mistakenly used as a regular
source due to an unrelated bug in filter code and scene loading code.
The filter and the source it belongs to both had the same names, and the
source loading code found the filter and mistakenly used it as the
source instead of the actual source with the same name.
(Note: Also modified the obs-ffmpeg plugin module)
Allows the ability for frame data to pass 8-bit grayscale images (Y800
color format).
Closesjp9000/obs-studio#515
Adds deinterlacing API functions. Both standard and 2x variants are
supported. Deinterlacing is set via obs_source_set_deinterlace_mode and
obs_source_set_deinterlace_field_order.
This was implemented in to the core itself because deinterlacing should
happen before effect filters are processed, but after async filters are
processed. If this were added as a filter, there is the possibility
that a different filter is processed before deinterlacing, which could
mess with the result. It was also a bit easier to implement this way
due to the fact that that deinterlacing may need to have access to the
previous async frame.
Effects were split in to separate files to reduce load time (especially
for yadif shaders which take a significant amount of time to compile).
Instead of just updating the async texture variables directly in the
source, allow the ability to pass the async texture variables via
function parameters to allow the ability to parse more than one frame to
more than one texture.
This code is primarily intended to be used to upload/convert the
"previous" async frame for the deinterlacer (if necessary).
(Note: This commit also modifies obs-filters and text-freetype2)
This simplifies writing of effects. DrawMatrix is no longer necessary
because there are no sources that require drawing with a color matrix
other than async sources, and async sources are automatically processed
and don't defer their initial render stage to filters.
Under certain circumstances, the timing_adjust variable would cause line
1161 to continually trigger over and over again. The "loop detection"
code incorrectly made it so that any timestamp that was just simply
below the expected value would be seen as a jump. After that, the
timing_adjust variable would be set for the frame again, and then the
audio would see it as a jump again after that, and those two things
would continue endlessly. This would cause stuttering particularly with
certain devices (particularly elgato/lgp/hdpvr) where the audio/video
data are decoded and sent at varying/different/unpredictable times.
To fix this issue, it should not detect values below as jumps, but
instead should only do it for values that exceed the MAX_TS_VAR (maximum
timestamp variance) value.
If obs_source::audio_ts is set to 0 (such as by discard_if_stopped in
obs-audio.c), but the push_back variable in the source_output_audio_data
function in obs-source.c was being set to true (meaning it's within the
seamless audio smoothing threshold), it would cause it to never reset
the obs_source::audio_ts value, and thus all audio data from the source
would become perpetually ignored by the audio subsystem until there was
finally some sort of timestamp jump that caused it to call
source_output_audio_place, and thus reset obs_source::audio_ts.
obs_source::audio_ts is only reset in source_output_audio_place, not in
source_output_audio_push_back, so the most simple solution is to just
call source_output_audio_push_back is obs_source::audio_ts is 0.
There's technically no need to clear the audio data here, nor is there
any need to try to trick the timestamp in to a different position. It
can simple just reset the audio timing.
Prevents a possible case where audio data might be deleted when it's not
necessary to delete any.
This variable is used to detect whether audio has stopped -- if audio
stops, it detects that no new data is coming in, and resets the audio
position so that it eliminates the chance of causing the audio buffering
to go haywire if audio starts up again. However, this variable was not
being reset every time the value changes, which it should.
The seamless audio looping code would erroneously trigger for things
that weren't loops, causing the audio data to continually push back and
ignore timestamps, thus going out of sync.
There does need to be loop handling code, but due to the fact that other
things may need to trigger this code, it's best just to clear the audio
data and start from a fresh sync point. Unfortunately for the case of
loops, this means the window in which audio data loops and video frames
loop need to be muted.
This fixes an age-old issue where audio samples could be lost or audio
could temporarily go out of sync in the case of looping videos. When
audio/video data is looping, there's a window between when the audio
data resets its timestamp value and when the video data resets its
timestamp value. This method simply pushes back the audio data while in
that window and does not modify sync, and when it detects that its out
of the loop window it simply forces a resync of the audio data in the
circular buffer.
This ensures that minimal audio data is lost in the loop process, and
minimizes the likelihood of any sort of sync issues associated with
looping.
Instead of applying the resampler offset right away (to each audio
packet), apply the resampler offset when the timestamps are converted to
system timestamps. This fixes an issue where if audio timestamps reset
to 0 (for whatever reason), the offset would cause the timestamp to go
in to the negative.
Mostly only used for transitions with the intention of automatically
creating transitions which don't require configuration, returns whether
the source has any properties or not (whether it's configurable)
(Note: This commit also modifies UI)
Instead of using signals, use designated callback lists for audio
capture and audio control helpers. Signals aren't suitable here due to
the fact that signals aren't meant for things that happen every frame or
things that happen every time audio/video is received. Also prevents
audio from being allocated every time these functions are called due to
the calldata structure.
Transition sources are implemented by registering a source type as
OBS_SOURCE_TYPE_TRANSITION. They're automatically marked as video
composite sources, and video_render/audio_render callbacks must be set
when registering the source. get_width and get_height callbacks are
unused for these types of sources, as transitions automatically handle
width/height behind the scenes with the transition settings.
In the video_render callback, the helper function
obs_transition_video_render is used to assist in automatically
processing and rendering the audio. A render callback is passed to the
function, which in turn passes to/from textures that are automatically
rendered in the back-end.
Similarly, in the audio_render callback, the helper function
obs_transition_audio_render is used to assist in automatically
processing and rendering the audio. Two mix callbacks are used to
handle how the source/destination sources are mixed together. To ensure
the best possible quality, audio processing is per-sample.
Transitions can be set to automatically resize, or they can be set to
have a fixed size. Sources within transitions can be made to scale to
the transition size (with or without aspect ratio), or to not scale
unless they're bigger than the transition. They can have a specific
alignment within the transition, or they just default to top-left.
These features are implemented for the purpose of extending transitions
to also act as "switch" sources later, where you can switch to/from two
different sources using the transition animation.
Planned (but not yet implemented and lower priority) features:
- "Switch" transitions which allow the ability to switch back and forth
between two sources with a transitioning animation without discarding
the references
- Easing options to allow the option to transition with a bezier or
custom curve
- Manual transitioning to allow the front-end/user to manually control
the transition offset
(Note: test and UI are also modified by this commit)
API Changed (removed "enum obs_source_type type" parameter):
-------------------------
obs_source_get_display_name
obs_source_create
obs_get_source_output_flags
obs_get_source_defaults
obs_get_source_properties
Removes the "type" parameter from these functions. The "type" parameter
really doesn't serve much of a purpose being a parameter in any of these
cases, the type is just to indicate what it's used for.
The new audio subsystem fixes two issues:
- First Primary issue it fixes is the ability for parent sources to
intercept the audio of child sources, and do custom processing on
them. The main reason for this was the ability to do custom
cross-fading in transitions, but it's also useful for things such as
side-chain effects, applying audio effects to entire scenes, applying
scene-specific audio filters on sub-sources, and other such
possibilities.
- The secondary issue that needed fixing was audio buffering.
Previously, audio buffering was always a fixed buffer size, so it
would always have exactly a certain number of milliseconds of audio
buffering (and thus output delay). Instead, it now dynamically
increases audio buffering only as necessary, minimizing output delay,
and removing the need for users to have to worry about an audio
buffering setting.
The new design makes it so that audio from the leaves of the scene graph
flow to the root nodes, and can be intercepted by parent sources. Each
audio source handles its own buffering, and each audio tick a specific
number of audio frames are popped from the front of the circular buffer
on each audio source. Composite sources (such as scenes) can access the
audio for child sources and do custom processing or mixing on that
audio. Composite sources use the audio_render callback of sources to do
synchronous or deferred audio processing per audio tick. Things like
scenes now mix audio from their sub-sources.
(Note: This commit breaks libobs compilation. Skip if bisecting)
Adds a "composite" source type which is used for sources that composite
one or more sub-sources. The audio_render callback is called for
composite sources to allow those types of sources to do custom
processing of the audio of its sub-sources.
(Note: This commit breaks libobs compilation. Skip if bisecting)
This variable is somewhat redundant. Volume is already known/accessible
to front-ends.
(Note: This commit breaks libobs compilation. Skip if bisecting)
Removes audio lines and stores the circular buffer for the audio on the
source itself.
(Note: This commit breaks libobs compilation. Skip if bisecting)
The mixers that a source was assigned to were originally stored in the
audio line. This will store it in the sources themselves instead.
Originally this was programmed to call the recursive height/width
functions if the source type was an input with the intention of not
calling it on filters, but instead of doing that just program it to do
just that: only call the recursive height/width functions if it's not a
filter.
This was originally used for calculating audio volume if transitions
were active, but transitions won't work that way so tracking the active
transitions is no longer needed.
Renames:
----------------------------------------
obs_source_add_child
obs_source_remove_child
obs_source_enum_sources
obs_source_enum_tree
obs_source_info::enum_sources
To:
----------------------------------------
obs_source_add_active_child
obs_source_remove_active_child
obs_source_enum_active_sources
obs_source_enum_active_tree
obs_source_info::enum_active_sources
These functions/callbacks had misleading names: they originally implied
any child sources, when they actually meant active child sources that
are being used to render video or audio. It's important that the
function names represent their actual purpose.
(Note: This commit breaks UI compilation. Skip if bisecting)
API Removed:
------------------------
obs_add_source
API Changed:
------------------------
obs_source_remove: Now just marks/signals a source for removal
The concept of "user sources" is flawed: it was something that the
front-end was forced to deal with if it wanted to automate source
saving/loading, and often it had to code around it. That's not how
saving/loading should work, a front-end should be allowed to manage
lists of sources in the way it explicitly chooses, and it should be able
to choose which sources it wants to save/load.
This shouldn't happen anymore because crop was fixed, but if a filter
returns 0x0 size and is invalid it shouldn't stop the filter chain.
Instead, it should just be skipped.
When an async video source is about to be rendered, the async texture
should be updated before any effect filtering occurs, rather than right
when it's about to render.
Fixes a few bugs:
- If the async texture hadn't drawn for its first time, and the source
has an effect filter, it would never end up rendering the first
frame due to the fact that it would fail on obs-source.c:2434 for the
first filter, causing it to never actually render the source, and thus
never get to a point in which it could call set_async_texture_size to
establish the async texture width/height for the first time.
- Any time the async texture size changed, it would only update the
async texture size at the end of the filter loop, which means that the
first frame after a size change would use the old size for the filters
rather than update to the new size right away.
To prevent confusion with the new obs_source_valid function which
displays a warning, rename the "source_valid" function to "data_valid"
to emphasize that it's checking for the validity of the internal data as
well as the source itself.
API removed:
--------------------
gs_effect_t *obs_get_default_effect(void);
gs_effect_t *obs_get_default_rect_effect(void);
gs_effect_t *obs_get_opaque_effect(void);
gs_effect_t *obs_get_solid_effect(void);
gs_effect_t *obs_get_bicubic_effect(void);
gs_effect_t *obs_get_lanczos_effect(void);
gs_effect_t *obs_get_bilinear_lowres_effect(void);
API added:
--------------------
gs_effect_t *obs_get_base_effect(enum obs_base_effect effect);
Summary:
--------------------
Combines multiple near-identical functions into a single function with
an enum parameter.
API changed from:
obs_source_info::get_name(void)
obs_output_info::get_name(void)
obs_encoder_info::get_name(void)
obs_service_info::get_name(void)
API changed to:
obs_source_info::get_name(void *type_data)
obs_output_info::get_name(void *type_data)
obs_encoder_info::get_name(void *type_data)
obs_service_info::get_name(void *type_data)
This allows the type data to be used when getting the name of the
object (useful for plugin wrappers primarily).
NOTE: Though a parameter was added, this is backward-compatible with
older plugins due to calling convention. The new parameter will simply
be ignored by older plugins, and the stack (if used) will be cleaned up
by the caller.
When an audio filter is applied to a video source that also has
accompanying audio, it would cause the video from the source to stop
rendering.
The original code this was from was to prevent audio-only sources from
rendering video, but I neglected to make sure that this would not apply
to filters, and thus when an audio filter is on a source with video, the
code would kill the video.
Due to the fact that async timestamps themselves can be susceptible to
minor jitter from certain types of inputs, increase the allowable jitter
compensation value to ensure that the rendered frame timing from async
video sources is always as close as possible to the compositor.
When the framerate of the source is the same as the framerate as the
compositor, this (combined with the fact that clamped video timing now
being used with async video frames) helps ensure that buffered async
video sources will sync up their rendering to the compositor as
accurately as possible despite jitter from the source's timestamps.
If there is no jitter in the source's timestamps then it'll always sync
up perfectly with the compositor, thanks to clamped video timing.
When playing back buffered async frames, this reduces the probability
that new frames will be missed/skipped due to jitter in the system
timestamps.
If a buffered async source is playing at the same framerate as the
compositor and there is no jitter in the async source's timestamps, then
the async source will play back perfectly in sync with the compositor
thanks to this change, ensuring that there's no skipped or missed frames
in video playback.
When buffering is enabled for an async video source, sometimes minor
drift in timestamps or unexpected delays to frames can cause frames to
slowly buffer more and more in memory, in some cases eventually causing
the system to run out of memory.
The circumstances in which this can happen seems to depend on both the
computer and the devices in use. So far, the only known circumstances
in which this happens are with heavily buffered devices, such as
hauppauge, where decoding can sometimes take too long and cause
continual frame playback delay, and thus continual buffering until
memory runs out. I've never been able to replicate it on any of my
machines however, even after hours of testing.
This patch is a precautionary measure that puts a hard limit on the
number of async frames that can be currently queued to prevent any case
where memory might continually build for whatever reason. If it goes
over the limit, it clears the cache to reset the buffering.
I had a user with this problem test this patch with success and positive
feedback, and the intervals between buffering resets were long to where
it wasn't even noticeable while streaming/recording.
Ideally when decoding frames (such as from those devices), frame
dropping should be used to ensure playback doesn't incur extra delay,
although this sort of hard limit on the frame cache should still be
implemented regardless just as a safety precaution. For DirectShow
encoded devices I should just switch to faruton's libff for decoding and
enable the frame dropping options. It would probably explain why no
one's ever reported it for the media source, and pretty much only from
DirectShow device usage.
Fixes a crash that could happen if any of the mutexes are used in the
create callback, or before the obs_source_init function is called.
I'm not sure how this function order slipped because it seems fairly
obvious that these mutexes should be created before the create callback.
Had this crash happen to me when creating a WASAPI output source, the
create callback of the WASAPI source creates a thread which outputs
audio, and that thread managed to call obs_source_output_audio before
the obs_source_init function was called, which in turn caused it to try
to use a null mutex.
When caching a new frame, keep a reference to the frame while copying to
ensure that the frame is not potentially destroyed for whatever reason
while that data is being copied.
The obs_source::async_reset_texture variable can cause a data race
between threads to occur because it could be set to true in one thread
then changed back to false in another thread. This could cause the
async texture to not update its size when it's supposed to, which can
cause a crash or corruption when copying data from a frame of a
differing size.
The solution to this is to:
- Delete the async_reset_texture variable, and make the
set_async_texture_size function change the texture size if the
async_width, async_height, or async_format variables differ from the
frame's width/height/format. Those variables are then only ever set
in the libobs graphics thread.
- Make the cache_video function use separate variables from other
functions to detect a change in size (due to the fact that the texture
size should only be resized in the libobs graphics thread). These
variables are async_cache_width, async_cache_height, and
async_cache_format, which are only be set in the thread that calls
obs_source_output_video.
How to replicate the data race:
- On OSX, use window capture on a textedit window, then continually
resize the textedit window.
Due to a bad 'if' expression, when a filter that is not last in the
chain is disabled or being bypassed, it ends up still calling the
filter's video processing function unintentionally.
This fix makes sure that it only calls the appropriate render functions
if the next filter target is the source, otherwise it will just call
obs_source_video_render to process the next filter in the chain.
How to replicate the bug:
1. Create two crop filters on the same source
2. Give each crop filter a different distinct value
3. Disable both crop filters
4. The image would still be cropped
This fixes an issue where cache frames would not free at all after
having been allocated with no upper limit on the cached frame size. If
cached frames go unused for a specific period of time, they are
deallocated and removed from the cache.
This is preferable to having an upper cache limit due to the potential
for async delay filtering.
Under certain circumstances the cache could be prone to growing too
large unintentionally. Setting a hard maximum limit should prevent
memory from growing if we suddenly get a lot of frames.
Async frames are only swapping when rendering, or when not visible.
This is a flawed design due to the fact that there are certain
circumstances where the source is neither visible nor currently
rendering.
This is what caused a memory leak when scene items were marked as
invisible, because if a source has an async child source and decides not
to render that source for whatever reason, the child source would not
process the async frames at all, and the cache would just grow.
To fix this, simply moving the async frame cycle to tick fixes the issue
due to the fact that tick is always called regardless of circumstance.
Filtering the video before it's output to the texture means that it
happens after all the processing on the timestamps and such of the video
data. This way, the video filter does not have to worry about what's
currently buffered, and it won't affect timing.
When OBS is shutting down, if for some reason the filter is destroyed
before the parent source is destroyed, it would try to remove itself
from the source, but it would decrement the reference and try to destroy
itself again while already in the process of destroying itself.
So, the solution was simply to make sure that if it's removing itself
from the source that it doesn't decrement its own reference.
