The last piece in the puzzle for doing hardware conversions
automatically is ensuring we only consider valid target formats for the
conversion. Although it is unintuitive, some vaapi drivers can expose a
different set of formats for uploads vs for conversions, and that is
the case on the Intel hardware I have here.
Before this change, we would use the upload target list, and our
selection algorithm would pick a format that doesn't work for
conversions, causing everything to fail. Whoops.
Successfully obtaining the conversion target format list is a bit of a
convoluted process, with only parts of it encapsulated by ffmpeg.
Specifically, ffmpeg understands the concept of hardware configurations
that can affect the constraints of a device, but does not define what
configurations are - that is left up to the specific hwdevice type.
In the case of vaapi, we need to create a config for the video
processing endpoint, and use that when querying for constraints.
I decided to encapsulate creation of the config as part of the hwdec
init process, so that the constraint query can be down in the
hwtransfer code in an opaque way. I don't know if any other hardware
will need this capability, but if so, we'll be able to account for it.
Then, when we look at probing, instead of checking for what formats
are supported for transfers, we use the results of the constraint query
with the conversion config. And as that config doesn't depend on the
source format, we only need to do it once.
Historically, we have not attempted to support hw->hw format conversion
in the autoconvert logic. If a user needed to do these kinds of
conversions, they needed to manually insert the hw format's conversion
filter manually (eg: scale_vaapi).
This was usually fine because the general rule is that any format
supported by the hardware can be used as well as any other. ie: You
would only need to do conversion if you have a specific goal in mind.
However, we now have two situations where we can find ourselves with a
hardware format being produced by a decoder that cannot be accepted by
a VO via hwdec-interop:
* dmabuf-wayland can only accept formats that the Wayland compositor
accepts. In the case of GNOME, it can only accept a handful of RGB
formats.
* When decoding via VAAPI on Intel hardware, some of the more unusual
video encodings (4:2:2, 10bit 4:4:4) lead to packed frame formats
which gpu-next cannot handle, causing rendering to fail.
In both these cases (at least when using VAAPI with dmabuf-wayland), if
we could detect the failure case and insert a `scale_vaapi` filter, we
could get successful output automatically. For `hwdec=drm`, there is
currently no conversion filter, so dmabuf-wayland is still out of luck
there.
The basic approach to implementing this is to detect the case where we
are evaluating a hardware format where the VO can accept the hardware
format itself, but may not accept the underlying sw format. In the
current code, we bypass autoconvert as soon as we see the hardware
format is compatible.
My first observation was that we actually have logic in autoconvert to
detect when the input sw format is not in the list of allowed sw
formats passed into the autoconverter. Unfortunately, we never populate
this list, and the way you would expect to do that is vo-query-format
returning sw format information, which it does not. We could define an
extended vo-query-format-2, but we'd still need to implement the
probing logic to fill it in.
On the other hand, we already have the probing logic in the hwupload
filter - and most recently I used that logic to implement conversion
on upload. So if we could leverage that, we could both detect when
hw->hw conversion is required, and pick the best target format.
This exercise is then primarily one of detecting when we are in this
case and letting that code run in a useful way. The hwupload filter is
a bit awkward to work with today, and so I refactored a bunch of the
set up code to actually make it more encapsulated. Now, instead of the
caller instantiating it and then triggering the probe, we probe on
creation and instantiate the correct underlying filter (hwupload vs
conversion) automatically.
This is not technically necessary, because we never touch the fd again
after passing to cuda, but having it there could lead to future code
accidentally using it.
All hwdecs should respect the probing flag and demote their lgoging to
verbose level, so that initialisation failures during probing do not
spam the user. I forgot to do this for the Vulkan hwdec.
The libplacebo sync abstraction is deprecated and we should be using
the more explicit Vulkan semaphore helpers instead. This means that
more of the book keeping moves to our side, but it's not too bad.
There are two main things going on here:
1. After a lot of back and forth, I decided to write the new code with
timeline semaphores to streamline things, and that also means all
the variables are separate - which makes the #ifdefs easier to read.
Which is important because:
2. While pl_sync owned the exported fd/handle, pl_vulkan_sem does not,
so we are responsible for managing them. That means reversing the
previous logic - we now can pass an original fd to CUDA and never
think about it again, while we have to clean up a Win32 Handle
because CUDA will not take ownership.
ffmpeg was previously allocating images for frames as the code size,
rather than the presentation one (1088 vs 1080 in the most common
example). Using the coded size when wrapping images for libplacebo
resulted in incorrect scaling from 1088 -> 1080, but even using the
presentation size wasn't perfect, as discussed in the original
commit.
However, ffmpeg has now been updated to use the presentation size for
the frame images, after discussions that concluded this must be done
because there is no way for a frame consumer to fix the dimensions
without copying the frame.
With that ffmpeg change, we can just use the normal layout information
like all the other hwdecs.
Vulkan Video Decoding has finally become a reality, as it's now
showing up in shipping drivers, and the ffmpeg support has been
merged.
With that in mind, this change introduces HW interop support for
ffmpeg Vulkan frames. The implementation is functionally complete - it
can display frames produced by hardware decoding, and it can work with
ffmpeg vulkan filters. There are still various caveats due to gaps and
bugs in drivers, so YMMV, as always.
Primary testing has been done on Intel, AMD, and nvidia hardware on
Linux with basic Windows testing on nvidia.
Notable caveats:
* Due to driver bugs, video decoding on nvidia does not work right now,
unless you use the Vulkan Beta driver. It can be worked around, but
requires ffmpeg changes that are not considered acceptable to merge.
* Even if those work-arounds are applied, Vulkan filters will not work
on video that was decoded by Vulkan, due to additional bugs in the
nvidia drivers. The filters do work correctly on content decoded some
other way, and then uploaded to Vulkan (eg: Decode with nvdec, upload
with --vf=format=vulkan)
* Vulkan filters can only be used with drivers that support
VK_EXT_descriptor_buffer which doesn't include Intel ANV as yet.
There is an MR outstanding for this.
* When dealing with 1080p content, there may be some visual distortion
in the bottom lines of frames due to chroma scaling incorporating the
extra hidden lines at the bottom of the frame (1080p content is
actually stored as 1088 lines), depending on the hardware/driver
combination and the scaling algorithm. This cannot be easily
addressed as the mechanical fix for it violates the Vulkan spec, and
probably requires a spec change to resolve properly.
All of these caveats will be fixed in either drivers or ffmpeg, and so
will not require mpv changes (unless something unexpected happens)
If you want to run on nvidia with the non-beta drivers, you can this
ffmpeg tree with the work-around patches:
* https://github.com/philipl/FFmpeg/tree/vulkan-nvidia-workarounds
We will need the full ra_ctx to be able to look up all the state
required to initialise an ffmpeg vulkan hwcontext, so pass let's
pass the ra_ctx instead of just the ra.
HEVC hardware decode with drm wasn't working on the RPi 4. Mpv would
report the image format (rpi4_8 for 8-bit and rpi4_10 for 10-bit) wasn't
supported. The change to hwdec_drmprime.c identifies these two formats
as NV12 because it functions exactly the same. The change to
dmabuf_interop_gl.c adds support for P030 which rpi4_10 uses. These
changes were tested on a Pi 4 with this fork of ffmpeg:
https://github.com/jc-kynesim/rpi-ffmpeg.
Signed-off-by: EmperorPenguin18 <60635017+EmperorPenguin18@users.noreply.github.com>
A longstanding pain point of the drm VOs is the relative lack of state
sharing. While drm_common does provide some sharing, it's far less than
other platforms like x11 or wayland. What we do here is essentially copy
them by creating a new vo_drm_state struct and using it in vo_drm and
context_drm_egl. Much of the functionality that was essentially
duplicated in both VOs/contexts is now reduced simple functions in
drm_common. The usage of the term 'kms' was also mostly eliminated since
this is libdrm nowadays from a userspace perspective.
This handle type was only needed for backwards compatibility with
windows 7. Dropping it allows us to simplify the code: there is no
longer a need for runtime checks, as the handle type can now be
statically assigned based on the platform.
The motivating usecase here, apart from code simplification, is a
desired switch to timeline semaphores, which (in the CUDA API) only
supports the non-KMT win32 handles.
It's worth pointing out that we need no runtime check for
IsWindows8OrGreater(), because the `export_caps.sync` check will already
fail on versions of windows not supporting PL_HANDLE_WIN32.
H264 hardware decode with v4l2m2m wasn't working on the RPi 4. Mpv would
report the image format (yuv420p) wasn't supported. The change to
hwdec_drmprime.c is to explicitly say that the format now is supported.
The change to dmabuf_interop_gl.c is to specify the colour format of the
planes before generating egl images. These changes were tested on a Pi 4
with this fork of ffmpeg: https://github.com/jc-kynesim/rpi-ffmpeg.
Signed-off-by: EmperorPenguin18 <60635017+EmperorPenguin18@users.noreply.github.com>
Wayland VO that can display images from either vaapi or drm hwdec
The PR adds the following changes:
1. a context_wldmabuf context with no gl dependencies
2. no-op ra_wldmabuf and dmabuf_interop_wldmabuf objects
no-op because there is no need to map/unmap the drmprime buffer,
and there is no need to manage any textures.
Tested on both x86_64 and rk3399 AArch64
This has been a long standing problem that I think was the root cause
for wm4's vaapi shitlist. As DRM explicit supports all possible rgba
orderings, but OpenGL only guarantees support for 'rgba8', something
has to happen to support import of dmabufs with non-native ordering.
Although undocumented, it appears that somewhere in the import on the
GL side, the ordering is swizzled to rgba - which makes sense as that's
the only way to be able to support importing all the different formats.
But we didn't know this, and we do our own swizzling based on the
original imgfmt - which matches the drm format. So we get double
swizzling and messed up colours.
As there is no way for us to express that the GL swizzle happened to
the rest of mpv, the easiest thing to do is fudge the format to the
natural ordering so the GL doesn't change anything, then our swizzling
will do the right thing in the end.
Note that this change doesn't handle 0rgb and 0bgr because they seem to
be affected by some additional feature/bug that is mangling the
ordering somewhere that equally affects Vulkan interop. That could be a
vaapi driver bug or something going on in ffmpeg.
The logic around trying to establish what formats are supported by
vaapi is confusing, and it turns out that we've been doing it wrong.
Up until now, we've been going through the list of decoding profile
endpoints and checking the formats declared in those profiles to build
our list.
However, the set of formats that the vaapi driver can actually support
is potentially a superset of those supported by any given profile
endpoint. This master list is exposed by libavutil via the
av_hwframe_transfer_get_formats() function, and so we should use that
list as our starting point.
Perhaps surprisingly, this change actually removes no code, because we
still want the logic that enumerates endpoints and finds supported
formats for endpoints. We need this because we must have at least one
known sw format to initialise the hwframe_ctx with.
Additionally, while in the general case,
av_hwframe_transfer_get_formats can return different formats depending
on what format you initialise the hwframe_ctx with, I happen to have
read the libavutil code, and it doesn't care, so we just need to call
it once, we know we'll get back all known formats.
In practice, with an Intel vaapi driver, this will result in us
supporting a handful of extra formats for hwuploads - particularly
yuv420p (so no need to convert to nv12 first) and various orderings
of rgb(a).
We've had some annoying names for interops, which we can't simply
rename because that would break config files and command lines. So we
need to put a little more effort in and add a concept of legacy names
that allow us to continue loading them, but with a warning.
The two I'm renaming here are:
* vaapi-egl -> vaapi (vaapi works with Vulkan too)
* drmprime-drm -> drmprime-overlay (actually describes what it does)
* cuda-nvdec -> cuda (cuda interop is not nvdec specific)
It turns out that it's generally more useful to look up hwdecs by image
format, rather than device type. In the situations where we need to
find one, we generally know the image format we're dealing with. Doing
this avoids us having to create mappings from image format to device
type.
The most significant part of this change is filling in the image format
for the various hw interops. There is a hw_imgfmt field today today, but
only a couple of the interops fill it in, and that seems to be because
we've never actually used this piece of metadata before. Well, now we
have a good use for it.
sfan5 found a few things after I pushed the change, so this fixes them.
* Use-after-free on drm_device_Path
* Not comparing render_fd against -1
* Not handling dup() errors
In the confusing landscape of hardware video decoding APIs, we have had
a long standing support gap for the v4l2 based APIs implemented for the
various SoCs from Rockship, Amlogic, Allwinner, etc. While VAAPI is the
defacto default for desktop GPUs, the developers who work on these SoCs
(who are not the vendors!) have preferred to implement kernel APIs
rather than maintain a userspace driver as VAAPI would require.
While there are two v4l2 APIs (m2m and requests), and multiple forks of
ffmpeg where support for those APIs languishes without reaching
upstream, we can at least say that these APIs export frames as DRMPrime
dmabufs, and that they use the ffmpeg drm hwcontext.
With those two constants, it is possible for us to write a
hwdec-interop without worrying about the mess underneath - for the most
part.
Accordingly, this change implements a hwdec-interop for any decoder
that produces frames as DRMPrime dmabufs. The bulk of the heavy
lifting is done by the dmabuf interop code we already had from
supporting vaapi, and which I refactored for reusability in a previous
set of changes.
When we combine that with the fact that we can't probe for supported
formats, the new code in this change is pretty simple.
This change also includes the hwcontext_fns that are required for us to
be able to configure the hwcontext used by `hwdec=drm-copy`. This is
technically unrelated, but it seemed a good time to fill this gap.
From a testing perspective, I have directly tested on a RockPRO64,
while others have tested with different flavours of Rockchip and on
Amlogic, providing m2m coverage.
I have some other SoCs that I need to spin up to test with, but I don't
expect big surprises, and when we inevitably need to account for new
special cases down the line, we can do so - we won't be able to support
every possible configuration blindly.
I already added the equivalent logic for dmabuf_interop_pl previously
but I skipped the GL support because importing dmabufs into GL requires
explicitly providing the DRM format, and if you are taking a
multi-plane format and trying to treat each plane as a separate layer,
you need to come up with a DRM format for each synthetic layer.
But my initial testing has shown that the RockPRO64 board I've got to
work on drmprime hwdec will only produce NV12 in a single layer multi
plane format, and it doesn't have Vulkan support, so I have had to
tackle the GL multi-plane problem.
To that end, this change introduces the infrastructure to provide new
formats for synthetic layers. We only have lookup code for NV12 and
P010 as these were the only ones I could test.
Annoyingly, libva and libdrm use different structs to describe dmabufs
and if we are going to support drmprime, we must pick one format and do
some shuffling in the other case.
I've decided to use AVDRMFrameDescriptor as our internal format as this
removes the libva dependency from dmabuf_interop. That means that the
future drmprime hwdec will be able to populate it directly and the
existing hwdec_vaapi needs to copy the struct members around, but
that's cheap and not a concern.
With the files renamed, we can now disentangle the shared private
struct between the interops and hwdec_vaapi. We need this separation
to allow the future drmprime hwdec to use the interops.
This is the first in a series of changes that will introduce a drmprime
hwdec. As our vaapi hwdec is based around exporting surfaces as
drmprime dmabufs, we've actually got a lot of useful code already in
place in the GL/PL interops. I'm going to reorganise and adjust this
code to make the interops usable with the new hwdec as well.
The first step is to rename the files and functions. There are no
functional or other changes here. They will come next.
This is somewhat academic for now, as we explicitly ask for separate
layers and the scenarios where multi-plane images are required also use
complex formats that cannot be decomposed after the fact, but
nevertheless it is possible for us to consume simple multi-plane
images where there is one layer with n planes instead of n layers with
one plane each.
In these cases, we just treat the planes the same as we would if they
were each in a separate layer and everything works out.
It ought to be possible to make this work for OpenGL but I couldn't
wrap my head around how to provide the right DRM fourcc when
pretending a plane is a layer by itself. So I've left that
unimplemented.
When I introduced the concept of lazy loading of hwdecs by img format,
I did not propagate the probing flag correctly, leading to the new
normal loading path not runnng with probing set, meaning that any
errors would show up, creating unnecessary noise.
This change fixes this regression.
Our logging here today is very poor. We don't make it clear what
formats we are probing, or even that a certain format failed in most
cases. In the case where we do log the error, we don't make it clear
which format it was that failed.
The end result is that we have no idea what the possible and final
format spaces are, which makes it very hard to debug whether things are
working correctly, or to work on supporting additional formats.
We don't need to hold on to buffers longer than necessary. Doesn't
matter for vo_gpu but greatly matters for vo_gpu_next, since it persists
hwdec mapped textures for longer periods.
Unfortunately, only provides benefits for hwdecs which do explicit
copies in their decode path, which currently just means cuda and
d3d11va.
There's really nothing vulkan-specific about this hwdec wrapper, and it
actually works perfectly fine with an OpenGL-based ra_pl. This is not
hugely important at the time, but I still think it makes sense in case
we ever decide to make vo_gpu_next wrap OpenGL contexts to ra_pl instead
of exposing the underlying ra_gl.
Easiest way is by just using a designated struct initializer. If we
don't, `p->images` ends up inheriting random data, which leaks into e.g.
eglDestroyImageKHR.
It's a small miracle this never blew up before. Or maybe it did. Who
knows.
While testing support for the exotic formats used by Intel vaapi for
4:2:2 and 4:4:4 surfaces, I realised that we were enumerating all
endpoints and checking formats for them. The problem with this is
that decoding (VLD) endpoints are only a subset of what vaapi exposes.
All the encoding endpoints are there too, and there is also the
None profile that we don't care about, but which generates ffmpeg
warnings if you try and examine it.
So, let's only look at VLD endpoints. This will speed things up a
little bit and make the logging less noisy.
