vo_gpu: change --tone-mapping-desaturate algorithm

Comparing mpv's implementation against the ACES ODR reference samples
and algorithms, it seems like they're happy desaturating highlights
_way_ more aggressively than mpv currently does. And indeed, looking at
some example clips like The Redwoods (which is actually well-mastered),
the current desaturation produces unnatural-looking brightness fringes
where the sky meets the treeline.

Adjust the algorithm to make it apply to a much larger, more gradual
brightness region; and change the interpretation of the parameter. As a
bonus, the new parameter is actually sanely scaled (higher values = more
desaturation). Also, make it scale based on the signal level instead of
the luminance, to avoid under-desaturating bright blues.

DOCS/man/options.rst

@@ -4896,14 +4896,15 @@ The following video options are currently all specific to ``--vo=gpu`` and
     some drivers, so enable at your own risk.
 
 ``--tone-mapping-desaturate=<value>``
-    Apply desaturation for highlights that exceed this level of brightness. The
-    higher the parameter, the more color information will be preserved. This
-    setting helps prevent unnaturally blown-out colors for super-highlights, by
-    (smoothly) turning into white instead. This makes images feel more natural,
-    at the cost of reducing information about out-of-range colors.
+    Apply desaturation for highlights. The parameter essentially controls the
+    steepness of the desaturation curve. The higher the parameter, the more
+    aggressively colors will be desaturated. This setting helps prevent
+    unnaturally blown-out colors for super-highlights, by (smoothly) turning
+    into white instead. This makes images feel more natural, at the cost of
+    reducing information about out-of-range colors.
 
-    The default of 2.0 is somewhat conservative and will mostly just apply to
-    skies or directly sunlit surfaces. A setting of 0.0 disables this option.
+    The default of 1.0 provides a good balance that roughly matches the look
+    and feel of the ACES ODT curves. A setting of 0.0 disables this option.
 
 ``--gamut-warning``
     If enabled, mpv will mark all clipped/out-of-gamut pixels that exceed a

video/out/gpu/video.c

@@ -311,7 +311,7 @@ static const struct gl_video_opts gl_video_opts_def = {
     .gamma = 1.0f,
     .tone_mapping = TONE_MAPPING_MOBIUS,
     .tone_mapping_param = NAN,
-    .tone_mapping_desat = 2.0,
+    .tone_mapping_desat = 1.0,
     .early_flush = -1,
 };
 

video/out/gpu/video_shaders.c

@@ -565,18 +565,19 @@ static void pass_tone_map(struct gl_shader_cache *sc, float ref_peak,
 {
     GLSLF("// HDR tone mapping\n");
 
-    // Desaturate the color using a coefficient dependent on the luminance
-    GLSL(float luma = dot(dst_luma, color.rgb);)
-    if (desat > 0) {
-        GLSLF("float overbright = max(luma - %f, 1e-6) / max(luma, 1e-6);\n", desat);
-        GLSL(color.rgb = mix(color.rgb, vec3(luma), overbright);)
-    }
-
     // To prevent discoloration due to out-of-bounds clipping, we need to make
     // sure to reduce the value range as far as necessary to keep the entire
     // signal in range, so tone map based on the brightest component.
     GLSL(float sig = max(max(color.r, color.g), color.b);)
-    GLSL(float sig_orig = sig;)
+
+    // Desaturate the color using a coefficient dependent on the signal
+    if (desat > 0) {
+        GLSL(float luma = dot(dst_luma, color.rgb);)
+        GLSL(float coeff = max(sig - 0.18, 1e-6) / max(sig, 1e-6););
+        GLSLF("coeff = pow(coeff, %f);\n", 10.0 / desat);
+        GLSL(color.rgb = mix(color.rgb, vec3(luma), coeff);)
+        GLSL(sig = mix(sig, luma, coeff);) // also make sure to update `sig`
+    }
 
     if (!ref_peak) {
         // For performance, we want to do as few atomic operations on global
@@ -612,6 +613,7 @@ static void pass_tone_map(struct gl_shader_cache *sc, float ref_peak,
         GLSLHF("const float sig_peak = %f;\n", ref_peak);
     }
 
+    GLSL(float sig_orig = sig;)
     switch (algo) {
     case TONE_MAPPING_CLIP:
         GLSLF("sig = %f * sig;\n", isnan(param) ? 1.0 : param);