AMD AMF HQVBR cannot be used in any capacity

[pbaykalov]

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

32.0.4

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/RlBCrw9m5BncyZNI

OBS Studio Crash Log URL

No response

Expected Behavior

Starting from clean portable OBS and Radeon GPU
using just this settings should produce recorded files which do not exceed 6 Mb/s bitrate in any scenes.

Current Behavior

Bitrate is basically unlimited with any moving scene with lots of details (reaching 100-200 Mb/s).

Steps to Reproduce

1. Clean OBS
2. Set values as in picture
3. Record any detailed modern game/video like this: https://www.youtube.com/watch?v=nSE38xjMLqE

Anything else we should know?

AMD AMF rate control is described here:
https://github.com/GPUOpen-LibrariesAndSDKs/AMF/wiki/Rate%20Control%20Methods
https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_Video_Encode_API.md

It is not not mentioned anywhere that QP properties are used in HQVBR but in fact they are.
QP is set by default here:

obs-studio/plugins/obs-ffmpeg/texture-amf.cpp

Line 1167 in f4aaa5f

obs_data_set_default_int(settings, "cqp", 20);

As I understand it, it is then used here:

obs-studio/plugins/obs-ffmpeg/texture-amf.cpp

Line 1526 in f4aaa5f

amf_avc_update_data(enc, rc, bitrate * 1000, qp);

and because the value set as "default" then it is always used even though my recordEncoder.json does not contain it.
Consequentially, changing "Rate control" to CQP, modifying the value to "50" and changing back to HQVBR changes output bitrate: it becomes very small.

Unfortunately I cannot prove that I am right (and I am not sure about it anymore since I tried some most obvious workarounds after posting the issue and none work) because there is a literal check for what the RC is:
https://github.com/obsproject/obs-studio/blob/f4aaa5f0417c5ec40a3799551e125129fce1e007/plugins/obs-ffmpeg/texture-amf.cpp#L1324C12-L1324C61
but it does not use enc->amf_encoder->ClearProperty either which means that if an encoder ever gets QP values they are never unset afterwards.

I expect it to appy to all AMF codecs. There is also seems to be no workaround because enc->amf_encoder->ClearProperty cannot be called externally.

AMF HQVBR behaviour can also be demonstrated by these FFMPEG commands:
ffmpeg -i 'D:\docs\OBS\static pic.mkv' -c:v h264_amf -rc hqvbr -g 120 -b 10000000 output3.mp4 - expected behaviour
ffmpeg -i 'D:\docs\OBS\static pic.mkv' -c:v h264_amf -rc hqvbr -qmin 50 -g 120 -b 10000000 output3.mp4 - bitrate is overridden, output is small
ffmpeg -i 'D:\docs\OBS\static pic.mkv' -c:v h264_amf -rc hqvbr -qmax 10 -g 120 -b 10000000 output3.mp4 - bitrate is overridden, output is huge

As I understand it, this is potentially not the only problem with reusing parameters which should not be used. Ideally, all parameters should be cleared if they are not explicitly set from UI.

A quick workaround would be to at least make it possible to unset values from the "advances options" like this:
MinQP= which should call ClearProperty internally.

[pbaykalov]

I just learned that there is no ClearProperty in any shape or form. Which means that probably default values should be cached and restored when needed.

[pbaykalov]

I also tested whether most simple workaround would work and somehow I am still getting CQP behaviour.
pbaykalov@b32da5b
builds:
https://github.com/pbaykalov/obs-studio/actions/runs/20824813355

[pbaykalov]

I actually found it! OBS sets RateControlPreanalysisEnable=True for all rate control modes and it definitely breaks HQVBR.
Setting RateControlPreanalysisEnable=False through advanced options restores the correct behavior.
It's fun that this line is 4 years old what means that nobody ever used AMD H264 HQVBR in OBS for recording ever except me.
https://github.com/obsproject/obs-studio/blame/902f6d17e5a65742d16baf3cd30032979866dae7/plugins/obs-ffmpeg/texture-amf.cpp#L1616C24-L1616C40

Not sure if this correct AMF behaviour.