Color

Color

Postby Lyberta » 02 Jun 2019, 04:13

So I was watching a trash B-movie called "Ready Player One" and wanted to take a screenshot of it. As it was Rec.2020 file, I wanted to preserve original colors. What followed was a day of struggle and no success. But my mind was expanded and I really want to not make mistakes that most software and file formats did.

So for the purposes of this post I'm going to call pixel format with at least 10 bits per channel and at least Rec. 2020 color space HDR and pixel format with 8 bits per channel and less than Rec. 2020 color space LDR.

So.... pretty much nothing supports HDR. HDR data is extremely hard to work with as a result. But since even trash B-movies use HDR, only supporting LDR is shooting yourself in the foot.

From what I gathered so far, the only HDR usable raster images formats are:

  • OpenEXR
  • JPEG 2000 (optional)

Usable HDR vector image formats are:

  • None

That's horrible. In my own quest I've tried to export frames of that movie to OpenEXR using ffmpeg but ffmpeg doesn't have OpenEXR export. And when trying to export PNG it used incorrect tone mapping which produced wrong colors. Ugh.

Lessons learned:
  • LDR is dead. HDR is the only way. LDR is like ASCII, HDR is like Unicode.
  • BMP is useless
  • PNG is useless
  • JPG is useless
  • SVG is useless

So this topic is about how to handle colors correctly. Even before this quest I realized that the only correct way to represent colors in code is to use a floating point value for each channel:
{l Code}: {l Select All Code}
struct Color
{
    float red;
    float green;
    float blue;
    float alpha;
};


That gives nice 128 bits per pixel and also HDR calculation for free because values can exceed 1.0 so we can keep a lot of data without rounding errors or truncation. Now, of course, we want HDR color space too and we still want to load legacy formats that use legacy color spaces. So I guess we need a different type for each color space. This can be solved in C++ with templates:
{l Code}: {l Select All Code}
struct Rec2020 {};

struct sRGB {};

template <typename ColorSpace>
struct Color
{
   ...
};


Buuut, can we actually easily convert between color spaces? Is there an straightforward algorithm? What about non-RGB color models? What color space to use for a game engine? Do we need to hardcode it or let it be determined at runtime? So many questions...

Now for APIs. It looks like both OpenGL and Vulkan support these 32-bit floating point per channel buffers. However, what about displaying them? It looks like Vulkan got support of that as an extension just 3 months ago. Wait, extension? Really? I presume OpenGL is dead then I guess.

Ok, now - hardware. GPU-wise this is pretty simple. This article says you'll need at least Radeon RX 380 or Nvidia 900 series. Well, we don't care about Nvidia here because they destroyed Nouveau. AMD requires just proprietary firmware so it's not optimal but I'm more tolerant to proprietary firmware than proprietary drivers. The real kicker are HDR monitors. Since we're gamers here I only looked at gaming monitors and found prices to start from around 1800$. Yikes.

Software. I'm not sure if Mesa implements that Vulkan extension. Well, considering it was announced just 3 months ago, I guess not. I guess for now we need a tone mapping fallback in our graphics code. Speaking of tone mapping, I didn't even mention tone mapping. When I open that video file in VLC I get this in codec info:
Max. luminance: 4000 cd/m2
Min. luminance: 0050 cd/m2
Primary R: x=0.7080 y=0.2920
Primary G: x=0.1700 y=0.7970
Primary B: x=0.1310 y=0.0460
White point: x=0.3127 y=0.3290
MaxCLL: 725 cd/m2
MaxFALL: 162 cd/m2


These are runtime parameters so they must be stored somewhere. And I read HDR10+ can now alter luminance range on individual frames. So in order to convert pixel from one format to another you would need to take all of those parameters into account. Damn, it looks like working with color is very hard. That's why I've made this topic. I want to get colors right.
User avatar
Lyberta
 
Posts: 643
Joined: 19 Jun 2013, 10:45

Re: Color

Postby fluffrabbit » 02 Jun 2019, 15:15

Spielberg is an old man who relies on technicians to take care of his visuals. The technicians behind a big-budget film must use HDR now. It's gospel. I have a filmmaking background myself and I have never worked in HDR, and I suspect that open source tools aren't fully on that bandwagon yet because independents don't have the same distribution channels. A theater screen makes a digital image brighter and shows a lot more artifacts compared to an LCD monitor, but most people working on these tools only have LCD monitors.

I would like to add another HDR format to the list: Radiance HDR. This format is supported by stb_image and is very similar to OpenEXR. AFAIK ffmpeg does not support this format any better.

I presume OpenGL is dead then I guess.

You can render into an HDR framebuffer and then copy that memory to the client for display in a platform-specific way. Though not as performant as s/RGB, it's better than nothing if you really want to output HDR with OpenGL.

IMHO ffmpeg/libav sounds like a difficult way to get HDR information because the software is optimized for performance first and foremost, so it probably throws out extra color information as early in the pipeline as possible to save on memory bandwidth. If you can get the HDR color information somehow, I believe you can use stb_image to save it as OpenEXR or Radiance HDR.
fluffrabbit
 
Posts: 570
Joined: 11 Apr 2019, 11:17

Re: Color

Postby Lyberta » 02 Jun 2019, 18:18

fluffrabbit {l Wrote}:I would like to add another HDR format to the list: Radiance HDR. This format is supported by stb_image and is very similar to OpenEXR. AFAIK ffmpeg does not support this format any better.


Wikipedia {l Wrote}:It stores pixels as one byte each for RGB (red, green, and blue) values with a one byte shared exponent. Thus it stores four bytes per pixel.


Looks pretty unusable to me.
User avatar
Lyberta
 
Posts: 643
Joined: 19 Jun 2013, 10:45

Re: Color

Postby fluffrabbit » 02 Jun 2019, 20:13

tl;dr I'd give ffmpeg AV1 encoding followed by mp4box AV1->AVIF muxing a shot

That's to save space. stb_image and other tools will convert to and from HDR RGB floats.

There is also AVIF if quality is a concern. I would expect ffmpeg support to be good since it's based on a video codec, but I dunno about ffmpeg's HDR support considering a few years ago ffmpeg strictly dealt with 8-bit color channels.

EDIT: NOPE. Support for AVIF is garbage.

However, I can vouch for the efficacy of mp4box. The tried and true solution for many multimedia challenges on Linux is to chain multiple tools together.

AV1 support in ffmpeg is only available from 2019 distros onwards as the standard has only recently been stabilized (the development timescale was way too optimistic). So, assuming you're on Ubuntu 19.xx or something similar or using a PPA etc.:

{l Code}: {l Select All Code}
sudo apt install ffmpeg gpac

Then, encode the video as AV1 in HDR mode from -ss <wherever> of -t <less than 1 second>. Encoding guide here.

Then use mp4box from the gpac package to demux the first keyframe into an AVIF image file.
fluffrabbit
 
Posts: 570
Joined: 11 Apr 2019, 11:17

Re: Color

Postby Lyberta » 03 Jun 2019, 04:50

I'm pretty sure going from HEVC to AV1 would be lossy.
User avatar
Lyberta
 
Posts: 643
Joined: 19 Jun 2013, 10:45

Re: Color

Postby fluffrabbit » 03 Jun 2019, 10:04

Yes, going from anything to AV1 or JPEG2000 would be lossy. They're lossy formats, like h.265. OpenEXR is technically a lossless format, but there is colorspace conversion. It's kinda like how FLAC is a lossless format that doesn't 1:1 correlate with the sample format on an audio CD; it's lossless but there is still a numerical translation that inherently has a small loss. Though the loss would be greater going to AV1 due to lossy compression, the quality is still good and it would probably look better than JPEG2000.

If patents don't bother you, the only truly lossless conversion I can imagine would be demuxing h.265 keyframes into BPG or HEIF via mp4box.
fluffrabbit
 
Posts: 570
Joined: 11 Apr 2019, 11:17

Re: Color

Postby fluffrabbit » 03 Jun 2019, 11:45

MP4Box in Ubuntu (and probably other distros) is way outdated (2012). GPAC doesn't like to deal with the community because they're French and you know how the French are. They've got a huge backlog of bug reports, more than ffmpeg, so I guess the tool is more popular for whichever reason.

I have adapted the build instructions from here. Cloning the Git repo may take a while; it's 91 MB. The source files themselves take up 45 MB. I don't know why.

{l Code}: {l Select All Code}
git clone https://github.com/gpac/gpac.git


The build instructions are outdated, so even though I did my best to replace the referenced libraries with newer versions, it may not work 100%:

{l Code}: {l Select All Code}
sudo apt update
sudo apt install build-essential zlib1g-dev libfreetype6-dev libjpeg62-dev libpng-dev libopenjp2-7-dev libmad0-dev libfaad-dev libogg-dev libvorbis-dev libtheora-dev liba52-0.7.4-dev libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libavresample-dev libxv-dev x11proto-video-dev libgl1-mesa-dev x11proto-gl-dev linux-sound-base libxvidcore-dev libssl-dev libjack-dev libasound2-dev libpulse-dev libsdl1.2-dev dvb-apps libavcodec-extra libavdevice-dev libmozjs-52-dev


Then I tried building and installing with the *shudders* configure script. There is no CMake or scons.

{l Code}: {l Select All Code}
cd gpac
./configure
make
sudo make install


Half of the checks failed (big surprise there) but compilation worked perfectly. However, attempting to compose an image from a hevc results in failure.

{l Code}: {l Select All Code}
MP4Box -add-image "video.mkv":time=4841:primary -ab heix -new image.heic


I'm getting a lot of "does not start with sync marker". It's stressful using a tool that is so poorly documented. Supposedly this procedure is supposed to work.
fluffrabbit
 
Posts: 570
Joined: 11 Apr 2019, 11:17

Re: Color

Postby Lyberta » 03 Jun 2019, 19:53

fluffrabbit {l Wrote}:Yes, going from anything to AV1 or JPEG2000 would be lossy. They're lossy formats, like h.265. OpenEXR is technically a lossless format, but there is colorspace conversion. It's kinda like how FLAC is a lossless format that doesn't 1:1 correlate with the sample format on an audio CD; it's lossless but there is still a numerical translation that inherently has a small loss. Though the loss would be greater going to AV1 due to lossy compression, the quality is still good and it would probably look better than JPEG2000.

If patents don't bother you, the only truly lossless conversion I can imagine would be demuxing h.265 keyframes into BPG or HEIF via mp4box.


FLAC is lossless and can store 16 bit CDDA samples. A lot of torrents are published as .cue+.flac so you can burn a perfect copy of original CD. HEVC -> OpenEXR losslessness only depends if OpenEXR supports Rec. 2020 directly or have better colorspace that is a strict superset of Rec. 2020.

I thought that video programs can easily uncompress raw YCbCr in HEVC and then convert it to RGB and put into some HDR format (or no conversion if it supports YCbCr). And why would I bother with only demuxing keyframes?
User avatar
Lyberta
 
Posts: 643
Joined: 19 Jun 2013, 10:45

Re: Color

Postby fluffrabbit » 03 Jun 2019, 20:15

FLAC is lossless and can store 16 bit CDDA samples. A lot of torrents are published as .cue+.flac so you can burn a perfect copy of original CD.

I must have confused "signed" with "floating-point". Yeah, you're right.

HEVC -> OpenEXR losslessness only depends if OpenEXR supports Rec. 2020 directly or have better colorspace that is a strict superset of Rec. 2020.

True, but this is a hypothetical. I don't know about OpenEXR. Going straight from Rec. 2020 HEVC to Rec. 2020 EXR probably requires using libx265 and an EXR encoder directly. ffmpeg is a middle man that kludges everything into whatever pixel format it feels like.

video programs

There's your problem. ffmpeg is optimized for spraying images onto an RGB screen with the "gotta go fast" philosophy.

And why would I bother with only demuxing keyframes?

Because that may be the only way to do it with the standard open source command line AV toolset at this point in time. Sure, it would be great to take a Rec. 2020 image as rendered by a video decoder and put those Rec. 2020 pixels into an EXR. You might have to write software for that.
fluffrabbit
 
Posts: 570
Joined: 11 Apr 2019, 11:17

Re: Color

Postby Lyberta » 03 Jun 2019, 20:30

fluffrabbit {l Wrote}:Because that may be the only way to do it with the standard open source command line AV toolset at this point in time. Sure, it would be great to take a Rec. 2020 image as rendered by a video decoder and put those Rec. 2020 pixels into an EXR. You might have to write software for that.


Or wait which is what I am doing now.
User avatar
Lyberta
 
Posts: 643
Joined: 19 Jun 2013, 10:45

Who is online

Users browsing this forum: No registered users and 1 guest