Embedded video in SDL made simple

[bzt]

Hi All,

I got fed up how difficult it is to display a video in an SDL application. So I've come up with a solution, an stb-style header wrapper that starts a decoder thread and provides easily integratable data. Very small (ca. 300 SLoC in a single file) and GPL licensed. It relies on SDL2 mixer's header file (but only for one struct definition).

https://gitlab.com/bztsrc/sdlogv

There's an example fully featured OGG / OGV player in about 100 SLoC. Easy to follow, easy to understand code with lots and lots of comments.

Audio
The audio is provided in SDL2 mixer's format (Mix_Chunk), and the whole player is just a few lines using callbacks! You can use the usual mixer stuff with it (pause, resume, change volume, positioning effect etc.)

Audio integration example:

{l Code}: {l Select All Code}

static void callback(int channel)
{
    Mix_Chunk *audio = theora_audio(&ctx);
    if(audio) Mix_PlayChannel(channel, audio, 0);
}
Mix_ChannelFinished(callback);
callback(0);


Video
For the video, it is similarly simple. You just need a streaming texture that you render somewhere (same as you would do with any other textures). Then there's a function to get the next frame from dragon land, and another to update the texture with it when the time comes.

And video integration example:

{l Code}: {l Select All Code}

if(!video) video = theora_video(&ctx);
if(video && video->playms <= now) { theora_yuv2texture(video, texture, ctx.w, ctx.h); video = NULL; }

SDL_RenderCopy(renderer, texture, NULL, &rect);


Hope this will be useful to you. If GPL is an issue for someone, let me know and I'll reconsider dual licensing under MIT.

Cheer,
bzt

[dulsi]

Very nice. I've wondered about doing video in SDL2. I've never needed to do it so I never got around to investigating.

[bzt]

Very nice.

Thank you, please give a star to the repo :-)

I've wondered about doing video in SDL2. I've never needed to do it so I never got around to investigating.

It is always good to see a nice intro or a little scene before each level. Pretty common for commercial games. I guess most FOSS games haven't done this because playing a video was PITA up until now.

A little heads up, I've simplified the API, theora_video() and theora_yuv2texture() have been merged, and it now handles syncing audio / video properly with frame dropping if needed. Now playing video is really as simple as:

{l Code}: {l Select All Code}

theora_video(&ctx, texture);
SDL_RenderCopy(renderer, texture, NULL, &rect);


I've also added theora_playing() which returns true if the decoding thread is running or if it is done but there's still packets in the buffer to consume.

Cheers,
bzt

[smcameron]

Neat!

Perhaps this will give me the kick in the ass I need to add bridge-to-bridge video chat to Space Nerds in Space, like on Star Trek.

[bzt]

Neat!

Thanks!

Perhaps this will give me the kick in the ass I need to add bridge-to-bridge video chat to Space Nerds in Space, like on Star Trek.

I'm not so sure, encoding into theora can be time consuming creating a huge lag. But otherwise yes, you can definitely use this for a video chat.

Things to modify:
1. in theora_ctx replace FILE *f with a network socket
2. in theora_getdata() replace fread with a socket read
Voilá! Now you have a video chat receiver part! (You still have to write the recorder / sender part using libtheoraenc, see encoder_example.c)

If you decide to use a different encoding which is not as compressed as theora, but much faster to encode (like motion jpeg for example), then you would also need to modify theora_producer:
1. when you've received and decoded an audio chunk from the network stream, add it to ctx->chunk[ctx->ahead]
2. when you've received and decoded a video chunk from the network stream, add it to ctx->frame[ctx->vhead]
The rest of the code and logic can be left intact and it will work!

I'm a bit busy with my project ATM, so I don't think I'll have time in this year, but next year I will gladly help to implement a video chat which you can use in Space Nerds in Space (with theora or any other codec).

Cheers,
bzt

[smcameron]

I've not delved into video much before but I took a brief look at how to get video out of a camera a forked a project and made this thing: https://github.com/smcameron/webcam_viewer which just does the "capture from camera" part (and maybe prematurely the yuv to rgb part.) Does not do anything with audio.

BTW, I noticed you have a couple threads, a lot of "volatile", and no obvious locks... which screams red flags/data races/undefined behavior to me, but, maybe I'm missing something, lol.

As for the architecture for SNIS, it would have to be something like:

1. On the client, record audio and video, queue them up for encoding and transmission to server.
2. On the server, relay theora packets to destination client(s).
3. On the client receive and decode audio and video streams and queue the data for playback.

That is, not quite as simple as " in theora_ctx replace FILE *f with a network socket", as I have a queue in front of the socket, because there's other unrelated things going down that socket too. In theory, it's doable. But googling around trying to find people using theora for real-time encoding... not finding much, it all seems to be old stuff having to do with streaming movies from files on disk, etc., very little to nothing having to do with real-time encoding/streaming from a camera.

In any case, I made an issue to track adding video chat to SNIS: https://github.com/smcameron/space-nerd ... issues/315

Whether it ever happens or not... eh, maybe 30% chance it happens, 70% chance it doesn't.

[bzt]

I've not delved into video much before but I took a brief look at how to get video out of a camera a forked a project and made this thing: https://github.com/smcameron/webcam_viewer which just does the "capture from camera" part (and maybe prematurely the yuv to rgb part.) Does not do anything with audio.

That's great! SDL has a standardized interface for capturing audio.

BTW, I noticed you have a couple threads

Just one.

a lot of "volatile", and no obvious locks...

Yes, locks are evil and kill performance they must be avoided at all costs. It took me a great effort to come up with a completely lock-free solution.

which screams red flags/data races/undefined behavior to me, but, maybe I'm missing something, lol.

Yes, you are missing a very important part: under the hood a circular buffer structure is used, with one thread only producing (the theora decoder) and another only consuming (the theora_audio and theora_video functions called from your main loop). Separate threads modify separate pointers (indices in my implementation), no variable is written by multiple threads, so there can be no race condition. That's the very reason why I have chosen circular buffer as data structure: to avoid race conditions without locks. For further details, see Dijskra's producer - consumer synchronization without semaphores.

As for the architecture for SNIS, it would have to be something like:

1. On the client, record audio and video, queue them up for encoding and transmission to server.
2. On the server, relay theora packets to destination client(s).
3. On the client receive and decode audio and video streams and queue the data for playback.

That is, not quite as simple as " in theora_ctx replace FILE *f with a network socket",

Yes it is that simple for point 3., but as I've mentioned earlier that's just the video chat receiver part, you still have to implement the encoder and sender parts, points 1. and 2.

But googling around trying to find people using theora for real-time encoding...

Yeah, people use mjpeg for that, which isn't that efficient, produces worse quality, but can be encoded / decoded much faster without performance loss.

Btw, SNIS is a really cool game, I like it very much! Sadly it is a bit complicated for my boys, but maybe within a few years they grow enough to play together!

Cheers,
bzt

[nkirk01]

Hope this will be useful to you. If GPL is an issue for someone, let me know and I'll reconsider dual licensing under MIT.

Cheer,
bzt

Hi bzt,

A nice straight forward theora library of code. Well done!
It could prove useful to add into a project I am working on.

However, could you please also license it under MIT, as you have offered.

Thanks very much!
nkirk