Sometimes a particular project requires implementing an integration with the video source of a custom type. It could be because there’s a third party software system, or even a hardware, which has an integration SDK which allows to pull media data, but it’s not standard and it’s not available in viinex right away.
It’s still possible to have this integration with viinex, and this can be done without modifying viinex itself.
Let’s take a quick look at how this can be done.
First of all, let’s separate media sources which provide encoded media streams from those which send raw video — that is, un-encoded arrays of YUV or RGB pixels.
As for the first – there is an example available within the source code of viinex itself, it’s a media source plugin implementation for reading out the media data from files. This example is actually a part of viinex and is shipped with viinex, — but it can also be used as a reference implementation of such plugin. The source code is available on GitHub: https://github.com/viinex/vnxvideo/blob/master/src/FileVideoSource.cpp. We’ll discuss that example in a different post; for now, let’s concentrate on raw video sources.
One of the complexities about raw video sources is that the amount of data to be transferred is massive, when compared to encoded media, so sending that data between components over a socket or pipe is inefficient. The data should be copied as few times as possible. The common solution is to use a shared memory to provide access to the video frame data, along with a protocol to let clients know about the availability of the video frame. This idea is implemented in viinex with a so-called “local transport” component. There is a producer and consumer side with the local transport; the producer side may allocate the buffers for video frames and make the video source populate the data within those buffers. Once this is done, the transport informs the consumers that the frame is available. Consumers may do their job with the frame, and when it’s no longer needed by a consumer – the latter informs the producer that the frame is “freed”. The producer implements some kind of reference counting over the frame buffers, so that shared memory which is backing a video frame shared with a consumer is not accidentally modified while the consumer is processing that frame.
Now, speaking of the implementation of this approach within viinex, let’s recall that there is a “rawvideo” object which stands for a raw video source. Initially viinex could only pull video from DirectShow cameras or Video4Linux sources. Later on that object gained the functionality which allows getting the raw video frames from a local transport. Here’s the config example for such a setup:
{ "type": "rawvideo", "name": "raw1", "capture": { "type": "localtransport", "address": "TestRawSrc1" }, "encoder": { "type": "cpu", "quality": "best_quality", "profile": "baseline", "preset": "ultrafast", "dynamic": true } },
The viinex object type here is “rawvideo” and object name is “raw1”. Although it’s a raw source, — within viinex it’s as good as encoded video sources, that it – this object can be linked directly with streaming objects (webrtc and rtsp servers), media archive or recording controller, and so on. Another thing to notice – “capture.type” being set to “localtransport”, which indicates that this video grabber should consume video frames from viinex local transport. The value of “capture.address” is what we call the “rendezvous point” for the local transport: it’s a string which defines the names of IPC objects for the particular local transport instance. Those named IPC objects are shared memory segment, and either a named pipe on Windows, or UNIX domain socket on Linux.
This is what happens at viinex side; other than that – the rest of viinex’ setup depends on the application but there won’t be anything else specific to the nature of this video source. What about the producer side, then?
Here’s the gist which contains the full implementation of a test video source: https://gist.github.com/gzh/68d3aecbbb12aab923a3017593efbdfb.
It can be compiled with
g++ custom-video-source.cpp -o custom-video-source -lvnxvideo
(viinex needs to be installed on the host where this gets built). When the resulting binary is being run – it would create a local transport producer at rendezvous point “TestRawSrc1” and generate some test video, — the featured image of this post shows how it looks like; on the real video the rectangles slide over time and the colors change too. If viinex runs in parallel with this binary – viinex would act as local transport consumer, it’ll get the raw video frames, encode them upon demand, send for re-streaming and so on – depending on the actual config.
The most important fragments of the code of this test video source implementation are:
- Lines 79-88 – the local transport server gets created. Notice that WithShmAllocatorStr is just a wrapper over the function vnxvideo_with_shm_allocator_str which is exported from vnxvideo shared library. For that reason, one can’t use C++ exception handling to throw an exception from the lambda that we want to be called, which actually calls the vnxvideo_local_server_create function. This vnxvideo library exports C functions for the most part, to make it possible to use that library not only from C++.
- Line 95: vnxvideo_rawproc_set_format should be called whenever the color format or resolution of the frames being sent changes.
- Lines 101-109: a buffer is allocated for a frame in the shared memory pool. Just like with creation of local transport server, the function vnxvideo_raw_sample_allocate needs to be called within the vnxvideo_with_shm_allocator_ptr call, and the latter is wrapped with WithShmAllocatorPtr for slightly better readability.
- Lines 114-129: application figures out the addresses of the frame buffer by calling vnxvideo_raw_sample_get_data. Then this memory can be filled with the code specific to the video source. In this example, the sample image is being generated in lines 118-129: it fills the image in YUV 4:2:0 planar format (I420) plane by plane, line by line, pixel by pixel. For a real video source implementation there’ll be a call to an integration SDK, saying that it can populate the data directly into the specified buffer.
And that’s about it. It depends on the application needs how to actually organize the deployment; if there’s more than one video source, they may be co-located in the same process. Also, the lifecycle of that process is another question; one may find it convenient to manage its lifecycle from viinex by means of the “process” object. But these are side questions. What’s important is that this technique makes it possible to integrate arbitrary third party raw video sources with viinex, while keeping viinex itself unmodified and even not running any code in viinex’ address space, yet making sure that the data is being passed from the video source to viinex without additional overhead.