We’d like to share some thoughts and experience of authoring viinex configuration.
While viinex configuration format is straightforward, so that the config can be written manually, — as the size of the system grows, it can quickly become tedious and error-prone.
If you’re developing a large system and implement the code to generate viinex config anyway, — this won’t be an issue. However there can be corner cases where viinex gets integrated in an ad-hoc manner, there’ll likely be a few installations of this kind, however the size of the config in each case is, say, dozens of cameras.
We had faced an example of this kind recently, where the custom video analytics had to be deployed for a few dozens of video channels. The results coming from the analytics module had to be exported into a higher level data warehousing software, and this was done by means of viinex built-in scripts. However what about the configuration?
In that particular case the “configuration” was provided by customer in the form of a CSV file containing basic description of a video channel, along with RTSP URL, credentials to access the stream, geolocation, — some info which was required to be passed into the data warehouse.
After thinking over and trying different approaches we found a solution which seems suitable for similar cases, — that’s a Jsonnet language (https://jsonnet.org/). We were considering other choices, like bash, Python, and JavaScript (node) for this purpose, but Jsonnet seems to be more practical solution. Node or Python require an installation of a general purpose programming tool and infrastructure – if config needs to be edited and updated on the target host, this means this infrastructure needs to be installed there, which is either inconvenient or infeasible. Bash is available virtually everywhere, however as a programming language it’s not expressive enough, — one would have to write the templates of viinex config, almost ready for every particular case; it’ll likely become fragile with the need for changes any bigger than just parameter substitution.
Jsonnet, on the other hand, is a small programming language for transforming JSON documents. It’s a pure functional programming language (pure, like in Haskell: its functions don’t have side effects), and its standard library is limited, namely it does not have any IO, so can be safely installed on the target host, — and the implementation is just a binary executable (on Debian-originated linux it can also be installed from the apt repository). With that said – Jsonnet is a superset of JSON, so it can be really easy to start using. For simple templating one can start with re-using the existing viinex configurations, say, to re-use some parts of it, and then evolve the complexity of applied transformations upon the need.
Let’s consider how the simple configuration for a system with NVR-like functionality can be built with viinex using Jsonnet. Say we want the user input to be simple, and for that reason we’ll design a YAML input file format like follows:
creds:
- ['admin', '$ecretpassword!!1'] # default one goes first
- ['admin', '12345']
onvif:
- addr: "192.168.0.128"
id: fy
name: Front yard
desc: Front yard and street overview
rec: permanent
- addr: '192.168.0.125'
id: attic
name: Attic
rec: none
- addr: '192.168.0.111'
id: porch
name: Porch
desc: Front yard and porch
cred: 1
rec: motion
rtsp:
- url: "rtsp://admin:12345@192.168.32.121"
id: s1e3
- url: "rtsp://admin:12345@192.168.32.119"
id: s1e4
app:
metrics: true
rtspsrv: true
webrtc: true
websrv: true
Section `creds` is to avoid repeating the credentials when setting different cameras. Sections `onvif` and `rtsp` are for adding ONVIF cameras or RTSP video sources. Under each of these, `rec` specifies how the media should be recorded – permanently, by a motion detector, or if it’s not recorded at all. `cred` can be specified to provide credentials different from the default (references the index of credentials pair in `creds` section). The `app` section sets the flags of what should be available in the resulting config. That’s it. This is a very simplified view, however it can really be used to configure a system, so let’s take this format as an example.
Jsonnet can perfectly take YAML as input. One can create library files with helper functions written in Jsonnet, like follows:
mk_onvif: function (name, addr, auth, profile) {
type: "onvif",
name: name,
host: addr,
auth: auth,
profile: profile,
enable: ["video", "audio", "events", "ptz"],
rtpstats: true,
transport: ["udp"],
},
mk_recctl: function(name, prerecord, postrecord) {
type: "recctl",
name: "recctl_" + name,
prerecord: prerecord,
postrecord: postrecord
},
mk_rule_motion: function(name) {
type: "rule",
name: "rule_" + name,
filter: ["MotionAlarm"]
},
These are most elementary units – the functions which may take some parameters and produce an element (or several elements) of viinex configuration. Basically, the building blocks which are yet to be combined into the config accordingly to the application requirements.
Then, we could traverse the input with these helpers to build arrays of entities which can be then translated into viinex configuration and links between them:
make_nvr_app (cid, conf): {
local onvifSet = if "onvif" in conf then conf.onvif else [],
local rtspSet = if "rtsp" in conf then conf.rtsp else [],
local srcOnvif =
std.map(function(c) common.mk_onvif(common.mk_cam_name(cid, c.id),
c.addr,
conf.creds[if "cred" in c then c["cred"] else 0],
if "profile" in c then c.profile else null) + camOrigMeta(c),
onvifSet),
local srcRtsp =
std.map(function(c) common.mk_rtsp(common.mk_cam_name(cid, c.id), c.url) +
(if "cred" in c then {auth: conf.creds[c.creds]} else {}) + camOrigMeta(c),
rtspSet),
sources: srcOnvif + srcRtsp,
local recPermanent = std.filter(function (s) "rec" in s.__orig && s.__orig.rec == "permanent", self.sources),
local recMotion = std.filter(function (s) "rec" in s.__orig && s.__orig.rec == "motion", self.sources),
record: if std.all(std.map(function(s) s.__orig.rec == "none", self.sources)) then null else { motion: recMotion, permanent: recPermanent },
webrtc: true,
rtspsrv: false,
websrv: true,
wamp: false,
metrics: true,
events: "sqlite"
}
This is where the structure of the application can be formed: this function defines which viinex objects should be present in the resulting config, and what purpose each of these objects should serve for. This code may look rough or lame, — but its purpose is to illustrate the idea: we want to generate the config for the NVR “application”, and this particular function carries the knowledge of what exactly this app does, and which viinex elements are required to implement this app.
Once the “app” object is formed, it’s time to build the actual viinex configuration out of it:
build_viinex_config: function (cid, app) {
local mediaSources = app.sources,
local storages = if app.record != null then [common.mk_storage(cid+"_1")] else [],
local recctls = if app.record != null then [common.mk_recctl(s.__orig.id, 5, 5) for s in app.record.motion],
local rules = if app.record != null then [common.mk_rule_motion(s.__orig.id) for s in app.record.motion],
local linksRecctlRule = if app.record != null
then [[recctls[i].name, rules[i].name, common.mk_cam_name(cid, app.record.motion[i].__orig.id)]
for i in std.range(0, std.length(app.record.motion)-1)]
else [],
local linksRecPermanent = if app.record != null
then [[common.namesOf(storages), common.namesOf(app.record.permanent)]]
else [],
local webrtcs = if app.webrtc then [common.mk_webrtc(cid+"_0")] else [],
local rtspsrvs = if app.rtspsrv then [common.mk_rtspsrv(cid+"_0", 1554)] else [],
local websrvs = if app.websrv then [common.mk_webserver(cid+"_0")] else [],
local metrics = if app.metrics then [common.mk_metrics(cid)] else [],
local databases = if app.events == "sqlite"
then [common.mk_db_sqlite(cid+"_1")]
else [],
local wamps = [],
local publishers = webrtcs + rtspsrvs + websrvs + wamps,
local mediaPublishers = webrtcs + rtspsrvs + websrvs,
local apiPublishers = websrvs + wamps,
local apiProviders = mediaSources + storages + webrtcs + metrics + databases,
local metricsProviders = mediaSources + storages,
local eventProducers = mediaSources + storages,
objects: std.map(cleanupOrig, mediaSources + storages + publishers + metrics + recctls + rules + databases),
links: [
[common.namesOf(mediaSources), common.namesOf(mediaPublishers)],
[common.namesOf(mediaPublishers), common.namesOf(storages)],
[common.namesOf(apiPublishers), common.namesOf(apiProviders)],
[common.namesOf(metrics), common.namesOf(metricsProviders)],
[common.namesOf(recctls), common.namesOf(storages)],
[common.namesOf(databases), common.namesOf(eventProducers)]
] + linksRecctlRule + linksRecPermanent
},
Here, the “app” construct internal for our Jsonnet program is transformed into syntactically and semantically correct configuration for viinex. This function knows which sections should be present in the final JSON document, how they are related, how viinex objects should be linked, and so on. For different types of applications it’s likely to grow and have additional elements, like video analytics modules, scripts and so on.
When the above is done, viinex config can be generated with the command
jsonnet --ext-str CID=ClusterId --ext-str-file confYaml=sample-config.yaml main.jsonnet --ext-str OSName=${OSName} -o sample-config.json
The file `sample-config.json` can be used as viinex configuration file. The above Jsonnet code examples are excerpts from a real Jsonnet program which is available on Github https://github.com/viinex/config-templates and can be used to generate a simple NVR-like config for viinex.
While Jsonnet is really good at transforming one JSON into another (we’ll get to it), — it’s powerful enough to implement a basic CSV parser. This is why we were able to write a Jsonnet app to take the CSV provided by the customer, and with literally no changes convert it into viinex configuration with required functionality:
local camsCsv = std.extVar("camsCsv");
// . . .
local parseCamsCsv1 (s) =
local f = std.slice(std.split(s, ";"), 1, 9, 1);
if std.length(f) < 8 || !std.startsWith(f[3], "rtsp://")
then null
else {
desc: f[0],
name: removeNonAlnum(f[1]),
origin_id: std.split(f[0]," ")[0],
ip: f[2],
urlraw: f[3],
urlproxy: f[7],
urlparsed: parseRtspUrl(f[3]),
urlproxyparsed: parseRtspUrl(f[7]),
location: parseLocation(f[6])
};
local cams = std.filter(function (x) x != null, std.map(parseCamsCsv1, std.split(camsCsv, "\n")));
local camNames = ['cam_' + SID + '_' + cams[i-1].name for i in std.range(1, std.length(cams))];
local rendererNames = ['cam_' + SID + '_' + cams[i-1].name + "_rend" for i in std.range(1, std.length(cams))];
local alprNames = ['cam_' + SID + '_' + cams[i-1].name + "_alpr" for i in std.range(1, std.length(cams))];
local scriptNames = ['script_' + SID + '_' + cams[i-1].name for i in std.range(1, std.length(cams))];
and then these arrays can be mapped into viinex objects for cameras, video analytics-related objects, scripts, — to finally get the resulting viinex config:
{
objects: [camObject(camNames[i - 1], cams[i - 1]),
for i in std.range(1, std.length(cams))]
+ std.map(rendererObject, rendererNames)
+ [alprObject(alprNames[i-1], rendererNames[i-1])
for i in std.range(1, std.length(cams))]
+ [scriptObject(scriptNames[i-1], alprNames[i-1], camNames[i-1], cams[i-1])
for i in std.range(1, std.length(cams))]
+ [webrtc, metrics, websrv, db, storage, rtspsrv],
links: [
[metrics.name, camNames],
[metrics.name, rendererNames],
[metrics.name, [db.name, storage.name]],
[[websrv.name, webrtc.name, rtspsrv.name], camNames],
[[websrv.name, webrtc.name, rtspsrv.name], storage.name],
[[db.name, websrv.name], alprNames],
[websrv.name, [webrtc.name, metrics.name, db.name]],
[storage.name, camNames],
[[websrv.name, storage.name, db.name], scriptNames],
]
+ [[camNames[i-1], rendererNames[i-1]] for i in std.range(1, std.length(cams))]
+ [[scriptNames[i-1], alprNames[i-1]] for i in std.range(1, std.length(cams))]
}
Similarly to the previous example, the command to generate viinex configuration invokes jsonnet with `–ext-str camsCsv=customer-provided.csv` command line parameter, and directly transforms that CSV file into viinex configuration for that specific business application.
As a conclusion, we’d like to emphasize that while it’s possible to manually write viinex configuration, we encourage the users to automate this job for typical use cases. (Which exact use cases are typical – depends on the business applications and requirements, here’s why this automation cannot be implemented directly in viinex: that would make it less generic). Jsonnet can be a good solution for this: it’s powerful enough to handle different input formats, yet small and safe to be even shipped to the target production system. The advantages of that approach are that:
- When needed, configuration can be easily parameterized with external variables;
- Once Jsonnet modules incorporate the complexity of viinex configuration structure – when it’s time to deploy the system with viinex included, there’ll likely be only a simple question on which config pattern to use, — but no longer a need to review all the elements of viinex config over again;
- With using config generation approach, specifically transforming the user input into viinex configuration with Jsonnet code, the format of user input can be simplified (see above for YAML or CSV-based config examples), and additional validation can be implemented.