Video manipulation with WebAssembly
The previous article of this series discussed the possibility of modifying video streams in a browser. The new “Insertable Streams for MediaStreamTrack API” makes it possible, by allowing the web-application to modify the frames before the stream is sent further. The example in the previous article used plain JavaScript, and according to the test, the performance was moderate.
Now it’s time to test the famous WebAssembly technology, and see how it performs on this field.
The goal of the experiment is the same. We will create the same green screen effect, and the only difference will be that frame processing will be offloaded to a WebAssembly function.
WebAssembly, in a nutshell, is a kind of binary-code running in a browser environment, that offers excellent performance metrics. https://en.wikipedia.org/wiki/WebAssembly
Implement it!
Let’s modify the previous example and swap the transformer function to it’s WebAssembly counterpart.
In this example, WebAssembly’s source is good old C. First, let’s take a look at it and later, we will discuss the JavaScript binding and compilation process. The C source is very simple and looks very similar to the JavaScript counterpart.
There’s one difference that we should point out: the initializer functions. The JavaScript host will call the buffer initializer functions, then the WebAssembly module allocates the buffer and returns the corresponding memory address (pointer).
During the operation, the static shared buffers are accessed from both the JavaScript host and the WebAssembly module, so no frame-data will be copied.
The WebAssembly compilation process
This part is pretty simple. In this example, I’m going to avoid the difficulties of installing the Emscripten framework by using an online WebAssembly compiler: https://webassembly.studio/
The JavaScript binding
Let’s look at the following snippet that shows how to connect the host environment ( JavaScript ) with the WebAssembly module. More information on this topic can be found in the “WebAssembly Studio” documentation here: https://github.com/wasdk/WebAssemblyStudio/blob/99806ca60cbe560f5f1bd86246c1f226a8b6ea2f/templates/hello_world_c/README.md
I took the binding code from the basic example in the WebAssembly Studio.
After we have instantiated the module, the JavaScript host will be able to access its exported functions from the “instance.exports” object, just like the shared memory under the “memory.buffer” property.
The performance
Surprisingly, the WebAssembly module doesn’t perform substantially better, at least for this particular use case. It’s maybe faster by 5–8%, but not more!
