I originally had a simple goal: do a computer vision task solely from my browser. Having seen some very nice colour transfer effects before, I figured I could try a simple one. So I found a nice and popular algorithm: color Transfer by jrosebr1 and started analysing it.
I quickly realised that there might be a problem since it mostly relied on OpenCV which is a Python library.
The python-OpenCV library is quite impressive and allows plenty of advanced applications for minimal effort (such as image or video processing and object detection). The question was: can we do all that in the browser?
And then began my dreadful experience …
Don’t expect OpenCV.js to do all the things you can do with OpenCV in C / C++ / python: it is not as advanced. So web computer vision is still a pretty complex task.
- OpenCV.js does not implement the same functions as for the other languages
- There is no way to know in advance if something is going to work other than testing it and praying
- The tutorials on the official website help a lot but:
- they don’t cover everything
- they make us wonder if the only functions we can use are the ones that we see in the tutorial
- It feels like reverse engineering, debugging along the way and comparing if the values are the same. Spoiler alert: they are not!
- The library isn’t lightweight either: approximately 7 Mo, so it will take a while to load for an end user (3 seconds at 20 MB/s)
.js file as the ones used in the python algorithm.
The expected outcome is to be able to pick one colorful picture and a black and white picture (from your browser), click a button and after a few seconds see the black and white picture colorised: computer vision!
Get OpenCV js:
Here you have a choice,
- you can be lazy (like me) and directly download the opencv.js file from this link
- or clone the repo and build it yourself: instructions here.
Set up a playground:
The first step was to set up a basic webpage to be able to pick our input images and display the output image from our wonderful color transfer algorithm.
This wasn’t the main focus of this experiment so I basically took the one from the RisingStack article and customised it:
This section shows the breakdown of the computer vision algorithm I chose to translate.
1. The first step in jrosebr1’s algorithm is converting our image from the RGB space to the LAB space:
What’s the LAB space you might ask? As Wikipedia puts it:
It expresses colour as three numerical values, L* for the lightness and a* and b* for the green–red and blue-yellow colour components.
I have —unsuccessfully— looked around for what this
astype('float32') does and if there is an equivalent in opencv.js but haven’t found anything . Let’s hope this doesn’t break too much…
2. Next we want to get the image stats:
So far so good, but the tricky part is in the
Here as you can see I used
cv.meanStdDev. Why didn’t I use
.std()? Simply because they don’t exist in Opencv.js
So after some googling around, I found this post mentioning
cv.meanStdDev which seemed to do what I wanted.
I tested and printed the stats output for the source image and … I don’t have the same values Not sure if it comes from the
But except for
bMean, the values are only off by a bit so I figured I could go on and see what the output images would look like.
3. And now we need to translate and scale all the values in the
Looks easy right?
⚠️⛔️ Well, that’s because it does not work!
I looked a bit at what this LAB image was made of and this is the OpenCV matrix for the L part of the Lab plan of a picture of a sunset (the
b part are built the same with different values):
Without helper functions to do arithmetic operations, I don’t think it’s worth spending more time with this library.
Conclusion: should you use it?
My experience with computer vision in the browser using OpenCV.js has really not been great, but that doesn’t mean you shouldn’t use it!
There are some breath-taking demos of in browser computer vision done with OpenCV on their website such as:
So in my opinion, if what you want to do can be done in a couple openCV calls without additional manipulation, just go for it!
Look at the tutorials and hopefully, you’ll find out how the functions you want to use work.
However, if you want to do more than what OpenCV offers out of the box, like manipulating pixels, I suggest you try and find another way of doing it (in a python backend for instance)