Doing Machine Vision on Google Coral with OpenCV

While developing computer vision applications for embedded devices, I realized that execution time is going to be the number one priority. However, after failing to install all required libraries for such a task, I understood that an easy setup process is going to be of great value as well. What I am specifically talking about is installing OpenCV for Python applications on Google Coral Dev board. Coral was meant to run inference at incredible speeds. But of what use is Coral to me if I can’t even process video frames with OpenCV?

While everything worked flawlessly on a desktop computer, I went to great lengths when installing OpenCV 4.1.1 on Coral. If you are in the same situation, you might have just found a solution! While there is a guide which I followed thoroughly, it didn’t quite work for OpenCV 4.1.1.

Therefore, this post explains how to install OpenCV 4.1.1 on Google Coral Dev board and it is an extension to this and this guide, but with changes and explanations of what and why I changed it.

Requirements

Here is a list of requirements you should do before following this guide:

Flash a system image on the board.
Connect to your device using either ssh or serial console.

Install OpenCV dependencies. Here is how to install everything that is needed for a successful OpenCV build:

sudo apt-get install build-essential cmake unzip pkg-config
sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libxvidcore-dev libx264-dev
sudo apt-get install libgtk-3-dev
sudo apt-get install libatlas-base-dev gfortran
sudo apt-get install python3-dev

Install git:

sudo apt-get update
sudo apt-get install git

Quick solution

Download the prebuilt libraries from this repository.

  git clone https://github.com/pjalusic/opencv4.1.1-for-google-coral.git

Copy the cv2.so file to /usr/local/lib/python3.7/dist-packages/.

  cp opencv4.1.1-for-google-coral/cv2.so /usr/local/lib/python3.7/dist-packages/cv2.so 

Copy other .so files to /usr/local/lib/.

  sudo cp -r opencv4.1.1-for-google-coral/libraries/. /usr/local/lib

Check if it works.

  python3
  >>> import cv2
  >>> cv2.__version__
  '4.1.1'

If it works, you are done. You can now link cv.so file to your virtual environment if you’d like. However, if you prefer building OpenCV yourself or it didn’t work using the quick solution, here is a guide.

Build your own libraries

Create a swap file for the build.
```
  sudo fallocate -l 2G /swapfile
  sudo chmod 600 /swapfile
  sudo mkswap /swapfile
  sudo swapon /swapfile
```
Google Coral has only 1GB of RAM which is less than required for the build job. Therefore, we create a swapfile. I am using a 2GB swap file to eliminate any possibility of errors while building, which occurred when using only 1GB swap file.
Mount USB formatted to ext file system (for formatting try this for mac, or this for linux).
```
  sudo /sbin/fdisk -l
  # at the end is your device, for instance /dev/sda1
  sudo mount /dev/sda1 /mnt
```
Instead of USB you can use an SD card. Whichever works, but it should have at least 4GB of memory.

Download OpenCV 4.1.1.

  cd /mnt
  wget -O opencv.zip https://github.com/opencv/opencv/archive/4.1.1.zip
  wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.1.1.zip
  unzip opencv.zip
  unzip opencv_contrib.zip
  mv opencv-4.1.1 opencv
  mv opencv_contrib-4.1.1 opencv_contrib

Make changes in opencv/cmake/OpenCVFindLibsPerf.cmake file. Before compiling OpenCV, you need to do something extra to avoid errors. This might be only necessary for version 4.1.1, but it may help with others as well. I created a public gist containing a git diff patch file. Clone the repository into your cmake folder and apply changes.
```
  cd opencv/cmake
  git clone https://gist.github.com/pjalusic/7feb9cd722a437c0f6eea34622ca44c8.git
  cp 7feb9cd722a437c0f6eea34622ca44c8/OpenCVFindLibsPerf.diff .
  git apply OpenCVFindLibsPerf.diff
```
If there are errors when applying the patch, double check you downloaded OpenCV 4.1.1 or do this by hand. There are exactly 5 lines commented. If it was successful, part of OpenCVFindLibsPerf.cmake file from line 43 should look like this:
```
  if(WITH_EIGEN AND NOT HAVE_EIGEN)
 find_package(Eigen3 3.0.0)
 if(Eigen3_FOUND)
   #if(TARGET Eigen3::Eigen)
     # Use Eigen3 imported target if possible
     #list(APPEND OPENCV_LINKER_LIBS Eigen3::Eigen)
     #set(HAVE_EIGEN 1)
   else()
     if(DEFINED EIGEN3_INCLUDE_DIRS)
       set(EIGEN_INCLUDE_PATH ${EIGEN3_INCLUDE_DIRS})
       set(HAVE_EIGEN 1)
     elseif(DEFINED EIGEN3_INCLUDE_DIR)
       set(EIGEN_INCLUDE_PATH ${EIGEN3_INCLUDE_DIR})
       set(HAVE_EIGEN 1)
     endif()
   #endif()
 ...
```
As you can see there are a few changes. The first one is using Eigen3 3.0.0 instead of Eigen QUIET. Secondly, we comment out one if check which gave errors. What all of this does is fixes the compatibility problem with Eigen library because CMake can’t see it. Mind the last comment on endif().

Create a python virtual environment.

  cd ~
  wget https://bootstrap.pypa.io/get-pip.py
  sudo python3 get-pip.py
  sudo pip install virtualenv virtualenvwrapper
  sudo rm -rf ~/get-pip.py ~/.cache/pip

Add the following lines to ~/.bashrc file (ie. nano ~/.bashrc) for virtualenv and virtualenvwrapper.

  export WORKON_HOME=$HOME/.virtualenvs
  export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
  source /usr/local/bin/virtualenvwrapper.sh

Source that file.
```
  source ~/.bashrc
```
Create a virtual environment, activate it and install numpy (for some reason I was getting errors until I connected to the device through ssh).
```
  mkvirtualenv cv -p python3
  workon cv
  pip install numpy
```

Compile OpenCV 4.1.1.

  cd /mnt/opencv
  mkdir build && cd build
  cmake -D CMAKE_BUILD_TYPE=RELEASE \
  -D CMAKE_INSTALL_PREFIX=/usr/local \
  -D INSTALL_PYTHON_EXAMPLES=ON \
  -D INSTALL_C_EXAMPLES=OFF \
  -D OPENCV_ENABLE_NONFREE=ON \
  -D OPENCV_EXTRA_MODULES_PATH=/mnt/opencv_contrib/modules \
  -D PYTHON_EXECUTABLE=~/.virtualenvs/cv/bin/python \
  -D ENABLE_FAST_MATH=1 \
  -D ENABLE_NEON=ON -D WITH_LIBV4L=ON \
  -D WITH_V4L=ON \
  -D BUILD_EXAMPLES=ON ..

Start building.
```
  make -j$(nproc)
```
This will utilize all cores for the build process. It, however, still takes a couple of hours.
Install OpenCV.
```
  sudo make install
  sudo ldconfig
```

Link to the virtual environment.

  cd /usr/local/lib/python3.7/dist-packages/cv2/python-3.7
  sudo mv cv2.cpython-37m-aarch64-linux-gnu.so cv2.so
  cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
  ln -s /usr/local/lib/python3.7/dist-packages/cv2/python-3.7/cv2.so cv2.so

Test it.

  workon cv
  python
  >>> import cv2
  >>> cv2.__version__
  '4.1.1'

Well, this was a bit painful and a long process that took too much time. Obviously, the next step is to speed up the build process. How? Well, an idea is to move the build process to an ARM device in the cloud. But that’s a story for another blog post.

Additional information, notes and problems

Build OpenCV in background.

If you would like to do any process in the background, you should check tmux or similar tools. Here is how to do it using tmux:
- sudo apt-get install tmux
- tmux → creates a new session. Run workon cv once again. Run make -j$(nproc) command while in the session
- Control+b, d → leave session
- tmux ls → show sessions
- tmux attach -t 0 → return to that session
Permission errors while building OpenCV.

If you encounter permission errors while compiling, building or installing OpenCV, you could always go with the root user option. So before building or opening session with tmux do sudo su. That will also mean that any sessions you create will be bound to root user.
Problem with reflashing.

I stumbled on a serious issue when trying to reflash the device to test my tutorial once again. Following this official guide on reflashing, my Mac was stuck at mdt devices step. Nothing was showing up. Apparently, it is an ongoing issue. My solution was just switching to Windows computer. To follow the guide and successfully reflash the device all I needed to do was:
- Install python3.
- Install mdt with pip3.
```
pip3 install --user mendel-development-tool
```
- Get fastboot tool from SDK Platform Tools and manually add fastboot.exe to path

Blog