nvidia tk1使用记录--基本环境搭建

前言

　　项目最开始是在X86+Nvidia（ubuntu+opencv+cuda）平台上实现，达到了期望性能，最近考虑将其移植到嵌入式平台，特别是最近nvidia出了tegra X1，基于和我们使用的gtx960相同的架构的gpu且是64位 8核心cpu，于是开始了尝试移植过程。当前TX1才出来，购买不是那么方便，于是拿它的前一代TK1进行一个最基本的评估。下面记录了评估过程的一些操作。

tk1系统更新

　　拿到的tk1时，板子自带的系统版本(非内核版本)是Jetson TK1 R19.3 - June 2014，非常老了，有必要对其进行升级。当前(2016/04/11)，最新的系统版本为Linux For Tegra R21.4，它的相关介绍及下载包地址在这里。系统更新有两种方法，不管是哪种方式，都需要用数据线（一端接tk1的micro usb口，一端接PC机的usb口）连接开发板与主机，然后上电的时候让设备进入recovery模式，最后在主机上启动升级。

注意：tk1开箱上电启动时，默认是在非图形界面下的，ls一下会看到目录NVIDIA-INSTALLER，cd进去，执行sudo ./installer来进行图形化安装，然后sudo reboot重启即可进入图形界面了。默认的用户名是ubuntu，密码ubuntu。

自动更新

　　JetPack for L4T傻瓜式自动更新，提供图形界操作。它是由官方提供的升级包，可以说它是所有需要安装包的集合,包括最新系统,cuda,opencv,cross-develop-environment,samples等等,非常方便,缺点就是主机系统只能是ubuntu12.04或者ubuntu14.04，安装后的软件包版本可能并非期望的,比如opencv是2.4。关于这种更新方式的详细介绍请参考这里，我没有采用这种升级方式，所以不再过多描述。

手动更新

　　这种升级方式实际上和上面的升级方式一样，只不过它仅仅是上面升级里面的一个步骤而已，JetPack for L4T不仅仅更新系统，还会同时更新其他软件包，如cuda、opencv等等。具体升级的步骤如下：

1. 下载内核及驱动包http://developer.download.nvidia.com/embedded/L4T/r21_Release_v4.0/Tegra124_Linux_R21.4.0_armhf.tbz2；
2. 下载文件系统包
   http://developer.download.nvidia.com/embedded/L4T/r21_Release_v4.0/Tegra_Linux_Sample-Root-Filesystem_R21.4.0_armhf.tbz2;
3. 构建升级包(包括了boot、kernel、fs)，执行sudo tar --numeric-owner -jxpf Tegra124_Linux_R21.4.0_armhf.tbz2解压内核及驱动包，然后cd linux_for_tegra/rootfs下,再次解压文件系统包sudo tar --numeric-owner -jxpf ../../Tegra_Linux_Sample-Root-Filesystem_R21.4.0_armhf.tbz2，然后cd ../，即返回到linux_for_tegra，执行sudo ./apply_binaries.sh构建升级包。注意，执行命令前,请先确保系统还有20G空间,如果没有,那么将14GiB改下点吧,虽然这样会导致板子上flash没充分利用,但至少能够更新系统;
4. 开始升级，USB端插到ubuntu PC上，小头插到TK1上，也就是耳机口旁边的口。TK1上有三个按钮，最左边的是电源，中间的是reset，最右边的是force recoverry，按住最右边的force recoverry 按钮别松手，再按一下中间的reset键，指示灯会闪一下，板子就重启进入了刷机模式。然后在ubuntu PC上，当前目录，即linux_for_tegra目录下，执行终端命令：sudo ./flash.sh -S 14GiB jetson-tk1mmcblk0p1，这个时候ubuntu上会弹出一个设备，就是tk1了，然后就会进入刷机过程，若干小时后，最后成功了的话，会出现success字样。这时重启TK1开发板，就发现这板子已经成功被初始化了。进入ubuntu桌面环境，默认的用户名是ubuntu，密码ubuntu。

tk1 cuda安装

　　cuda的安装可以参考elinux的指导，这部分也没有需要特别注意的，确保cuda安装在opencv之前以及所有操作都是在tk1系统里进行的就可以了。下面还是把关键步骤给摘抄下来吧(略加修改)！

Download the .deb file for the CUDA Toolkit for L4T from http://developer.download.nvidia.com/embedded/L4T/r21_Release_v3.0/cuda-repo-l4t-r21.3-6-5-prod_6.5-42_armhf.deb

On the device, install the .deb file and the CUDA Toolkit. eg:

cd ~/Downloads
# Install the CUDA repo metadata that you downloaded manually for L4T
sudo dpkg -i cuda-repo-l4t-r21.3-6-5-prod_6.5-42_armhf.deb
# Download & install the actual CUDA Toolkit including the OpenGL toolkit from NVIDIA. (It only downloads around 15MB)
sudo apt-get update
# Install "cuda-toolkit-6-0" if you downloaded CUDA 6.0, or "cuda-toolkit-6-5" if you downloaded CUDA 6.5, etc.
sudo apt-get install cuda-toolkit-6-5
# Add yourself to the "video" group to allow access to the GPU
sudo usermod -a -G video $USER

Add the 32-bit CUDA paths to your .bashrc login script, and start using it in your current console:

echo "# Add CUDA bin & library paths:" >> ~/.bashrc
echo "export PATH=/usr/local/cuda/bin:$PATH" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=/usr/local/cuda/lib:$LD_LIBRARY_PATH" >> ~/.bashrc
source ~/.bashrc

Verify that the CUDA Toolkit is installed on your device:

nvcc -V

tk1 opencv安装

　　opencv的安装可以参考elinux的指导，我这里还是对每一个步骤详细的描述下，因为这里会遇到些上面链接里没有指出的问题，另外要注意，下面所有操作都是在tk1系统里进行的。opencv的安装有两种方式：

方法一，直接采用nvidia提供的安装包。优点：简单，且同时增加了cpu neon的优化；缺点：版本不够新2.4，基于opencv-3.0.0实现的代码可能需要修改代码。操作方法如下：

1. 从http://developer.download.nvidia.com/embedded/OpenCV/L4T_21.2/libopencv4tegra-repo_l4t-r21_2.4.10.1_armhf.deb下载opencv deb包
2. 执行sudo dpkg -i libopencv4tegra-repo_l4t-r21_2.4.10.1_armhf.deb安装
3. sudo apt-get update
4. sudo apt-get install libopencv4tegra libopencv4tegra-dev

方法二，从源码编译。优点：能够使用最新的opencv版本代码；缺点：复杂。下面的操作步骤中包含了对OpenGL的使能，需要注意的是如果不添加-DWITH_QT=ON，那么-DWITH_OPENGL=ON将没有效果，因此如果你想开启OPENGL使能，还得先参考后面怎么安装qt环境。参考 ...

If you haven't added the "universal" repository to Ubuntu, then do it now:

sudo add-apt-repository universe
sudo apt-get update

Now you need to install many libraries:

# Some general development libraries
sudo apt-get -y install build-essential make cmake cmake-curses-gui g++
# libav video input/output development libraries
sudo apt-get -y install libavformat-dev libavutil-dev libswscale-dev
# Video4Linux camera development libraries
sudo apt-get -y install libv4l-dev
# Eigen3 math development libraries
sudo apt-get -y install libeigen3-dev
# OpenGL development libraries (to allow creating graphical windows)
sudo apt-get -y install libglew1.6-dev
# GTK development libraries (to allow creating graphical windows)
sudo apt-get -y install libgtk2.0-dev

Download the source code of OpenCV for Linux onto the device. eg: Open a web-browser to "www.opencv.org" & click on "OpenCV for Linux/Mac", or from the command-line you can run this on the device:

wget https://github.com/Itseez/opencv/archive/3.1.0.zip

Unzip the OpenCV source code:

cd Downloads
unzip 3.1.0.zip
mv opencv-3.1.0 ~

Configure OpenCV using CMake:

cd ~/opencv-3.1.0
mkdir build
cd build
cmake -DWITH_CUDA=ON -DCUDA_ARCH_BIN="3.2" -DCUDA_ARCH_PTX="" -DBUILD_TESTS=OFF -DWITH_OPENGL=ON -DWITH_QT=ON -DBUILD_PERF_TESTS=OFF ..

sudo make -j4 install

Finally, make sure your system searches the "/usr/local/lib" folder for libraries:

echo "# Use OpenCV and other custom-built libraries." >> ~/.bashrc
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/" >> ~/.bashrc
source ~/.bashrc

编译FAQ

错误1

 error: #error Please include the appropriate gl headers before including cuda_gl_interop.h
 #error Please include the appropriate gl headers before including cuda_gl_interop.h

错误2

/home/ubuntu/build/opencv/modules/highgui/src/window_QT.cpp:3150:12: error: 'GL_PERSPECTIVE_CORRECTION_HINT' was not declared in this scope
     glHint(GL_PERSPECTIVE_CORRECTION_HINT, GL_NICEST);

可以尝试下面的解决办法:

there is a workaround, if you delete folllowing lines in /usr/local/cuda/include/cuda_gl_interop.h

#ifndef GL_VERSION
#error Please include the appropriate gl headers before including cuda_gl_interop.h
#endif
#else

then gl headers is anyway included and opencv compiles.. with some rain dancing for me)
in my case, i had to add

#define GL_PERSPECTIVE_CORRECTION_HINT 0x0C50

to modules/highgui/src/window_QT.cpp and turn off TBB support and cvv module (i was building with opencv_contrib)

ref: https://github.com/Itseez/opencv/issues/5205

ref: http://code.opencv.org/issues/3411

tk1 qt安装方式

注意：下面所有操作都是在tk1系统里进行的

方法一，直接采用仓库里已经提供到安装包。优点：简单；缺点：一般不是最新的版本，而且是基于OpenES的，对于之前代码是基OpenGL编写的需要做到OpenES的移植。具体安装步骤：

sudo apt-get install build-essential qt5-default qtcreator -y

which will load Qt Creator. Once Qt Creator is installed, there is still a little setup needed.

Open Qt Creator, and go to:

    Tools->Options->Build & Run->Compilers

Click the ‘Add’ button and select ‘GCC’. In the ‘Compiler path:’ text box, place the path to the gcc compiler. On a standard installation the path is: /usr/bin/gcc.

You should also check the ‘Kits’ setting, available from the ‘Kits’ tab. Selecting the Desktop (default) kit, you should see Compiler: GCC. The name GCC should match the Compiler named in the previous tab.

The installation was shown on Linux for Tegra (L4T) release 21.3. Note that no special compiler flags were set. Also, CUDA was not configured.

ref: http://jetsonhacks.com/2015/04/22/install-qt-creator-on-nvidia-jetson-tk1/

方法二，从源代码编译。优点：能够基于任意版本(比如最新的qt)编译，且可以基于OpenGL而不基于OpenES；缺点：复杂，编译时间长。详细步骤参考下面两个链接:

http://blog.csdn.net/deyili/article/details/51012106

blog.csdn.net/deyili/article/details/49868987

需要注意的是,如果遇到如下编译错误:

zlib.h:86:5: error: 'z_const' does not name a type

，那么尝试在configue的时候去掉-zlib

如果在configure的时候xcb不过，通过官方解决方法(安装xcb包http://doc.qt.io/qt-5/linux-requirements.html), 如果还解决不了,那么尝试直接修改configure，如下:

    #if [ "$CFG_XKBCOMMON" != no ] && compileTest qpa/xcb "xcb" $QMAKE_CFLAGS_XCB $QMAKE_LIBS_XCB; then
    if [ "$CFG_XKBCOMMON" != no ]; then

编译Qt Creator

　　同样给出参考链接，里面已经写的很详细了。

关于OpenGL和OpenES的Qt

This board is powerful enough to build everything on its own without any cross-compilation. Configuring and building Qt is no different than in any desktop Linux environment. One option that needs special consideration however is -opengl es2 because Qt can be built either in a GLX + OpenGL or EGL + OpenGL ES configuration.

For example, the following configures Qt to use GLX and OpenGL:

configure -release -nomake examples -nomake tests

while adding -opengl es2 requests the usage of EGL and OpenGL ES:

configure -release -opengl es2 -nomake examples -nomake tests

If you are planning to run applications relying on modern, non-ES OpenGL features, or use CUDA, then go for the first. If you however have some existing code from the mobile or embedded world relying on EGL or OpenGL ES then it may be useful to go for #2.

The default platform plugin will be xcb, so running Qt apps without specifying the platform plugin will work just fine. This is the exact same plugin that is used on any ordinary X11-based Linux desktop system.

http://blog.qt.io/blog/2015/03/03/qt-weekly-28-qt-and-cuda-on-the-jetson-tk1/

另外一个问题(在iss上出现过的视频断层问题tearing,应该可以通过该方法解决)
Vsync gotchas

Once the build is done, you will most likely run some OpenGL-based Qt apps. And then comes the first surprise: applications are not synchronized to the vertical refresh rate of the screen.

When running for instance the example from qtbase/examples/opengl/qopenglwindow, we expect a nice and smooth 60 FPS animation with the rendering thread throttled appropriately. This unfortunately isn’t the case. Unless the application is fullscreen. Therefore many apps will want to replace calls like show() or showMaximized() with showFullScreen(). This way the thread is throttled as expected.

A further surprise may come in QWidget-based applications when opening a popup or a dialog. Unfortunately this also disables synchronization, even though the main window still covers the entire screen. In general we can conclude that the standard embedded recommendation of sticking to a single fullscreen window is very valid for this board too, even when using xcb, although for completely different reasons.

http://blog.qt.io/blog/2015/03/03/qt-weekly-28-qt-and-cuda-on-the-jetson-tk1/

cuda于Qt的结合

blog.qt.io/blog/2015/03/03/qt-weekly-28-qt-and-cuda-on-the-jetson-tk1/

FAQ

opencv error: the function/feature is not implemented

http://answers.opencv.org/question/84639/opencv-error-the-functionfeature-is-not-implemented/

/usr/include/qt4/QtOpenGL/qgl.h:85:17: error: 'GLdouble' has a previous declaration as 'typedef GLfloat GLdouble'

http://answers.ros.org/question/179989/rgbdslam_v2-error-with-make/

总结

　　评估过程可以说非常的顺利，一周内将整个项目全部移植了过来，而且基本没有代码改动^_!!!!经过测试，我们的拼接算法单路能够带到65帧，2路能够代码35帧，3路没测试，因为tk1只有2G内存，系统本身用了将近700MB，而我们程序2路的时候用了1000多M，只剩50多M了。总的来说，对于移植到TX1上，心里更加有底了…………

参考

1. Boot Jetson TK1 in recovery mode
2. Jetson TK1
3. Linux For Tegra R21.4
4. Building_Qt_5_from_Git

完！
2016年5月