TK1之ros -31

  • 1、ROS在ARM的安装

1.1 个性配置
1) server上启动VPN
2) client上设置证书
3) network设置

  • 2、ROS在TK1的安装

opencv4tegra vs opencv4
2.1 HACK 1
2.2 HACK 2
2.3 HACK 3
2.4 HACK 4

  • 3、深度学习机器人

3.1 wifi
3.2 SD card

1、ROS在ARM的安装

1.1 Set Locale 设置你的环境

Boost and some of the ROS tools, all require that the system locale be set.
Linux中通过locale来设置程序运行的不同语言环境,locale由ANSI C提供支持。locale的命名规则为<语言>_<地区>.<字符集编码>
如zh_CN.UTF-8,zh代表中文,CN代表大陆地区,UTF-8表示字符集。

You can set it with:
$ sudo update-locale LANG=C LANGUAGE=C LC_ALL=C LC_MESSAGES=POSIX
LC_ALL 它是一个宏,如果该值设置了,则该值会覆盖所有LC_*的设置值。注意,LANG的值不受该宏影响。
“C”是系统默认的locale,”POSIX”是”C”的别名。所以当我们新安装完一个系统时,默认的locale就是C或POSIX。
LC_ALL=C 是为了去除所有本地化的设置,让命令能正确执行。
LC_MESSAGES , 提示信息的语言。

If there is a problem. Then try (other languages could be added):
$ export LANGUAGE=en_US.UTF-8
$ export LANG=en_US.UTF-8
$ export LC_ALL=en_US.UTF-8
$ locale-gen en_US.UTF-8
$ dpkg-reconfigure locales

1.2 Setup sources.list

Setup your computer to accept software from the ARM mirror on packages.ros.org.
Due to limited resources, there are only active builds for Trusty armhf (14.04), since this is the stable, long-term Ubuntu release and is the most-requested distribution in conjunction with ROS Indigo.

$ sudo sh -c ‘echo “deb http://packages.ros.org/ros/ubuntu trusty main” > /etc/apt/sources.list.d/ros-latest.list’

1.3 Set up keys 设置密码

$ sudo apt-key adv –keyserver hkp://ha.pool.sks-keyservers.net –recv-key 0xB01FA116
or,
$ sudo apt-key adv –keyserver hkp://ha.pool.sks-keyservers.net –recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116

If you can try the following command by adding :80 if you have error:
gpg: keyserver timed out error due to a firewall
$ sudo apt-key adv –keyserver hkp://ha.pool.sks-keyservers.net:80 –recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116

If you have error
GPG error: Clearsigned file isn’t valid, got ‘NODATA’ (does the network require authentication?)
$ such as first apt-get clean then apt-get update has no effect, what you need is to get a TiZi.

1.4 make sure Debian package index is up-to-date:
$ sudo apt-get update

1.5
Desktop Install, include ROS, rqt, rviz, and robot-generic libraries
$ sudo apt-get install ros-indigo-desktop

NOT desktop-full, include ROS, rqt, rviz, robot-generic libraries, 2D/3D simulators and 2D/3D perception .
$ sudo apt-get install ros-indigo-desktop-full

1.6 Initialize rosdep
Before you can use ROS, you will need to install and initialize rosdep. rosdep enables you to easily install system dependencies for source you want to compile and is required to run some core components in ROS.
$ sudo apt-get install python-rosdep
$ sudo rosdep init
$ rosdep update

1.7 Environment setup
It’s convenient if the ROS environment variables are automatically added to your bash session every time a new shell is launched:
$ echo “” >> ~/.bashrc
$ echo “# Source ROS indigo setupenvironment:” >> ~/.bashrc
$ echo “source /opt/ros/indigo/setup.bash” >> ~/.bashrc
$ source ~/.bashrc

1.8 Getting rosinstall
rosinstall is a frequently used command-line tool in ROS that is distributed separately. It enables you to easily download many source trees for ROS packages with one command. To install this tool on Ubuntu, run:
$ sudo apt-get install python-rosinstall

1.x 设置env
检查ROS_ROOT和ROS_PACKAGE_PATH路径是否设置:
$ printenv | grep ROS
如果未设置则刷新环境:
$ source /opt/ros/indigo/setup.bash

1.9 Verifying OS name

Make sure your OS name defined at /etc/lsb-release is as the following.
Since ros does not recognize Linaro as an OS, this is necessary.
The following is for Ubuntu 14.04, trusty. Modify the release number and name as per your target.
$ vi /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION=”Ubuntu 14.04″

1.10 个性配置
1) server上启动VPN
setup vpn network
cp client.conf
cp ca.crt, ta.key
cp Client012.crt, Client012.key
chmod mod 600 ta.key Client012.key

2)client上设置证书
import cert file for firefox and specialy for chrome
需要根据证书里Issued to的域名信息修改 https certification:
$ vi /etc/hosts
10.10.0.1 dehaou14-n501jw
to visit srv

3)network设置
tegra-ubuntu上要合适配置/etc/hosts,并运行roscore。
pc和srv上要合适配置/etc/hosts,并:
export ROS_MASTER_URI=http://tegra-ubuntu:11311。
export ROS_HOSTNAME=dehaou14-n501jw
完成后可以测试:
$ rosnode ping /somenode

1.11 Using RVIZ notice:
It is not recommended to run rviz on most ARM-based CPUs.
They’re generally too slow, and the version of OpenGL provided by the software (mesa) libraries it not new enough to start rviz.

‘IF’ you have a powerful board with a GPU and vendor-supplied OpenGL libraries, it might be possible to run rviz.
The IFC6410 and the NVidia Jetson TK1 are two such boards where rviz will run, although neither is fast enough for graphics-heavy tasks such as displaying pointclouds.

NOTES:
Note that rviz will segfault , if you have the GTK_IM_MODULE environment variable set, so it’s best to unset it in your ~/.bashrc:
unset GTK_IM_MODULE

REF: http://wiki.ros.org/indigo/Installation/UbuntuARM

2、ROS在TK1的安装

opencv4tegra vs opencv4

2.1 HACK 1 – cv_bridge

With the latest opencv4tegra 21.2 released by Nvidia, the compatibility problems with cv_bridge and image_geometry packages have been solved, so installing OpenCV ROS Packages from PPA does not force opencv4tegra to be uninstalled.
???really???

but a few issues still remain since ROS searches for OpenCV v2.4.8, but opencv4tegra is based on OpenCV v2.4.12.
BUT, There are yet a bit of incompatibility since cv_bridge and image_geometry search for OpenCV 2.4.8 in “/usr/lib/arm-linux-gnueabihf” ,but opencv4tegra is based on OpenCV 2.4.12 and is installed in “/usr/lib/”.
These diversities do not allow to compile external packages based on OpenCV.

To solve the problem you can follow this guide:
http://myzharbot.robot-home.it/blog/software/ros-nvidia-jetson-tx1-jetson-tk1-opencv-ultimate-guide/
What we must “say” to cv_bridge and image_geometry is to not search for OpenCV in the default ARM path “/usr/lib/arm-linux-gnueabihf”, but in “/usr/lib” and that the current version of OpenCV is 2.4.12 and not 2.4.8. finally we must remove the references to the module OpenCL because Nvidia does not provide it.

1) Files to be modified
/opt/ros//lib/pkgconfig/cv_bridge.pc
/opt/ros//lib/pkgconfig/image_geometry.pc
/opt/ros//share/cv_bridge/cmake/cv_bridgeConfig.cmake
/opt/ros//share/image_geometry/cmake/image_geometryConfig.cmake

2) You can backup and modify each file using the following commands (example for ROS Indigo):
sudo cp /opt/ros/indigo/lib/pkgconfig/cv_bridge.pc /opt/ros/indigo/lib/pkgconfig/cv_bridge.pc-bak
sudo cp /opt/ros/indigo/lib/pkgconfig/image_geometry.pc /opt/ros/indigo/lib/pkgconfig/image_geometry.pc-bak
sudo cp /opt/ros/indigo/share/cv_bridge/cmake/cv_bridgeConfig.cmake /opt/ros/indigo/share/cv_bridge/cmake/cv_bridgeConfig.cmake-bak
sudo cp /opt/ros/indigo/share/image_geometry/cmake/image_geometryConfig.cmake /opt/ros/indigo/share/image_geometry/cmake/image_geometryConfig.cmake-bak

sudo gedit /opt/ros/indigo/lib/pkgconfig/cv_bridge.pc &
sudo gedit /opt/ros/indigo/lib/pkgconfig/image_geometry.pc &
sudo gedit /opt/ros/indigo/share/cv_bridge/cmake/cv_bridgeConfig.cmake &
sudo gedit /opt/ros/indigo/share/image_geometry/cmake/image_geometryConfig.cmake &

3) Modifications for each file
remove each instance “/usr/lib/arm-linux-gnueabihf/libopencv_ocl.so.2.4.8;“
replace each instance of “/usr/lib/arm-linux-gnueabihf/” with “/usr/lib“
replace each instance of “2.4.8” with “2.4.12” (or the current version of OpenCV in opencv4tegra package)

4)After edited, code files like this.

REF: http://myzharbot.robot-home.it/blog/software/ros-nvidia-jetson-tx1-jetson-tk1-opencv-ultimate-guide/

2.2 HACK 2 – opencv

Note about SIFT/SURF in the nonfree module: OpenCV4Tegra doesn’t include the opencv_nonfree package (containing SIFT & SURF feature detectors) since those algorithms are patented by other companies and therefore anyone using opencv_nonfree is at risk of liability.

Please note that opencv4tegra does not include “nonfree” module, so if your algorithms use SIFT or SURF and you want full CUDA support, the only solution is to compile OpenCV by yourself following this guide:
http://elinux.org/Jetson/Installing_OpenCV.

Remember that compiling OpenCV by yourself you will lose Nvidia optimizations on the code running on the CPU that give 3-4 FPS more on heavy algorithms not running on CUDA.

If you need something from the nonfree module, you have 2 options:
1) Analyze the public OpenCV source code then copy/paste the parts of the nonfree module that you want (eg: SURF feature detector) from OpenCV into your own project. You will have the CPU optimizations of OpenCV4Tegra for most of your code and will have the GPU module and will have the non-optimized patented code that you need from the nonfree package such as SURF. So this option gives full performance (for everything except the nonfree code) but is tedious.
2) Ignore OpenCV4Tegra, and instead, download & build public OpenCV (by following the instructions below: for natively compiling the OpenCV library from source). You will still have the GPU module but not any CPU optimizations, but you won’t need to spend time ripping out parts of the OpenCV non-free module code. So this option is easiest but produces slower code if you are running most of your code on CPU.

instructions: Natively compiling the OpenCV library from source onboard the device
Note: Compiling OpenCV from source will not give you NVIDIA’s CPU optimizations that are only available in the closed-source prebuilt OpenCV4Tegra packages.

1) If you haven’t added the “universal” repository to Ubuntu, then do it now:
sudo add-apt-repository universe
sudo apt-get update

2) Now you need to install many libraries:
# Some general development libraries
sudo apt-get -y install build-essential make cmake cmake-curses-gui g++
# libav video input/output development libraries
sudo apt-get -y install libavformat-dev libavutil-dev libswscale-dev
# Video4Linux camera development libraries
sudo apt-get -y install libv4l-dev
# Eigen3 math development libraries
sudo apt-get -y install libeigen3-dev
# OpenGL development libraries (to allow creating graphical windows)
sudo apt-get -y install libglew1.6-dev
# GTK development libraries (to allow creating graphical windows)
sudo apt-get -y install libgtk2.0-dev

3) Download the source code of OpenCV for Linux onto the device.
eg: Open a web-browser to “www.opencv.org” & click on “OpenCV for Linux/Mac”, or from the command-line you can run this on the device:
wget http://downloads.sourceforge.net/project/opencvlibrary/opencv-unix/2.4.10/opencv-2.4.10.zip

4) Unzip the OpenCV source code:
cd Downloads
unzip opencv-2.4.10.zip
mv opencv-2.4.10 ~

5) Configure OpenCV using CMake:
cd opencv-2.4.10/
mkdir build
cd build
cmake -DWITH_CUDA=ON -DCUDA_ARCH_BIN=”3.2″ -DCUDA_ARCH_PTX=”” -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF ..

6) If you want to customize any more of the build settings such as whether to support Firewire cameras or Qt GUI,
it is easiest to use the curses interactive version of CMake from here on:
ccmake ..
(Change any settings you want, then click Configure and Generate).

7) Now you should be ready to build OpenCV and then install it.
Unfortunately, OpenCV is currently experiencing a problem with CMake where installing the built libraries (that normally takes a few seconds) re-compiles the whole OpenCV (that normally takes close to an hour).
So to save time, instead of running “make -j4 ; make install”, we will build & install OpenCV using a single command.

8) To build & install the OpenCV library using all 4 Tegra CPU cores (takes around 40 minutes), including copying the OpenCV library to “/usr/local/include” and “/usr/local/lib”:
sudo make -j4 install

9)Finally, make sure your system searches the “/usr/local/lib” folder for libraries:
echo “# Use OpenCV and other custom-built libraries.” >> ~/.bashrc
echo “export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/” >> ~/.bashrc
source ~/.bashrc

REF: http://elinux.org/Jetson/Installing_OpenCV
REF: http://www.jetsonhacks.com/2015/06/14/ros-opencv-and-opencv4tegra-on-the-nvidia-jetson-tk1/

2.3 HACK3

background advantages that OpenCV4Tegra has versus regular OpenCV.

OpenCV4Tegra is a CPU and GPU accelerated version of the standard OpenCV library. OpenCV stands for “Open Computer Vision”, the de-facto standard Computer Vision library containing more than 2500 computer vision & image processing & machine learning algorithms.
this is for the current 2.4.10 release of OpenCV vs. 2.4.10 release of OpenCV4Tegra.

here are three versions of OpenCV that you can run on the Jetson:
“Regular” OpenCV
OpenCV with GPU enhancements
OpenCV4Tegra with both CPU and GPU enhancements

“Regular” OpenCV is OpenCV that is compiled from the OpenCV repository with no hardware acceleration. This is typically not used on the Jetson, as GPU enhancements are available for OpenCV.
OpenCV with GPU enhancements is designed for CUDA GPGPU acceleration. This is part of the standard OpenCV package.
OpenCV4Tegra is a free, closed source library available from NVIDIA which includes ARM NEON SIMD optimizations, multi-core CPU optimizations and some GLSL GPU optimizations.
So why wouldn’t you always use OpenCV4Tegra? The answer lies in the actual OpenCV library itself; there are two proprietary patented algorithms, SIFT and SURF, which exist in opencv-nonfree. Because these are patented, NVIDIA does not include them in their distribution of OpenCV4Tegra. Therefore if your code does not use SIFT or SURF, then you can use OpenCV4Tegra and get the best performance.
Why use SIFT and/or SURF? The quick answer is that when people are doing feature detection, SIFT/SURF are two of the most popular algorithms in use today. One application is simultaneous Localization And Mapping (SLAM) used mostly in the robotics/drone world. One of the most popular packages of which is Semi-Direct Monocular Visual Odometry (SVO)
Another application which uses SIFT/SURF is deep learning, such as the package Caffe Deep Learning Framework.

Alternatives?
The first alternative is that if you do not have any need of SIFT/SURF in your application, you can use OpenCV4Tegra and enjoy the best performance. There is a rub, but also a possible workaround.
If you need SIFT/SURF you can:
Use OpenCV4Tegra, analyze the public OpenCV source code, and then copy/paste the parts of the nonfree module that you want (eg: SURF feature detector) from OpenCV into your own project. You will have the CPU optimizations of OpenCV4Tegra for most of your code and will have the GPU module and will have the non-optimized patented code that you need from the nonfree package such as SURF. So this option gives full performance (for everything except the nonfree code) but is tedious (and difficult to maintain).
Ignore OpenCV4Tegra, and instead, download & build public OpenCV. You will still have the GPU module but not any CPU optimizations, but you won’t need to spend time ripping out parts of the OpenCV non-free module code. So this option is easiest but produces slower code if you are running most of your code on CPU.

Opinion
If you need SIFT/SURF, then you should just build OpenCV from source, otherwise use OpenCV4Tegra.

Note : OpenCV 3.0 handles SIFT/SURF in a separate repository, opencv_contrib repo.
This may make it easier in the future to combine OpenCV4Tegra with SIFT/SURF, but because OpenCV4Tegra is still at release 2.4.10, this remains to be seen.

2.4 HACK4

ROS的OpenCV的bridge的bug

Installed the grinch kernel + Cuda4Tegra + OpenCV4Tegra + OpenCV4Tegra-dev.
Everything went smoothly until I installed a ros package called “ros-indigo-cv-bridge”, which is useful to translate ROS’ image messages to OpenCV’s matrix format. I broke my package system when trying to install it!

HACK1:
Ros-indigo-cv-bridge depends heavily on libopencv-dev and it seems that OpenCV4Tegra-dev is of no use when apt-get tries to install all dependencies.
I get the following error from apt-get, for every component included in libopencv-dev:
dpkg: error processing archive /var/cache/apt/archives/libopencv-core-dev_2.4.8+dfsg1-2ubuntu1_armhf.deb (–unpack):
trying to overwrite ‘/usr/include/opencv2/core/wimage.hpp’, which is also in package libopencv4tegra-dev 2.4.8.2
So,
my guess is, there must be a way to make apt-get to look into Opencv4Tegra to solve all dependencies when trying to install Ros-Indigo-CV-Bridge, but I don’t know how to do it.
Or,
the apt-get result is completely misleading.
->Don’t know if you solved this…but I ran in to the same trouble here when trying ROS/CV with the Tegra version of OpenCV. I ended up creating a “fake” package (using equivs) that tells apt that libopencv + libopencv-dev is already installed. This worked nicely for me and now I am running the tegra-version of opencv under ROS. Very nice…but a little hackish solution to the problem!
Anyhow, this was the contents of the input file for “equivs-build”:
Section: misc
Priority: optional
Standards-Version: 3.9.2

Package: libopencv-dev-dummy
Version: 2.4.8
Maintainer: yourname <yourname@somemail>
Provides: libopencv-calib3d-dev, libopencv-calib3d2.4,
libopencv-contrib-dev, libopencv-contrib2.4,
libopencv-core-dev, libopencv-core2.4,
libopencv-dev,
libopencv-facedetect-dev, libopencv-facedetect2.4,
libopencv-features2d-dev, libopencv-features2d2.4,
libopencv-flann-dev, libopencv-flann2.4,
libopencv-gpu-dev, libopencv-gpu2.4,
libopencv-highgui-dev, libopencv-highgui2.4,
libopencv-imgproc-dev, libopencv-imgproc2.4,
libopencv-imuvstab-dev, libopencv-imuvstab2.4,
libopencv-legacy-dev, libopencv-legacy2.4,
libopencv-ml-dev, libopencv-ml2.4,
libopencv-objdetect-dev, libopencv-objdetect2.4,
libopencv-ocl-dev, libopencv-ocl2.4,
libopencv-photo-dev, libopencv-photo2.4,
libopencv-softcascade-dev, libopencv-softcascade2.4,
libopencv-stitching-dev, libopencv-stitching2.4,
libopencv-superres-dev, libopencv-superres2.4,
libopencv-video-dev, libopencv-video2.4,
libopencv-videostab-dev, libopencv-videostab2.4,
libopencv-vstab, libopencv-vstab2.4

Description: empty dummy package
no description

This will get you a “dummy-package” that you simply install using “sudo dpkg -i libopencv-dev-dummy_2.4.8_all.deb”. After this, all other packages that depend on opencv will install without trying to install the SW-version of opencv. Make sure you have installed the CUDA version before running this…

Note that the CUDA-version of OpenCV does not contain the nonfree package, i.e. SURF etc. Have not tried to solve that yet…
–> It solved the issue with cv-bridge, but not with each other package that relies on OpenCV.
I tried to install OpenNI2-camera and I went to the starting point. Each deb that has dependency on OpenCV must be modified using this method.

Try for example
sudo apt-get install ros-indigo-rgbd-launch ros-indigo-openni2-camera ros-indigo-openni2-launch

Actually the list of packages to be hacked is the following:
ros-indigo-cv-bridge
ros-indigo-depth-image-proc
ros-indigo-image-geometry
ros-indigo-image-proc
ros-indigo-rqt-image-view

—> Regarding my last comment, ros still thinks to look in usr/lib for the opencv libraries, but they aren’t there. Instead they are in /usr/lib/arm-linux-gnueabihf. I installed _L0g1x_’s fix, but the packages we are using are looking for the opencv libraries in usr/lib, thus giving me an error that they can’t find them. Not sure how to fix this. I thought that the opencv4tegra installed them in usr/lib?
aha, im not sure what packages you are using (please mention) and why it cant find the opencv libraries in /usr/lib since opencv4tegra actually does install into /usr/lib (which it shouldnt, it should install into /usr/lib/arm-linux-gnueabihf , just like the native ros arm opencv install does). My fix accounted for the incorrect path set by the opencv4tegra library, when it should actually be the other way around: The opencv4tegra deb should be modified to be install all the opencv4tegra libraries into /usr/lib/arm-linux-gnueabihf instead of /usr/lib. The issue of an update overwriting the tegra opencv libs will still then exists if you update opencv through a ros update.

—->
The problem currently is that opencv4tegra contain’s packages that install the modified nvidia opencv libs and names them differently then the native opencv libs.
For example:
Nvidia – libopencv4tegra, libopencv4tegra-dev
Native – libopencv, libopencv-dev
This causes a issue for users who use packages that specify in their DEBIAN/control that they depend on libopencv (one package example would be all the ROS computer vision related packages, well at least the ones that use OpenCV).

Inevitably, i know its difficult to not name the modified opencv4tegra libraries different from the native opencv libs to prevent from an upstream opencv update overwriting the opencv4tegra lib modifications. I have a few ideas about a way to possibly fix this that im currently trying out at this moment, but I would also like to hear the input of the opencv4tegra package maintainer on what his thoughts are on dealing with this issue.

!!! I spoke with the OpenCV4Tegra maintainer, and he said there is a temporary work-around you can use for now: !!!
Install ROS
Remove public OpenCV packages within ROS
Install OpenCV4Tegra
Install ROS again

ROS will then work and use the OpenCV4Tegra optimizations. We will try to fix the problem in the next release to enable smooth package replacement without ROS removal.
OpenCV4Tegra is an NVIDIA-only library and so it only gets updated with L4T releases or if you update it manually (downloading OpenCV4Tegra packages from the NVIDIA website). I haven’t tried the suggested ROS work-around above, but if you try it and it fails then let me know and I’ll get more details from them.

!!! My OS verion is Linux tegra-ubuntu 3.10.40 and ROS version is indigo. !!!
I have been using opencv without any issues, specially with CAFFE framework.
I have also followed some tutorials in the jetson (for sanity check) and everything is working allright
Right now I have tried to follow the image transports tutorials (http://wiki.ros.org/image_transport/Tutorials/PublishingImages) but when I run catkin_make I am obtaining this error:
” No rule to make target `/usr/lib/arm-linux-gnueabihf/libopencv_videostab.so.2.4.8′, needed by `/home/ubuntu/image_transport_ws/devel/lib/image_transport_tutorial/my_publisher’. Stop.”

***********************************
OpenCV4Tegra has GPU support and it is optimized for Tegra TK1 SoC, so I want to use its power for my algorithms 😉
— I forgot where I read this, but that dummy package works as is for 19.3, and that you have to slightly modify the dummy package for it to work with 21.2 opencv4tegra.
— Okay so i successfully installed cv-bridge, and can compile some sample code from the cv bridge tutorials, and run the node no problem. There still need to be some modifications, but i made a modified .deb for ros-indigo-cv-bridge and changed around a few things:
–First off, the default armhf deb for ros-indigo-cv-bridge sets different lib paths for setting the libraries in the cmake. For example, in the cv_bridgeConfig.cmake inside the .deb:
set(libraries “cv_bridge;/usr/lib/arm-linux-gnueabihf/libopencv_videostab.so;/usr/lib/arm-linux-gnueabihf/libopencv_video.so;…..
needs to instead be
set(libraries “cv_bridge;/usr/lib/libopencv_videostab.so;/usr/lib/libopencv_video.so;….
I took out the /arm-linux-gnueabihf/ out of the path because opencv4tegra libraries are installed in /usr/lib.
To get the cv-bridge debian so i could edit it, i did the following commands:
sudo apt-get install -d ros-indigo-cv-bridge ## -d just downloads, not install
cd /var/cache/apt/archives
sudo cp ros-indigo-cv-bridge_1.11.6-0trusty-20141201-2058-+0000_armhf.deb ~/Downloads
cd ~/Downloads
mkdir ros-indigo-cv-bridge-extracted
sudo dpkg-deb -R ros-indigo-cv-bridge_1.11.6-0trusty-20141201-2058-+0000_armhf.deb ros-indigo-cv-bridge-extracted
All dpkg-deb -Rdoes is extracts the .deb WITH the DEBIAN folder so that you can edit the DEBIAN/control file. In the control file, i deleted a few things: libopencv-dev, libopencv-python, libopencv-core2.4, libopencv-imgproc2.4, since these were all already installed by opencv4tegra debian.
Once i edited all those things, i think built the package like so:
sudo dpkg-deb -b ros-indigo-cv-bridge-extracted ros-indigo-cv-bridge-tegra_1.11.6-l0g1x-2.deb
and then finally just use sudo dpkg -i ros-indigo-cv-bridge-tegra_1.11.6-l0g1x-2.deb to install it.
I dont think i missed any steps, but attached is where the .deb file i made is. Just use dpkg -i to install it (after CUDA and opencv4tegra have been installed)
It would be nice if the ros arm buildfarm actually had a cv-bridge debian for the jetson.. maybe?
EDIT1: I wanted to clarify when i edit the DEBIAN/control file; when i say i “removed” libopencv-dev, libopencv-python, libopencv-core2.4, libopencv-imgproc2.4, all the re-movement does is remove what dependencies the package installer should check for;
EX) if there is a dependency listed in the control file, and the package manager sees that it is not installed on the system it will try to install that dependency (separately, like as its own .deb file). So since we know that libopencv-dev, libopencv-python, libopencv-core2.4, libopencv-imgproc2.4 is already installed from the opencv4tegra .deb , we can remove them from the ‘Depends:’ line in DEBIAN/control
EDIT 2: Steps to take if you just download the .deb i made:
sudo dpkg -i .deb
sudo apt-get update
sudo apt-get install libopencv4tegra libopencv4tegra-dev
EDIT 3: The modified cv-bridge.deb i made is only a quick fix/hack. I am currently working on making a permanent fix by modifying just the oepncv4tegra.deb, so that you wont have to use the quick fix cv-bridge hack and can update cv-bridge whenever with apt-get upgrade. Should have this done within the next day or two. For now i have rearranged things around so opencv4tegra libs actually in fact go in the /usr/lib/arm-linux-gnueabihf/ folder where they should be. Im trying to see if i can get this to work without a dummy package, but if there isnt another way, the dummy package will be included inside the opencv4tegra-modified.deb so that it will automatically install with everything.
REF: http://answers.ros.org/question/202643/opencv4tegra-and-ros-howto-jetson-tk1/

3、深度学习机器人

深度学习机器人组成是深度学习TK1 + 机器人Turtlebot:

artificial robot

The Kobuki mobile base is by the Korean firm Yujin Robot. The mobile base has two wheels, IR range and cliff sensors, a factory-calibrated gyroscope, a built-in rechargeable battery and various ports for powering the rest of the robot and for communications.
The nVidia Jetson TK1 is a small embedded PC, rather like a souped-up Raspberry Pi.
The Kinect is a popular peripheral for those frittering away their time with the xBox.

blocks

3.1 wifi
检查是哪一个接口来支持无线连接的:
$ iwconfig

first, check the WiFi is working with the Network Manager by typing:
$ nmcli dev
DEVICE TYPE STATE
eth2 802-3-ethernet connected
wlan2 802-11-wireless disconnected

Now connect to local 2.4Ghz Wifi network (I couldn’t get it to work with a 5Ghz one) by typing:
$ sudo nmcli dev wifi connect password

If your SSID has spaces in, then enclose it in quotes, e.g. ‘My network’. As usual with sudo commands you’ll be asked to authenticate with the ‘ubuntu’ password. Assuming you have no error message, then typing

all you need is the IP address of the WiFi interface:
$ ifconfig

try logging in via the WiFi interface with the IP from the last step:
$ ssh ubuntu@10.0.1.2

test robot
$ roslaunch turtlebot_bringup minimal.launch

$ roslaunch turtlebot_teleop keyboard_teleop.launch

Testing Caffe
Caffe is a tool for creating and running CUDA-accelerated neural networks. The most important thing it can potentially do for your robot is allow it to recognise objects in photographs or video streams.

3.2 ds card
增加SD card

The Deep Learning TK1 comes with 16Gb built-in flash on the Jetson TK1 board. That’s fine to begin with, but after downloading a few Caffe models, you’ll be out of space.
Fortunately, the TK1 comes with an SD Card slot for adding extra storage. This post describes how to add and configure an SD Card to give yourself more room.

Choose the fastest, biggest SD Card eg SanDisk 64Gb SD Card for around $60.The card is a class 10 / U3 designed for 4k video files and claims to have a read-write speed of 90Mb/s.

分区clk设备,增加主分区:
$ sudo fdisk /dev/mmcblk0
-a
-p
-1
-w
$ lsblk
— mmcblk1 179:32 0 29.7G 0 disk

制作文件系统:
sudo mkfs.ext4 /dev/mmcblk1p1

Done

$ sudo blkid
/dev/mmcblk1: UUID=”d417ef49-09d9-4fd2-9351-e0e1413a2f8f” TYPE=”ext4″
/dev/mmcblk1p1: UUID=”fd7a0700-efaf-47a5-a118-9202607b46e8″ TYPE=”ext4″

制作安装点:
sudo mkdir /media/sdmount

修改fatab:
The /etc/fstab file contains a list of devices that need mounting on each boot. We’re going to add the mount point for the card to this file, so it gets automatically mounted each time the robot is switched on.
sudo cp /etc/fstab /etc/fstab.orig
sudo vim /etc/fstab
showing only:
# UNCONFIGURED FSTAB FOR BASE SYSTEM
At the end of the file add a line with this format:
UUID= /media/sdmount ext4 defaults,users 0 0
Let’s unpack this. We’re telling the system that we want to mount a partition with the specified UUID at the mount point /media/sdmount (which we just created). ‘ext4’ specifies the filesystem type, which we formatted earlier. The options defaults,users sets the partition with read-write permissions for all users (see more options here under “Filesystem Independent Mount Options”). The final two parameters which are both zero specify whether we want to dump or auto check the filesystem (more details under “Editing Ubuntu’s filesystem table”).

安装:
mount all devices specified in /etc/fstab:
sudo mount -a

Now able to access the card at the mount point. Type:
ls /media/sdmount

建立连接点:
Create a symbolic link to your user directory
cd ~
sudo mkdir /media/sdmount/sdcard

Then change the ownership of the target directory so the ubuntu user can read and write to it:
sudo chown ubuntu :media/sdmount/sdcard

sudo chown ubuntu ~/sdcard

Finally, link it:
ln -s /media/sdmount/sdcard sdcard

Reboot .
Finally reboot and make sure the subdirectory is present and working.

实验环境是:
mkdir ~/sdcard
sudo mount /dev/mmcblk1p1/ ~/sdcard
if needed:
sudo chown ubuntu:ubuntu ~/sdcard
ifnotused
sudo umount /dev/mmcblk1p1

REF: http://www.artificialhumancompanions.com/adding-sd-card-deep-learning-robot/

3.3 Kubiko
$ sudo apt-get install ros-indigo-turtlebot
//ros-indigo-turtlebot-apps ros-indigo-turtlebot-interactions ros-indigo-turtlebot-simulator ros-indigo-kobuki-ftdi ros-indigo-rocon-remocon ros-indigo-rocon-qt-library ros-indigo-ar-track-alvar-msgs

this uses 100M / 400M space. YES

whiling setting up ros-indigo-librealsense, give ERROR: Module uvcvideo not found.
So setup error.
dependence is: ros-indigo-turtlebot –> ros-indigo-librealsense –> ros-indigo-librealsense-camera –> uvcvideo

RealSense 3D是一套感知计算解决方案,包括世界上最小的3D摄像头.RealSense camera support under ROS is still relatively new
maybe need to recompile the Grinch kernel with the necessary UVCVideo patches

A) First, librealsense needs to be installed on the Jetson TK1. This involves building a new kernel to support the video modes of the RealSense camera in the UVC module, and building the librealsense library.
1.1 First, operating system level files must be modified to recognize the camera video formats. When doing development on Linux based machines you will frequently hear the terms “kernel” and “modules”. The kernel is the code that is the base of the operating system, the interface between hardware and the application code.
A kernel module is code that can be loaded into the kernel image at will, without having to modify the kernel. These modules provide ancillary support for different types of devices and subsystems. The code for these modules is either in the kernel itself, in which case it is called ‘built-in’, or designated to built as a module. When built as a module the compiled code is stored separately from the kernel, typically with a .ko extension. The advantage of having a module is that it can be easily changed without having to rebuild the entire kernel.
We will be building a module called uvcvideo to help interface with the RealSense camera.
A convenience script has been created to help with this task in the installLibrealsense repository on the JetsonHacks Github account.
$ git clone https://github.com/jetsonhacks/installLibrealsense.git
$ cd installLibrealsense/UVCKernelPatches
$ ./getKernelSources.sh
Once the kernel sources have been downloaded and decompressed into the /usr/src directory, a configuration editor opens. In the configuration editor, set the local version of the kernel to that of the current configuration. The current local version number is available through the command:
$ uname -r
which displays:
3.10.40-gdacac96
The local version number consists of the digits following the 40 in this case, i.e. -gdacac96.
Remember the – sign, it is important! This identifier is used to ensure that the module matches the build of the kernel and should match exactly. Place the local version number in the field: General Setup -> Local version – append to kernel release:

Next, we will be modify the USB Video Class (UVC) to understand RealSense video formats. The option to compile UVC as a module is located in:
Device Drivers -> Multimedia Support -> Media USB Adapters -> USB Video Class (UVC)

Once you find the entry, right-click on the entry until you see a small circle. The circle indicates that the option will be compiled as a module. Save the configuration file.

A patch file is provided to apply on the module source and a shell script is provided to apply the patch. Again, these are convenience files, you may have to modify them for your particular situation.
$ ./applyUVCPatch.sh

Next, compile the kernel and module files:
$ ./buildKernel.sh

This takes several minutes as the kernel and modules are built and the modules installed. Once the build is complete, you have a couple of options. The first option is to make a backup of the new kernel and modules to place them on a host system to flash a Jetson system with the new kernel. We will not be covering that here, but for reference:
The second option is to copy the kernel over to the boot directory. A convenience script is provided:
$ ./copyzImages.sh

In addition to copying the new kernel into the boot directory, the newly built module, uvcvideo, is added to the file /etc/modules to indicate that the module should be loaded at boot time.

The RealSense cameras require USB 3.0. The USB port is set for USB 2.0 from the factory. Also, the stock kernel uses what is called ‘autosuspend’ to minimize power usage for USB peripherals. This is incompatible with most USB video devices. If you have not changed these settings on your TK1, a convenience script has been provided:
$ ./setupTK1.sh

Now reboot the system.

Once the machine has finished rebooting, open a Terminal:
$ cd installLibrealsense
$ ./installLibrealsense.sh

This will build librealsense and install it on the system. This will also setup udev rules for the RealSense device so that the permissions will be set correctly and can be accessed from user space. Once installation is complete, you will be able to play with the examples. For example:
$ cd ~/librealsense/bin
$ ./cpp-config-ui

The example allows you to set the camera parameters. Hit the ‘Start Capture’ button to start the camera.
At the end of the RealSense camera installation, you should run some of the examples provided to make sure that the camera is installed and works properly.

Qt Creator
There are Qt Creator files in librealsense which may be used to build the examples and librealsense itself. A convenience script, ./installQtCreator.sh , is provided in the installLibrealsense directory to install Qt Creator.

Note: In an earlier article article on installing ROS on the Jetson TK1, we used the Grinch kernel. The Grinch kernel provides access to a large number of peripherals on the TK1. Because we modified the stock kernel for the RealSense camera in our first step, the Grinch kernel is not used here. If you need the Grinch kernel, you will have to recompile the Grinch kernel with the RealSense camera changes. That is an excercise left to the reader.

2.2 the second part of getting the R200 to work is to build and install librealsense.

B) Second, we need to have ROS installed on the Jetson. Once the RealSense camera is working, from a new Terminal install ROS:

REF: http://www.jetsonhacks.com/2016/06/20/intel-realsense-camera-installation-nvidia-jetson-tk1/

C) Third, we download the realsense_camera package installer:
Open a new Terminal, which will source the new environment setup by ROS, and:
$ git clone https://github.com/jetsonhacks/installRealSenseCameraROS.git
$ cd installRealSenseCameraROS

We need is a Catkin Workspace for our base of operations. There is a convenience script to create a new Catkin Workspace.
$ ./setupCatkinWorkspace [workspace name]

With the prerequisites installed, we’re ready to install the realsense_camera package:
$ ./installRealSense.sh [workspace name]

If you do not have a swap file enabled on your Jetson, there may be issues compiling the package because the TK1 does not have enough memory to compile this in one pass. The installation script has been changed since the video was filmed to compile using only one core to relieve memory pressure, i.e.
$ catkin_make -j1″

TK1之lenet -11

基于caffe的prototxt和solver结构,
采用mnist的训练图片和测试图片,
创建神经网络的结构和配置,
实现手写数字识别。

2 IPython – qtconsole – Notebook

2.1 IPython

Python的验证调试pdb在其ide里面是功能微弱的, 为此有IPython这一利器:
自动补全功能,使用tab键,如输入im后按tab键,自动补全import。
$ sudo apt-get install ipython
$ sudo apt-get install python-zmq

$ ipython
即可进入互式IDE编程环境。

可以使用使用魔法指令% :
%run test.py 直接运行python脚本
%pwd: 显示当前工作目录。
%cd: 切换工作目录。

后面的qt和book均使用了matplotlib这个著名的Python图表绘制扩展库,
支持输出多格式图像,可以使用多种GUI界面库交互式地显示图表。
这里用它的inline这个嵌入参数把matplotlib图表的显示在qtconsole或者notebook里面, 而不是单独弹出一个窗口。

2.2 qtconsole

IPython团队开发了一个基于Qt框架,目的是为终端应用程序提供诸如内嵌图片、多行编辑、语法高亮之类的富文本编辑功能的GUI控制台。
好处是并不出现shell控制台在背后运行,只有ipython的qt控制台运行。
$ sudo apt-get install ipython-qtconsole

$ ipython qtconsole –pylab=inline

2.3 Notebook

或者可以使用nootebook:
$ sudo apt-get install ipython-notebook

$ ipython notebook –pylab=inline
出现界面后,可以输入多行代码,shift+Enter运行即可。
Shift-Enter : run cell
Ctrl-Enter : run cell in-place
Alt-Enter : run cell, insert below

2.4 说明

iPython调试, 虽然可以代码块,方便的很;
但是如果出错也是很间接, kenel restarting之类的,都看不清

很多时候, 在python的一行行pdb调试,有时还是不可替代的。

后面的调试里, 从哪里启动python特别重要:
$ cd …CAFE_ROOT…
$ ipython or python … …
否则,不然又很多路经问题出错。

如果:
db_lmdb.hpp:14] Check failed: mdb_status == 0 (2 vs. 0) No such file or directory
这个是train和test的prototxt里面的data source 位置不对。
如果:
caffe solver.cpp:442] Cannot write to snapshot prefix ‘examples/mnist/lenet’.
这个可以是路径也可以是把lenet改成任一个zenet
如果:
ipython “Kernel Restarting” about solver = caffe.SGDSolver(‘mnist/lenet_auto_solver.prototxt’)
可能要在solver里面最后显示加上:
solver_type: GPU

3 设置prototxt

3.0
$ cd /home/ubuntu/sdcard/caffe-for-cudnn-v2.5.48
$ ipython qtconsole –pylab=inline

3.1 配置Python环境

# we’ll use the pylab import for numpy and plot inline
from pylab import *
%matplotlib inline

# Import caffe, adding it to sys.path if needed. (Make sure’ve built pycaffe)
# caffe_root = ‘../’ # It means this bash file should be run from {caffe_root}/examples/

import sys
caffe_root = ‘.’
sys.path.insert(0, caffe_root + ‘python’)

import caffe

3.2 挂于数据

可选动作, 如果ok不需要。

# Using LeNet example data and networks, if already downloaded skip this step.
## run scripts from caffe root
## import os
## os.chdir(caffe_root)
## Download data
## !data/mnist/get_mnist.sh
## Prepare data
## !examples/mnist/create_mnist.sh
## back to examples
## os.chdir(‘examples’)

3.3 关于网络

可选动作,如果ok不需要。

这个train和test网络文件是caffe安装时候提供了的,直接用即可。
也自己生成, 以下是建一个Leet的变体。

# We’ll write the net in a succinct and natural way as Python code that serializes to Caffe’s protobuf model format.
# This network expects to read from pre-generated LMDBs,
# but reading directly from ndarrays is also possible using MemoryDataLayer.

# We’ll need two external files:
# the net prototxt, defining the architecture and pointing to the train/test data
# the solver prototxt, defining the learning parameters
from caffe import layers as L, params as P
def lenet(db_path,batch_size):
n=caffe.NetSpec()
n.data,n.label=L.Data(batch_size=batch_size,backend=P.Data.LMDB,source=db_path, transform_param=dict(scale=1./255),ntop=2)
n.conv1=L.Convolution(n.data,kernel_size=5,num_output=20,weight_filler=dict(type=’xavier’))
n.pool1=L.Pooling(n.conv1,kernel_size=2,stride=2,pool=P.Pooling.MAX)
n.conv2=L.Convolution(n.pool1,kernel_size=5,num_output=50,weight_filler=dict(type=’xavier’))
n.pool2=L.Pooling(n.conv2,kernel_size=2,stride=2,pool=P.Pooling.MAX)
n.fc1 =L.InnerProduct(n.pool2,num_output=500,weight_filler=dict(type=’xavier’))
n.relu1=L.ReLU(n.fc1,in_place=True)
n.score=L.InnerProduct(n.relu1,num_output=10,weight_filler=dict(type=’xavier’))
n.loss=L.SoftmaxWithLoss(n.score,n.label)
return n.to_proto()

# write net to disk in a human-readable serialization using Google’s protobuf
# You can read, write, and modify this description directly
with open(‘examples/mnist/lenet_auto_train.prototxt’, ‘w’) as f: # this is train
f.write(str(lenet(‘examples/mnist/mnist_train_lmdb’, 64)))
with open(‘examples/mnist/lenet_auto_test.prototxt’, ‘w’) as f: # this is solver
f.write(str(lenet(‘examples/mnist/mnist_test_lmdb’, 100)))

这样train和test网络结构写入介质
名字可以自定义,但是要符合solver.prototxt里面的配置。
# you can view tranin net struct geneted pre-step
# $ vi examples/mnist/lenet_auto_train.prototxt
# you can view test net struct geneted pre-step
# $ vi examples/mnist/lenet_auto_test.prototxt

3.4 关于求解器

可选动作,如果ok不需要。

这个lenet_auto_solver.prototxt文件是caffe安装时候提供了的,
直接用即可。

也自己生成这个solver.prototxt.
from caffe.proto import caffe_pd2
s=caffe_pb2.SolverParameter()
s.random_seed=0
#下面格式参数与之前看到的相似
… …
#最后
with open(yourpath,‘w‘)as f:
f.write(str(s))

然后与上面类似
solver=None
solver=caffe.get_solver(yourpath)

这个solver文件大概像这样:
# $ vi examples/mnist/lenet_auto_solver.prototxt
# The train/test net protocol buffer definition
train_net: “examples/mnist/lenet_auto_train.prototxt”
test_net: “examples/mnist/lenet_auto_test.prototxt”

# test_iter specifies how many forward passes the test should carry out.
# In the case of MNIST, we have test batch size 100 and 100 test iterations,
# covering the full 10,000 testing images.
test_iter: 100

# Carry out testing every 500 training iterations.
test_interval: 500

# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005

# The learning rate policy
lr_policy: “inv”
gamma: 0.0001
power: 0.75

# Display every 100 iterations
display: 100

# The maximum number of iterations
max_iter: 10000

# snapshot intermediate results
snapshot: 5000
snapshot_prefix: “mnist/lenet”

3.5 加载求解器

# Let’s pick a device and load the solver
caffe.set_device(0)
caffe.set_mode_gpu() #using GPU

# load the solver
solver=None
# create train and test nets
# We’ll use SGD (with momentum), but other methods are also available.
solver = caffe.SGDSolver(‘examples/mnist/lenet_auto_solver.prototxt’)

说明:这个caffe的优化函数是非凸的,没有解析解,需要通过优化方法来求解。

3.6 检查网络

检查输出shape | 查看中间特征(blobs) | 参数(params)的维数
# each output is (batch size, feature dim, spatial dim)

中间特征(blobs)
[(k, v.data.shape) for k, v in solver.net.blobs.items()]
… …
[(‘data‘, (64, 1, 28, 28)),
(‘label‘, (64,)),
(‘conv1‘, (64, 20, 24, 24)),
(‘pool1‘, (64, 20, 12, 12)),
(‘conv2‘, (64, 50, 8, 8)),
(‘pool2‘, (64, 50, 4, 4)),
(‘fc1‘, (64, 500)),
(‘score‘, (64, 10)),
(‘loss‘, ())]

参数(params)
# just print the weight sizes (we’ll omit the biases)
[(k, v[0].data.shape) for k, v in solver.net.params.items()]
… …
[(‘conv1‘, (20, 1, 5, 5)),
(‘conv2‘, (50, 20, 5, 5)),
(‘fc1‘, (500, 800)),
(‘score‘, (10, 500))]

3.7 执行一次

这个在qtconsole会crash, 在notebook不会, python更不会。

在测试集和训练集上执行一个前向的过程
# check that everything is loaded as we expect,
# by running a forward pass on the train and test nets,
# and check that they contain our data.

# step does one full iteration, covering all three phases: forward evaluation, backward propagation, and update.
# forward does only the first of these.
# Step does only one iteration: a single batch of 100 images. If doing a full set of 20 batches (all 2000 inputs) is called an epoch

训练集
solver.net.forward() # train net
… …
{‘loss’: array(2.363983154296875, dtype=float32)}

测试集
solver.test_nets[0].forward() # test net (there can be more than one)
… …
{‘loss’: array(2.365971088409424, dtype=float32)}

显示训练集 8个数据的图像和他们的标签
imshow(solver.net.blobs[‘data’].data[:8,0].transpose(1,0,2).reshape(28,8*28),cmap=”gray”);axis(‘off’)

print ‘wewewerth’,solver.net.blobs[‘label’].data[:8]
… …
wewewerth [ 5. 0. 4. 1. 9. 2. 1. 3.]

显示测试集中的8个图像和他们的标签
imshow(solver.test_nets[0].blobs[‘data’].data[:8, 0].transpose(1, 0, 2).reshape(28, 8*28), cmap=’gray’)

print solver.test_nets[0].blobs[‘label’].data[:8]
… …
labels [ 7. 2. 1. 0. 4. 1. 4. 9.]

3.8 运行网络

前述确定载入了正确的数据及标签,开始运行solver,运行一个batch看是否有梯度变化。
Both train and test nets seem to be loading data, and to have correct labels.

执行一步SGB, 查看权值的变化
# Let’s take one step of (minibatch) SGD and see what happens.
solver.step(1)

第一层权值的变化如下图:20个5×5规模的滤波器
# Do we have gradients propagating through our filters?
# Let’s see the updates to the first layer, shown here as a 4*5 grid of 5*5 filters.
imshow(solver.net.params[‘conv1′][0].diff[:,0].reshape(4,5,5,5).transpose(0,2,1,3).reshape(4*5,5*5),cmap=’gray’);axis(‘off’)
… …

(-0.5, 24.5, 19.5, -0.5)
运行网络,这个过程和通过caffe的binary训练是一样的

3.9 自定义循环开始训练

# Let’s run the net for a while, keeping track of a few things as it goes.
# Note that this process will be the same as if training through the caffe binary :
# *logging will continue to happen as normal
# *snapshots will be taken at the interval specified in the solver prototxt (here, every 5000 iterations)
# *testing will happen at the interval specified (here, every 500 iterations)
# Since we have control of the loop in Python, we’re free to do many other things as well, for example:
# *write a custom stopping criterion
# *change the solving process by updating the net in the loop

上篇使用的是%timeit
%%time
niter=200
test_interval=25

# losses will also be stored in the log
train_loss=zeros(niter)
test_acc=zeros(int(np.ceil(niter/test_interval)))
output=zeros((niter,8,10))

# the main solver loop
for it in range(niter):
solver.step(1) # SGD by Caffe
# store the train loss
train_loss[it]=solver.net.blobs[‘loss’].data
# store the output on the first test batch
# (start the forward pass at conv1 to avoid loading new data)
solver.test_nets[0].forward(start=’conv1′)
output[it]=solver.test_nets[0].blobs[‘score’].data[:8]
# run a full test every so often
# (Caffe can also do this for us and write to a log, but we show here
# how to do it directly in Python, where more complicated things are easier.)
if it % test_interval ==0:
print ‘iteration’, it, ‘ testing…’
correct=0
for test_it in range(100):
solver.test_nets[0].forward()
correct+=sum(solver.test_nets[0].blobs[‘score’].data.argmax(1)
==solver.test_nets[0].blobs[‘label’].data)
test_acc[it//test_interval]=correct/1e4
… …
iteration 0 testing…
iteration 25 testing…
iteration 50 testing…
iteration 75 testing…
iteration 100 testing…
iteration 125 testing…
iteration 150 testing…
iteration 175 testing…
CPU times: user 19.4 s, sys: 2.72 s, total: 22.2 s
Wall time: 20.9 s

画出训练样本损失和测试样本正确率
# plot the train loss and test accuracy
_, ax1 = subplots()
ax2 = ax1.twinx()
ax1.plot(arange(niter), train_loss)
ax2.plot(test_interval * arange(len(test_acc)), test_acc, ‘r’)
ax1.set_xlabel(‘iteration’)
ax1.set_ylabel(‘train loss’)
ax2.set_ylabel(‘test accuracy’)
ax2.set_title(‘Test Accuracy: {:.2f}’.format(test_acc[-1]))
… …

# Above pic shows the loss seems to have dropped quickly and coverged (except for stochasticity)

画出分类结果
# Since we saved the results on the first test batch, we can watch how our prediction scores evolved.
# We’ll plot time on the x axis and each possible label on the y, with lightness indicating confidence.
for i in range(8):
figure(figsize=(2,2))
imshow(solver.test_nets[0].blobs[‘data’].data[i,0],cmap=’gray’)
figure(figsize=(20,2))
imshow(output[:100,i].T,interpolation=’nearest’,cmap=’gray’) #output[:100,1]. first 100 results

NoTE:
# 上面结果开起来不错,再详细的看一下每个数字的得分是怎么变化的
We started with little idea about any of these digits, and ended up with correct classifications for each.
If you’ve been following along, you’ll see the last digit is the most difficult, a slanted “9” that’s (understandably) most confused with “4”.

使用softmax分类
# Note that these are the “raw” output scores rather than the softmax-computed probability vectors.
# below, make it easier to see the confidence of our net (but harder to see the scores for less likely digits).
for i in range(8):
figure(figsize=(2, 2))
imshow(solver.test_nets[0].blobs[‘data’].data[i, 0], cmap=’gray’)
figure(figsize=(10, 2))
imshow(exp(output[:50, i].T) / exp(output[:50, i].T).sum(0), interpolation=’nearest’, cmap=’gray’)
xlabel(‘iteration’)
ylabel(‘label’)
… …
输出结果图片对比更明显, 这是上面代码的计算方式能够把低的分数和高的分数两极分化:
/*
for i in range(2):
figure(figsize=(2,2))
imshow(solver.test_nets[0].blobs[‘data’].data[i,0],cmap=’gray’)
figure(figsize=(20,2))
imshow(exp(output[:100,i].T)/exp(output[:100,i].T).sum(0),interpolation=’nearest’,cmap=’gray’)
*/

3.10 下一步

# now we’ve defined, trained, and tested LeNet
# there are many possible next steps:
1. 定义新的结构(如加全连接层,改变relu等)
2. 优化lr等参数 (指数间隔寻找如0.1 0.01 0.001)
3. 增加训练时间
4. 由sgd–》adam
5. 其他

http://caffe.berkeleyvision.org/tutorial/interfaces.html
blog.csdn.net/thystar/article/details/50668877
https://github.com/BVLC/caffe/blob/master/examples/00-classification.ipynb
https://github.com/BVLC/caffe/blob/master/examples/01-learning-lenet.ipynb

TK1之mnist -09

深度学习有几个有名的开源框架:Caffe,Theano,Torch,其中caffe的体系是:
* Caffe has several dependencies:
-CUDA is required for GPU mode. Caffe requires the CUDA nvcc compiler to compile its GPU code and CUDA driver for GPU operation.
-BLAS via ATLAS, MKL, or OpenBLAS. Caffe requires BLAS as the backend of its matrix and vector computations. There are several implementations of this library:ATLAS, Intel MKL or OpenBLAS.
-Boost, versoin >= 1.55
-protobuf, glog, gflags, hdf5
* Optional dependencies:
-OpenCV >= 2.4 including 3.0
-IO libraries: lmdb, leveldb (note: leveldb requires snappy)
-cuDNN for GPU acceleration (v5) , For best performance, Caffe can be accelerated by NVIDIA cuDNN.
* Pycaffe and Matcaffe interfaces have their own natural needs.
-For Python Caffe: Python 2.7 or Python 3.3+, numpy (>= 1.7), boost-provided boost.python,
The main requirements are numpy and boost.python (provided by boost). pandas is useful too and needed for some examples.
-For MATLAB Caffe: MATLAB with the mex compiler. Install MATLAB, and make sure that its mex is in your $PATH.

1 – Mnist

Caffe两个demo分别是mnist和cifar10,尤其是mnist,称为caffe编程的hello world。
mnist是个知名的手写数字数据库,据说是美国中学生手写的数字,由Yan LeCun进行维护。
mnist最初用于支票上的手写数字识别, 现在成了DL的入门练习库。
对mnist识别的专门模型是Lenet,是最早的cnn模型。

一般caffe的第一个hello world 都是对手写字体minist进行识别,主要三个步骤:
准备数据、修改配置、使用模型。
REF: http://yann.lecun.com/exdb/mnist/
http://caffe.berkeleyvision.org/gathered/examples/mnist.html

1.1 准备数据

* 下载数据:
$ ./data/mnist/get_mnist.sh

完成后,在 data/mnist/目录下有四个文件:
train-images-idx3-ubyte: 训练集样本 (9912422 bytes)
train-labels-idx1-ubyte: 训练集对应标注 (28881 bytes)
t10k-images-idx3-ubyte: 测试集图片 (1648877 bytes)
t10k-labels-idx1-ubyte: 测试集对应标注 (4542 bytes)

mnist数据训练样本为60000张,测试样本为10000张,每个样本为28*28大小的黑白图片,手写数字为0-9,因此分10类。

* 转换数据:
前述数据不能在caffe中直接使用,需要转换成LMDB数据:
$ ./examples/mnist/create_mnist.sh

转换成功后,会在 examples/mnist/目录下,生成两个文件夹,分别是:mnist_train_lmdb 和 mnist_test_lmdb, 里面存放的data.mdb和lock.mdb,就是我们需要的运行数据。

1.2 关于protobuf

Protocol Buffers简称为protobuf,是由google开源的一种轻便高效的结构化数据存储格式,可以用于结构化数据的串行化或者说序列化。protobuf完成的事情XML也可以,就是把某种数据结构的信息以某种格式保存起来,用于存储、传输协议等场合。
不用XML而另起炉灶一个是考虑,另外是protobuf有比较好的多语言支持(C++、Java、Python和生成机制以及其他一些内容体。

Protocol Buffers这种协议接口的结构很简单,类似:
package caffe;# 定义名称空间
message helloworld #定义类
{ #定义 filed
required int32 xx = 1; // 必须有的值
optional int32 xx = 2; //可选值
repeated xx xx=3; //可重复的
enum xx { #定义枚举类
xx =1;
}
}

例如,开发语言为C++,场景为模块A通过socket发送巨量订单信息给模块B。
则先写一个proto文件,例如Order.proto,在该文件添加名为”Order”的message结构:
message Order
{
required int32 time = 1;
required int32 userid = 2;
required float price = 3;
optional string desc = 4;
}

然后即可用protobuf内置编译器编译该Order.proto:
$ protoc -I=. –cpp_out=. ./Order.proto
protobuf会自动生成Order.pb.cc和Order.pb.h文件。

在发送方模块A, 就可以使用下面代码来序列化/解析该订单包装类:
$ vi Sender.cpp
Order order;
order.set_time(XXXX);
order.set_userid(123);
order.set_price(100.0f);
order.set_desc(“a test order”);
string sOrder;
order.SerailzeToString(&sOrder);
// 调用socket通讯库把序列化的字符串发送出去

在接收方模块B, 可以这样解析:
$ vi Receiver.cpp
string sOrder;
// 先通过网络通讯库接收到数据,存放到某字符串sOrder
Order order;
if(order.ParseFromString(sOrder)) // 解析该字符串
{
cout << "userid:" << order.userid() << endl << "desc:" << order.desc() << endl; } else cerr << "parse error!" << endl; 最后编译文件即可: $ g++ Sender.cpp -o Sender Order.pb.cc -I /usr/local/protobuf/include -L /usr/local/protobuf/lib -lprotobuf -pthread 测试下。。。 1.3 关于 caffe.proto 文件caffe.proto中就是用google protobuf定义的caffe所要用到的结构化数据。 在同级文件夹下还有两个文件caffe.pb.h 和caffe.pb.cc,这两个文件是由caffe.proto编译而来的。 $ vi src/caffe/proto/caffe.proto syntax = "proto2"; //默认就是二 package caffe; //各个结构封装在caffe包中,可以通过using namespace caffe; 或者caffe::**来调用 注意: proto文件为结构、prototext文件为配置。 1.4 caffe结构 caffe通过layer-by-layer的方式逐层定义网络,从开始输入到最终输出判断从下而上的定义整个网络。 caffe可以从四个层次来理解:Blob、Layer、Net、Solver,这四个部分的关系是: * Blob blob是贯穿整个框架的数据单元。存储整个网络中的所有数据(数据和导数),它在cup和GPU之间按需分配内存开销。Blob作为caffe基本的数据结构,通常用四维矩阵 Batch×Channel×Height×Weight表示,某一坐标(n,k,h,w)的物理位置为((n * K + k) * H + h) * W + w),存储了网络的神经元激活值和网络参数,以及相应的梯度(激活值的残差和dW、db)。其中包含有cpu_data、gpu_data、cpu_diff、gpu_diff、mutable_cpu_data、mutable_gpu_data、mutable_cpu_diff、mutable_gpu_diff这一堆很像的东西,分别表示存储在CPU和GPU上的数据。其中带data的里面存储的是激活值和W、b,diff中存储的是残差和dW、db,另外带mutable和不带mutable的一对指针所指的位置是相同的,只是不带mutable的只读,而带mutable的可写。 * Layer Layer代表了神经网络中各种各样的层,组合成一个网络。一般一个图像或样本会从数据层中读进来,然后一层一层的往后传。除了比较特殊的数据层之外其余大部分层都包含4个函数:LayerSetUp、Reshape、Forward、Backward。LayerSetup用于初始化层,开辟空间,填充初始值等。Reshape对输入值进行维度变换。Forward是前向传播,Backward是后向传播。 那么数据是如何在层之间传递的呢?每一层都会有一个(或多个)Bottom和top,分别存储输入和输出。bottom[0]->cpu_data()是存输入的神经元激活值,top就是存输出的,cpu_diff()存的是激活值的残差,gpu是存在GPU上的数据。如果这个层前后有多个输入输出层,就会有bottom[1],bottom[2]。。。每层的参数会存在this->blobs_里,一般this->blobs_[0]存W,this->blobs_[1]存b,this->blobs_[0]->cpu_data()存的是W的值,this->blobs_[0]->cpu_diff()存的梯度dW,b和db也类似,然后换成gpu是存在GPU上的数据,再带上mutable就可写了
凡是能在GPU上运算的层都会有名字相同的cpp和cu两个文件,cu文件中运算时基本都调用了cuda核函数。
* Net
Net就是把各种Layer按train_val.prototxt的定义堆叠在一起。先进行每个层的初始化,然后不断进行Update,每更新一次就进行一次整体的前向传播和反向传播,然后把每层计算得到的梯度计算进去,完成一次更新。注意每层在Backward中只是计算dW和db,而W和b的更新是在Net的Update里最后一起更新的。在caffe里训练模型的时候一般会有两个Net,一个train一个test。
* Solver
Solver是按solver.prototxt的参数定义对Net进行训练。先会初始化一个TrainNet和一个TestNet,然后其中的Step函数会对网络不断进行迭代。主要就是两个步骤反复迭代:不断利用ComputeUpdateValue计算迭代相关参数,比如计算learning rate,把weight decay加上什么;以及调用Net的Update函数对整个网络进行更新。

大致流程是:
sovler通过sovler.prototxt的参数配置初始化Net。 然后Net通过调用trainval.prototxt这些参数,来调用对应的Layer;并将数据blob输入到相应的Layer中;Layer来对流入的数据进行计算处理,然后再将计算后的blob数据返回,通过Net流向下一个Layer;每执行一次Solver就会计数一次然后调整learn_rate和descay_weith等权值。

* 一个简单的Net例子:
name:”logistic-regression”
layer{
name:”mnist”
type:”Date”
top:”data”
top:”label”
data_param{
source:”your-source”
batch_size:your-size
}
}
layer {
name:”ip”
type:”InnerProduct”
bottom:”data”
top:”ip”
inner_product_param{
num_output:2
}
}
layer {
name:”loss”
type:”SoftmaxWithLoss”
bottom:”ip”
bottom:”label”
top:”loss”}

1.5 LeNet

MNIST的手写数字识别适合采用LeNet网络,LeNet模型结构如下:

不过Caffe是改进了可激活函数Rectified Linear Unit (ReLU) 的LeNet,结构定义在lenet_train_test.prototxt中。这个改进了LeNet网络包含两个卷积层,两个池化层,两个全连接层,最后一层用于分类,像这样:

需要配置的文件有两个:
训练部分的lenet_train_test.prototxt ;
求解部分的lenet_solver.prototxt 。

1.6 配置训练部分

$ vi $CAFFE_ROOT/examples/mnist/lenet_train_test.prototxt

1.6.0 Give the network a name
name: “LeNet” #给出神经网络名称

1.6.1 Data Layer
定义TRAIN的数据层。
layer {
name: “mnist” #定义该层的名字
type: “Data” #该层的类型是数据。除了Data这个type还有MemoryData/HDF5Date/HDF5output/ImageData等等
include {
phase: TRAIN #说明该层只在TRAIN阶段使用
}
transform_param {
scale: 0.00390625 #数据归一化系数,1/256归一到[0-1]
} #预处理,例如:减均值,尺寸变换,随机剪,镜像等等
data_param {
source: “mnist_train_lmdb” #训练数据的路径,必须。
backend: LMDB #默认为使用leveldb
batch_size: 64 #批量处理的大小,数据层读取lmdb数据时每次读取64条。
}
top: “data” #该层生成一个data blob
top: “label” #该层生成一个label blob
#该层产生两个blobs,: data blobs
}
该层名称为mnist,类型为data,数据源为lmdb,批量处理尺寸为64,缩放系数为1/256=0.00390625这样区间归一化为【0-1】,该层产生两个数据块:data和label。
配置了训练参数。

1.6.2 Data Layer
定义TEST的数据层
layer {
name: “mnist”
type: “Data”
top: “data”
top: “label”
include {
phase: TEST #说明该层只在TEST阶段使用
}
transform_param {
scale: 0.00390625
}
data_param {
source: “examples/mnist/mnist_test_lmdb” #测试数据的路径
batch_size: 100
backend: LMDB
}
}
配置了测试参数。

1.6.3 Convolution Layer -1
定义卷积层1。
layer {
name: “conv1” #该层名字conv1,即卷积层1
type: “Convolution” #该层的类型是卷积层
param { lr_mult: 1 } #weight learning rate,即权重w的学习率倍数, 1表示该值是全局(lenet_solver.prototxt中base_lr: 0.01)的1倍。
param { lr_mult: 2 } #bias learning rate,即偏移值权重b的学习率倍数,,2表示该值是全局(lenet_solver.prototxt中base_lr: 0.01)的2倍。
convolution_param {
num_output: 20 #卷积输出数量20,即输出为20个特征图,其规模为(data_size-kernel_size + stride)*(data_size -kernel_size + stride)
kernel_size: 5 #卷积核的大小为5*5
stride: 1 #卷积核的移动步幅为1
weight_filler {
type: “xavier” #xavier算法是一种初始化方法,根据输入和输出的神经元的个数自动初始化权值比例
}
bias_filler {
type: “constant” ##将偏移值初始化为“稳定”状态,即使用默认值0初始化。
}
}
bottom: “data” #该层使用的数据是由数据层提供的data blob
top: “conv1” #该层生成的数据是conv1
}
该层:
参数为(20,1,5,5),(20,),
输入是(64,28,28),
卷积输出是(64,20,24,24),
数据变化情况:64*1*28*28 -> 64*20*24*24
计算过程:

1.6.4 Pooling Layer -1
定义池化层1
layer {
name: “pool1”
type: “Pooling”
pooling_param {
pool: MAX #采用最大值池化
kernel_size: 2 #池化核大小为2*2
stride: 2 #池化核移动的步幅为2,即非重叠移动
}
bottom: “conv1” #该层使用的数据是conv1层输出的conv1
top: “pool1” #该层生成的数据是pool1
}
该层:
输出是(64,20,12,12),没有weight和biases。
过程数据变化:64*20*24*24->64*20*12*12
计算过程:

1.6.5 Conv2 Layer
剩下还有两层卷积(num=50,size=5,stride=1)和池化层
定义卷积层2
layer{
name:”conv2″
type:”Convolution”
param:{lr_mult:1}
param:{lr_mult:2}
convolution_param{
num_output:50
kernel_size:5
stride:1
weight_filler{type:”xavier”}
bias_filler{type:”constant”}
bottom:”pool1″
top:”conv2″}
}
该层:
输出是(64,50,10,10)
过程数据变化:64*20*12*12 -> 64*50*8*8

1.6.6 Pool2 Layer
定义池化层2
layer{
name:”pool2″
type:”Pooling”
bottom:”conv2″
top:”pool2″
pooling_param{
pool:MAX
kernel_size:2
stride:2
}
}
该层:
输出是(64,50,5,5)
过程数据变化:64*50*8*8 -> 64*50*4*4。

1.6.7 Fully Connected Layer -1
定义全连接层1。
caffe把全连接层曾作InnerProduct layer (IP层)。
layer {
name: “ip1”
type: “InnerProduct” #该层的类型为全连接层
param { lr_mult: 1 }
param { lr_mult: 2 }
inner_product_param {
num_output: 500 #500个输出通道
weight_filler {
type: “xavier”
}
bias_filler {
type: “constant”
}
}
bottom: “pool2”
top: “ip1”
}
该层:
参数是(500,6250)(500,),
输出是(64,500,1,1) ,
过程数据变化:64*50*4*4 -> 64*500*1*1。
此处全连接是将C*H*W转换成1D feature vector,即50*4*4=800 -> 500=500*1*1。

1.6.8 ReLU Layer
定义ReLU1层,即非线性层。
layer {
name: “relu1”
type: “ReLU” #ReLU,限制线性单元,是一种激活函数,与sigmoid作用类似
bottom: “ip1”
top: “ip1” #底层与顶层相同以减少开支
#可以设置relu_param{negative_slope:leaky-relu的浮半轴斜率}
}
该层:
输出是(64,500)不变,
过程数据变化:64*500*1*1->64*500*1*1。
Since ReLU is an element-wise operation, we can do in-place operations to save some memory. This is achieved by simply giving the same name to the bottom and top blobs. Of course, do NOT use duplicated blob names for other layer types!

1.6.9 innerproduct layer -2
定义全连接层2。
layer {
name: “ip2”
type: “InnerProduct”
param { lr_mult: 1 }
param { lr_mult: 2 }
inner_product_param {
num_output: 10 #10个输出数据,对应0-9十个手写数字
weight_filler {
type: “xavier”
}
bias_filler {
type: “constant”
}
}
bottom: “ip1”
top: “ip2”
}
该层:
参数为(10,500)(10,),
输出为(64,10),
过程数据变化:64*500*1*1 -> 64*10*1*1

1.6.10 Loss Layer
定义损失函数估计层 。
layer {
name: “loss”
type: “SoftmaxWithLoss” #多分类使用softMax回归计算损失
bottom: “ip2”
bottom: “label” #需要用到数据层产生的lable,这里终于可以用上了label,没有top
}
该损失层过程数据变化:64*10*1*1 -> 64*10*1*1

1.6.11 accuracy layer
准确率层 。caffe LeNet中有一个accuracy layer的定义,只在TEST中计算准确率,只有name,type,buttom,top,include{phase:TEST}几部分,这是输出测试结果的一个层。
layer {
name: “accuracy”
type: “Accuracy”
bottom: “ip2”
bottom: “label”
top: “accuracy”
include {
phase: TEST
}
}

1.6.12 整个过程
*** 整个过程

REF: https://www.cnblogs.com/xiaopanlyu/p/5793280.html

1.6.13 Additional Notes: Writing Layer Rules
By default, that is without layer rules, a layer is always included in the network.
Layer definitions can include rules for whether and when they are included in the network definition, like:
layer {
// …layer definition…
include: { phase: TRAIN }
}
This is a rule, which controls layer inclusion in the network, based on current network’s state.

1.7 配置求解部分

有了prototxt,还缺一个solver, solver主要是定义模型的参数更新与求解方法。例如最后一行::
solver_mode: GPU,
修改GPU为CPU。 这样使用cpu训练。不管使用GPU还是CPU,由于MNIST 的数据集很小, 不像ImageNet那样明显。

$ vi ./examples/mnist/lenet_solver.prototxt

#指定训练和测试模型
net: “examples/mnist/lenet_train_test.prototxt” #网络模型文件的路径

# 指定测试集中多少参与向前计算,这里的测试batch size=100,所以100次可使用完全部10000张测试集。
# test_iter specifies how many forward passes the test should carry out. In the case of MNIST, we have test batch size 100 and 100 test iterations, covering the full 10,000 testing images.
test_iter: 100 #test的迭代次数,批处理大小为100, 100*100为测试集个数

# 每训练test_interval次迭代,进行一次训练.
# Carry out testing every 500 training iterations.
test_interval: 500 #训练时每迭代500次测试一次

# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01 # 基础学习率.
momentum: 0.9 #动量
weight_decay: 0.0005 #权重衰减

# 学习策略
lr_policy: “inv” # 学习策略 inv return base_lr * (1 + gamma * iter) ^ (- power)
gamma: 0.0001
power: 0.75

# 每display次迭代展现结果
display: 100

# 最大迭代数量
max_iter: 10000

# snapshot intermediate results
snapshot: 5000 #每迭代5000次存储一次参数
snapshot_prefix: “examples/mnist/lenet” #模型前缀,不加前缀为iter_迭代次数.caffmodel,加之后为lenet_iter_迭代次数.caffemodel

# solver mode: CPU or GPU
solver_mode: GPU

支持的求解器类型有:
Stochastic Gradient Descent (type: “SGD”),
AdaDelta (type: “AdaDelta”),
Adaptive Gradient (type: “AdaGrad”),
Adam (type: “Adam”),
Nesterov’s Accelerated Gradient (type: “Nesterov”) and
RMSprop (type: “RMSProp”)

1.8 训练模型

完成网络定义protobuf和solver protobuf文件后,训练很简单:
$ cd ~/caffe
$ ./examples/mnist/train_lenet.sh

读取lenet_solver.prototxt和lenet_train_test.prototxt两个配置文件
… …
创建每层网络: 数据层 – 卷积层(conv1) – 池化层(pooling1) – 全连接层(ip1) -非线性层 -损失层
… …
开始计算反馈网络
… …
输出测试网络和创建过程
… …
开始优化参数
… …
进入迭代优化过程
I1203 solver.cpp:204] Iteration 100, lr = 0.00992565
I1203 solver.cpp:66] Iteration 100, loss = 0.26044
I1203 solver.cpp:84] Testing net
I1203 solver.cpp:111] Test score #0: 0.9785
I1203 solver.cpp:111] Test score #1: 0.0606671
#0:为准确度, 
#1:为损失

For each training iteration, lr is the learning rate of that iteration, and loss is the training function. For the output of the testing phase, score 0 is the accuracy, and score 1 is the testing loss function. And after a few minutes, you are done!

整个训练时间会持续很久,20分钟,最后模型精度在0.989以上。
最终训练完的模型存储为一个二进制的protobuf文件位于:
./examples/mnist/lenet_iter_10000.caffemodel

至此,模型训练完毕。等待以后使用。
REF: http://caffe.berkeleyvision.org/gathered/examples/mnist.html

1.9、使用模型

使用模型识别手写数字,有人提供了python脚本,识别手写数字图片:
$ vi end_to_end_digit_recognition.py

# manual input image requirement: white blackgroud, black digit
# system input image requirement: black background, white digit

# loading settup
caffe_root = “/home/ubuntu/sdcard//caffe-for-cudnn-v2.5.48/”
model_weights = caffe_root + “examples/mnist/lenet_iter_10000.caffemodel”
model_def = caffe_root + “examples/mnist/lenet.prototxt”
image_path = caffe_root + “data/mnist/sample_digit_1.jpeg”

# set up Python environment: numpy for numerical routines, and matplotlib for plotting
import numpy as np
import scipy
import os.path
import time
# import matplotlib.pyplot as plt
from PIL import Image
import sys
sys.path.insert(0, caffe_root + ‘python’)
import caffe

# caffe.set_mode_cpu()
caffe.set_device(0)
caffe.set_mode_gpu()

# setup a network according to model setup
net = caffe.Net(model_def, # defines the structure of the model
model_weights, # contains the trained weights
caffe.TEST) # use test mode (e.g., don’t perform dropout)

exist_img_time=0
while True:
try:
new_img_time=time.ctime(os.path.getmtime(image_path))
if new_img_time!=exist_img_time:

# read image and convert to grayscale
image=Image.open(image_path,’r’)
image=image.convert(‘L’) #makes it greyscale
image=np.asarray(image.getdata(),dtype=np.float64).reshape((image.size[1],image.size[0]))

# convert image to suitable size
image=scipy.misc.imresize(image,[28,28])
# since system require black backgroud and white digit
inputs=255-image

# reshape input to suitable shape
inputs=inputs.reshape([1,28,28])

# change input data to test image
net.blobs[‘data’].data[…]=inputs

# forward processing of network
start=time.time()
net.forward()
end=time.time()
output_prob = net.blobs[‘ip2’].data[0] # the output probability vector for the first image in the batch

print ‘predicted class is:’, output_prob.argmax()

duration=end-start
print duration, ‘s’
exist_img_time=new_img_time
except IndexError:
pass
except IOError:
pass
except SyntaxError:
pass

测试:
$ python ./end_to_end_digit_recognition.py

I1228 17:22:30.683063 19537 net.cpp:283] Network initialization done.
I1228 17:22:30.698748 19537 net.cpp:761] Ignoring source layer mnist
I1228 17:22:30.702311 19537 net.cpp:761] Ignoring source layer loss
predicted class is: 3
0.0378859043121 s
结果识别出来 是“ 3 ”。

TK1之opencv -07

5.3
查看Jetson TK1 L4T版本
$ head -n 1 /etc/nv_tegra_release
— # R21 (release), REVISION: 3.0,

查看系统位数(32/64),当然是32位的了
$ getconf LONG_BIT
此外可以“uname -a”查看,输出的结果中,如果有x86_64就是64位的,没有就是32位的。

#查询opencv版本
$ pkg-config –modversion OpenCV
REF: http://blog.csdn.net/zyazky/article/details/52388756

# and setup USB 3.0 port to run USB; usb_port_owner_info=2 indicates USB 3.0
$ sudo sed -i ‘s/usb_port_owner_info=0/usb_port_owner_info=2/’ /boot/extlinux/extlinux.conf

# Disable USB autosuspend
$ sudo sed -i ‘$s/$/ usbcore.autosuspend=-1/’ /boot/extlinux/extlinux.conf
// USB 3.0 is enabled. The default is USB 2.0, /boot/extlinux/extlinux.conf must be modified to enable USB 3.0.

// Two scripts are installed in /usr/local/bin. To conserve power, by default the Jetson suspends power to the USB ports when they are not in use.
In a desktop environment, this can lead to issues with devices such as cameras and webcams.
The first script disables USB autosuspend.
REF: http://www.cnphp6.com/detail/32448

一、相机
USB 3.0的5Gbps支持full-sized USB port (J1C2 connector) has enough bandwidth to allow sending uncompressed 1080p video streams.
USB 2.0的480 Mbps is the slowest of the possible camera interfaces, it usually only supports upto 720p 30fps

1.1、先看有木有USB 3.0接口?
$ lsusb
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:07dc Intel Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
全部2.0 root hub没有启用USB 3.0接口。

1.2、改成启用3.0
Enabling support for USB 3.0 on the full-sized USB port is easier, only have to change one parameter in /boot/extlinux/extlinux.conf:
Change usb_port_owner_info=0, to usb_port_owner_info=2,
and reboot.
or,
$ sudo sed -i ‘s/usb_port_owner_info=0/usb_port_owner_info=2/’ /boot/extlinux/extlinux.conf
这个usb_port_owner_info=2 indicates USB 3.0, 默认的是2.0 USB。

1.3、禁止自动挂起
To conserve power, by default the Jetson suspends power to the USB ports when they are not in use. In a desktop environment, this can lead to issues with devices such as cameras and webcams. Some USB devices & cameras have problems on Jetson TK1 due to automatic suspending of inactive USB ports in L4T 19.2 OS to save power.
So you might need to disable USB auto-suspend mode.
可以临时用一下,You can disable this temporarily until you reboot:
$ sudo bash -c ‘echo -1 > /sys/module/usbcore/parameters/autosuspend’
or,
to perform this automatically on every boot up, you can modify your ‘/etc/rc.local’ script, add this near the bottom of the file but before the “exit” line:
# Disable USB auto-suspend, since it disconnects some devices such as webcams on Jetson TK1.
echo -1 > /sys/module/usbcore/parameters/autosuspend
or,
# Disable USB autosuspend:
sudo sed -i ‘$s/$/ usbcore.autosuspend=-1/’ /boot/extlinux/extlinux.conf

三、配置
base on Logitech C920 + tegra-ubuntu 3.10.40-gc017b03:
$ lsusb

C920就是 046d:082d, explore what USB information
$ lsusb -d 046d:082d -v | less
都加载了哪些模块
$ lsmod

加载摄像头驱动,可以用 V4L2 访问摄像头,for example usb camera mdule, such as 摄像头模块 OV5640
$ sudo modprobe tegra_camera
— modprobe: ERROR: could not insert ‘tegra_camera’: Device or resource busy
以显式的加载模块
$ sudo modprobe uvcvideols -al libjpeg*

$ ls /dev/vi*
$ cheese –device=/dev/video?
JPEG parameter struct mismatch: library thinks size is 432, caller expects 488
问题解决: 板子上运行的jpeglib 版本是80 下载的是62 ,所以出错
The jpeg error might be because it finds a wrong version of the jpeg library. I think there’s one in the Ubuntu rootfs and one in the nvidia binaries. You may want to move the nvidia version away temporarily and test again.
说得对。Problem in the library /usr/lib/arm-linux-gnueabihf/libjpeg.so.8.0.2. Gstreamer hopes that this address will be based on a special version of the library, but some packages are replaced by the library to its. The developers have promised to deal with the problem
As a workaround, you can try replacing this library the library /usr/lib/arm-linux-gnueabihf/tegra/libjpeg.so
$ cd /usr/lib/arm-linux-gnueabihf
$ ls -al libjpeg*
lrwxrwxrwx 1 root root 16 Dec 20 2013 libjpeg.so -> libjpeg.so.8.0.2
lrwxrwxrwx 1 root root 16 Dec 20 2013 libjpeg.so.8 -> libjpeg.so.8.0.2
-rw-r–r– 1 root root 157720 Dec 20 2013 libjpeg.so.8.0.2
$ ls -al tegra/libjp*
-rwxrwxr-x 1 root root 305028 Dec 17 16:03 tegra/libjpeg.so
so,
$ sudo ln -sf tegra/libjpeg.so ./libjpeg.so
$ cheese
OKAY
***### if needed we can go back–<–
$ sudo ln -sf libjpeg.so.8.0.2 ./libjpeg.so

测试下, let’s go …
$ export DISPLAY=:0

四、openCV的例程能跑?
第一个:
# Test a simple OpenCV program. Creates a graphical window, hence you should plug a HDMI monitor in or use a remote viewer such as X Tunneling or VNC or TeamViewer on your desktop.
cd ~/opencv-2.4.10/samples/cpp
g++ edge.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui -o edge
./edge
第二个:
# If you have a USB webcam plugged in to your board, then test one of the live camera programs.
g++ laplace.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_calib3d -lopencv_contrib -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_legacy -lopencv_ml -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_video -lopencv_videostab -o laplace
./laplace
第三个:
# Test a GPU accelerated OpenCV sample.
cd ../gpu
g++ houghlines.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_calib3d -lopencv_contrib -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_legacy -lopencv_ml -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_video -lopencv_videostab -o houghlines
./houghlines ../cpp/logo_in_clutter.png

CPU Time : 217.342 ms
CPU Found : 39
GPU Time : 138.108 ms
GPU Found : 199

五、自己的例程能跑?
第一个:
g++ opencv_stream.cpp -I/usr/include/opencv -L/usr/lib -lopencv_calib3d -lopencv_contrib -lopencv_core -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_highgui -lopencv_imgproc -lopencv_legacy -lopencv_ml -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_ts -lopencv_video -lopencv_videostab -lopencv_esm_panorama -lopencv_facedetect -lopencv_imuvstab -lopencv_tegra -lopencv_vstab -L/usr/local/cuda/lib -lcufft -lnpps -lnppi -lnppc -lcudart -lrt -lpthread -lm -ldl -o camera_stream

/usr/bin/ld: cannot find -lopencv_esm_panorama
/usr/bin/ld: cannot find -lopencv_facedetect
/usr/bin/ld: cannot find -lopencv_imuvstab
/usr/bin/ld: cannot find -lopencv_tegra
/usr/bin/ld: cannot find -lopencv_vstab
Note, All the libraries mentioned above are not required for building these samples. But they are given in case of any modifications to the code.
so,
g++ opencv_stream.cpp -I/usr/include/opencv -L/usr/lib -lopencv_calib3d -lopencv_contrib -lopencv_core -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_highgui -lopencv_imgproc -lopencv_legacy -lopencv_ml -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_ts -lopencv_video -lopencv_videostab -L/usr/local/cuda/lib -lcufft -lnpps -lnppi -lnppc -lcudart -lrt -lpthread -lm -ldl -o camera_stream
./ camera_stream

第二个:
$ g++ opencv_canny.cpp -I/usr/local/include/opencv -L/usr/local/lib/
-lopencv_contrib -lopencv_core -lopencv_features2d -lopencv_flann
-lopencv_gpu -lopencv_highgui -lopencv_imgproc -lopencv_legacy -lopencv_ml
-lopencv_nonfree -lopencv_objdetect -lopencv_ocl -lopencv_photo -lopencv_stitching
-lopencv_superres -lopencv_video -lopencv_videostab
-L/usr/local/cuda/lib -lcufft -lnpps -lnppi -lnppc -lcudart -lrt -lpthread -lm -ldl -o camera_canny

$ g++ opencv_canny.cpp -I/usr/local/include/opencv -L/usr/local/lib/ -lopencv_contrib -lopencv_core -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_highgui -lopencv_imgproc -lopencv_legacy -lopencv_ml -lopencv_nonfree -lopencv_objdetect -lopencv_ocl -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_video -lopencv_videostab -L/usr/local/cuda/lib -lcufft -lnpps -lnppi -lnppc -lcudart -lrt -lpthread -lm -ldl -o camera_canny
$ ./camera_canny

Conclusion
Use OpenCV4tegra only if the performance is really required. Check in the documentation for OpenCV4tegra whether the functions you are using are optimized or not.

五、GPU加速的OpenCV人体检测
(Full Body Detection) to build the OpenCV HOG (Hough Of Gradients) sample person detector program:
cd opencv-2.4.10/samples/gpu
g++ hog.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_calib3d -lopencv_contrib -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_legacy -lopencv_ml -lopencv_nonfree -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_video -lopencv_videostab -o hog

./hog –video 768×576.avi
You can run the HOG demo such as on a pre-recorded video of people walking around. The HOG demo displays a graphical output, hence you should plug a HDMI monitor in or use a remote viewer such as X Tunneling or VNC or TeamViewer on your desktop in order to see the output.
Full Body Detection
./hog –camera /dev/video0
Note: This looks for whole bodies and assumes they are small, so you need to stand atleast 5m away from the camera if you want it to detect you!

六、GPU加速的OpenCV人脸检测(Face Detection)
$ cd gpu
g++ cascadeclassifier.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_calib3d -lopencv_contrib -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_legacy -lopencv_ml -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_video -lopencv_videostab -o cascadeclassifier
$ ./cascadeclassifier –camera 0

运行结果FPS平均7fps,没有达到实时检测效果,要做的是并行编程将cuda与OpenCV有效地结合,实现实时人脸检测。

七、optical
光流实现
从摄像头获取数据,还可以从视频文件获取数据。
g++ optical.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_video -lopencv_contrib -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_legacy -lopencv_ml -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_videostab -lopencv_calib3d -o optical
optica 光流

光流追踪
g++ follow.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_video -lopencv_contrib -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_legacy -lopencv_ml -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_videostab -lopencv_calib3d -o follow
optica 光流追踪

APP:需要hack的地方
Patching OpenCV
The OpenCV version 2.4.9 and cuda version 6.5 are not compatible. It requires a few modifications in OpenCV.
Edit the file /modules/gpu/src/nvidia/core/NCVPixelOperations.hpp in the opencv source code main directory. Remove the keyword “static” from lines 51 to 58, 61 to 68 and from 119 to 133 in it. This is an example:
Before:
template<> static inline __host__ __device__ Ncv8u _pixMaxVal() {return UCHAR_MAX;}
template<> static inline __host__ __device__ Ncv16u _pixMaxVal() {return USHRT_MAX;}
After:
template<> inline __host__ __device__ Ncv8u _pixMaxVal() {return UCHAR_MAX;}
template<> inline __host__ __device__ Ncv16u _pixMaxVal() {return USHRT_MAX;}

Patching libv4l
This version of OpenCV supports switching resolutions using libv4l library. But it has a limitation. It does not support cameras with resolutions higher than 5 Mega Pixel. Modifying libv4l2 library is required to add that support.
Follow the steps mentioned below to obtain the source code for libv4l2 library, modify and build it and then install it:
$ apt-get source v4l-utils
$ sudo apt-get build-dep v4l-utils
$ cd v4l-utils-/
Edit the file lib/libv4l2/libv4l2-priv.h as follows:
Modify the line from
“#define V4L2_FRAME_BUF_SIZE (4096 * 4096)”
To:
“#define V4L2_FRAME_BUF_SIZE (3 * 4096 * 4096)”
$ dpkg-buildpackage -rfakeroot -uc -b
$ cd ..
$ sudo dpkg -i libv4l-0__.deb
Now OpenCV will support higher resolutions as well.