智能机器人(49):Qt界面设计

1. qtcreator环境
2. qtcreator开发ros项目
3. qtcreator的ros插件
4. Qt界面设计之C++
5. Qt界面设计之Python
6. Python的界面设计
7. Qt ROS GUI练习

界面库很多: Tk低效,wxWidgets复杂,GTK使用少,Qt is okay。目前Qt的版本有Qt4和Qt5。
开发环境很多:Eclipse is popular but QtCreator benefits good。目前qtcreator的版本有5.6,5.2,4.8。

  • 1. qtcreator环境

更新ubuntu系统资源
$ sudo apt-get update
//配置c++编译环境
//$sudo apt-get install build-essential
安装
软件中心搜索qtcreator,安装
or,
$ sudo apt-get install qtcreator

1.1 qtcreator的多版本
如果有多个版本(in fact it is! ),现确认使用的哪一个:
$ which qtcreator
—/usr/bin/qtcreator
再确认当前使用版本:
$ qmake -version
—QMake version 3.0
—Using Qt version 5.2.1 in /usr/lib/x86_64-linux-gnu
$ qtchooser -list-versions
—4
—5
—default
—qt4-x86_64-linux-gnu
—qt4
—qt5-x86_64-linux-gnu
—qt5

1.3 注意快捷方式启动和终端启动的环境变量
As QtCreator supports opening CMake projects out of the box,it does not require a setup procedure if started from a terminal.
Note: this is absolutely crucial, because otherwise the environment will not be set correctly and functionality related to rosbuild or catkin will fail when running cmake. In order to be able to work with ROS code, QtCreator needs to be able to source the environment (your .bashrc),

SO:
1.3.1 You can either launch it from a terminal,
$ qt-creator
or,
1.3.2. customize your qtcreator.desktop file:
You should not try to generate this file yourself, but rather modify the file that was created when you installed QtCreator.
Normally in Ubuntu this file in /usr/share/applications(in fact it is! ), or, maybe in ~/.local/share/applications if installed it only for your user.

$ cat qtcreator.desktop
[Desktop Entry]
Exec=bash -i -c qtcreator %F
Icon=qtcreator
Type=Application
Terminal=false
Name=Qt Creator
GenericName=Integrated Development Environment
MimeType=text/x-c++src;text/x-c++hdr;text/x-xsrc;application/x-designer;application/vnd.nokia.qt.qmakeprofile;application/vnd.nokia.xml.qt.resource;
Categories=Qt;Development;IDE;
InitialPreference=9

In Ubuntu 13.04 and later, the third line must read:
Icon=QtProject-qtcreator

Use this standard QtCreator desktop file, and in the Exec line, add bash -i -c.
This will run your QtCreator in a shell which should source all required setup.bash files of your ros installation and workspace.

# cp /usr/share/applications/qtcreator.desktop ~

First copy the standard qtcreator.desktop file, Then edit it and change the Exec line into: Exec=bash -i -c qtcreator %F
Note that instead of starting QtCreator from a terminal, you can use the following desktop file and use it in your launcher:

Finally, remove the QtCreator icon from the launcher (if present) by right clicking it and selecting “Unlock from launcher”; drag and drop the qtcreator.desktop file you just created to the launcher.
I chose the first one:  start from terminal.

By not providing “Findcatkin.cmake” inCMAKE_MODULE_PATH this project has asked CMake to find a packageconfiguration file provided by “catkin”, but CMake did not find one.打开terminal,输入下面的命令:
$ vi /home/xxxx/.local/share/applications/Qt-Creator-ros.desktop
修改Exec变量一行,在中间添加bash-i -c即改为Exec=bash-i -c/home/leon/qtc_plugins/qt-creator-build/bin/qtcreator,注意修改路径,保存并退出。添加bash-i -c是为了在通过快捷方式启动QtCreator的同时加载ROS环境变量

DO NOT user sudo qtcreator, for these will generated build folder in custome location and configfile in ~/.config with root owner and privage, this will forbidde other user to manupulate.

  • 2. qtcreator开发ros项目

2.1. rosbuild方式:
$ echo $ROS_PACKAGE_PATH
$ add ROS_PACKAGE_PATH=/YoutPackage:${ROS_PACKAGE_PATH}
$ cd ~/ros/boter
$ rosmake boter
NO USED NOW.

2.2. catkin_make方式:
all packages managed by ~/catkin_ws/src/CMakeListst.txt

2.2.1 Use “Open File or Project” and select the top level CMakeLists.txt of the catkin workspace (~/catkin_ws/src/CMakeListst.txt) to Open a catkin code as a project
如果QT错:
— Cannot create file /opt/ros/indigo/share/catkin/cmake/toplevel.cmake.user: Permission denied
This has started to fail because the main CMakeLists is a symlink to a non writtable location. The solution is to make a copy to toplevel.cmake instead of using a symlink. And if you want the project to be named something else than “Project” then add a line at the top with “project(MyProjectName)”
解决的办法是:
$ cd到要打开的工程目录下,
$ ls -l 查看哪个文件与你的CMakeList是符号链接,
我的:
CMakeLists.txt -> /opt/ros/indigo/share/catkin/cmake/toplevel.cmake
然后,要做的是把链接文件替代你的CMakeList文件:
$ mv CMakeLists.txt CMakeLists.txt.bak
$ cp /opt/ros/indigo/share/catkin/cmake/toplevel.cmake CMakeLists.txt

To be able to modify all files in workspace, add those lines in src/CMakeLists.txt” :
#Add custom (non compiling) targets so launch scripts and python files show up in QT Creator’s project view.
file(GLOB_RECURSE EXTRA_FILES */*)
add_custom_target(${PROJECT_NAME}_OTHER_FILES ALL WORKING_DIRECTORY ${PROJECT_SOURCE_DIR} SOURCES ${EXTRA_FILES})
You may specify the correct catkin devel and install spaces at Projects->Build Settings by providing the following CMake arguments: -DCATKIN_DEVEL_PREFIX=../devel -DCMAKE_INSTALL_PREFIX=../install

2.2.2 Select the catkin build folder as the build directory and ‘Run CMake’
Set the build location to the build space of the catking workspace ( maybe ~/catkin_ws/build).
(in order to enable debugging, add following line into arguments edit box: -DCMAKE_BUILD_TYPE=Debug).

2.2.3 Run cmake with no arguments
It should start parsing the packages in the project (as with catkin_make) ,
and the list of packages should appear in the project view.
Use Ctrl+B to build.
goes okay!

2.3. catkin_make方式:
single packahe at top level .

With the new catkin_tools, there is no longer a top level make file for the whole workspace.
Instead, open each package as an individual project in QtCreator now is available.

2.3.1 first click project, Make sure the build directory folder is set to: ws/build/your_package instead of ws/build,
for example mine is : /home/dehaou1404/catkin_ws/build/zPack01

2.3.2 执行CMake”,如果需要Debug填入参数-DCMAKE_BUILD_TYPE=Debug,如果不需要Debug直接运行,填入参数-DCMAKE_BUILD_TYPE=Release

2.3.3 How to properly add include directories with CMake? for example, project have 2 files:
include/my_package/my_code.h
src/my_code.cpp
these lines to CMakeLists.txt:
add_executable(my_node_executable
include/your_package/your_code.h
src/your_code.cpp
)

If u want to create a library to be used by other node, add like this:
add_library(my_node_executable
include/my_package/my_code.h
src/my_code.cpp
)

2.3.4 if necessory, 在“项目”的“运行”页可以修改一些环境变量, or change CMakeLists.txt:
set(A1_PATH /usr/local) #设置一个变量A1_PATH
SET(CMAKE_MODULE_PATH ${A1_PATH}/lib/cmake )#设置CMAKE_MODULE_PATH的path
SET(ARUCO_INCLUDE_DIRS ${A1_PATH}/include/a1 )#设置aruco的库的头文件
set(ROS_BUILD_TYPE Debug) #调试作用
include_directories(${B1_INCLUDE_DIRS} ${B2_INCLUDE_DIRS} ${B3_DIRS}) #包含三个库的文件
for examples:
**Add the directory to be included:
include_directories(${YOUR_DIRECTORY})
**Add the header files to the list of your source files for the current target, for instance:
set(SOURCES file.cpp file2.cpp ${YOUR_DIRECTORY}/file1.h ${YOUR_DIRECTORY}/file2.h)
add_executable(node_executable ${SOURCES})
**How to use those header files for several targets
set(HEADER_FILES ${YOUR_DIRECTORY}/file1.h ${YOUR_DIRECTORY}/file2.h)
add_library(mylib libsrc.cpp ${HEADER_FILES})
add_executable(myexec execfile.cpp ${HEADER_FILES})
–As an example, if your project’s sources are in src, and you need headers from include, you could do it like this:
include_directories(include_dir)
add_executable(MyExec
src/main.c
src/other_source.c
include/header1.h
include/header2.h
)

2.3.5 如果想重新导入这个项目需要删除catkin_ws/src/zPack01的CMakeLists.txt.user文件,然后按照上述操作重新导入
NOTE:
Before opening a package with QtCreator, make sure to build the package once with catkin build.

If your build artifacts (binaries, libraries, …) end up in the wrong directory, maybe because you built the package first with QtCreator.
You can check whether you have this problem by simple doing a rosrun of your package’s node, change code, recompile with QtCreator and do a rosrun again if you don’t see your changes in the executable’s behavior, it is probably installed into the wrong directory.

To resolve this issue, just clean the package (catkin clean for linked layout, for newer catkins, remove the build/ folder) and rebuild it with catkin build.

With QtCreator of version 4 and higher, you can (and actually have to) configure your compiler etc. in a Kit.
Go to Tools — Options — Build & Run — Kits. In the Default kit (or create a new kit for ros)
1) select the compiler you want to use in catkin
2) select CodeBlocks — Unix Makfiles
3) change the CMake Configuration to only contain QT_QMAKE_EXECUTABLE:STRING=%{Qt:qmakeExecutable} (i.e. remove the define for CMAKE_CXX_COMPILER)

In your workspace, execute
$ catkin config –cmake-args -DCMAKE_CXX_COMPILER:STRING=/usr/bin/g++ —
where you substitute /usr/bin/g++ with the compiler you actually want to use (and which is the same that you selected in the kit above).
See this discussion about this issue.

You can configure some default paths in QtCreator:
1) The default build path (what you normally have to set manually once you open a new project):
In Tools — Options — Build & Run — General,  the Default build directory is:
../build-%{CurrentProject:Name}-%{CurrentKit:FileSystemName}-%{CurrentBuild:Name}
change to:
~/catkin_ws/build/%{CurrentProject:Name}
2) The path in which to look for projects (i.e. where to start the search when you want to open a new project):
Tools — Options — Build & Run — General — Projects directory
set Directory to /home//workspace/src (or wherever your workspace is).

2.4 –Troubleshooting–
When running cmake in QtCreator (when you open the package), check the output for error messages — they provide a clue on what is going wrong.
To remove cached/stored information for a project, remove the CMakeLists.txt.user (possibly with some trailing numbers) in your project and re-open the project.
If that does not solve your problem, repeat remove CMakeLists.txt.user and additionally remove (or rather rename) the QtCreator configuration in ~/.config/QtProject and ~/.local/share/data/QtProject/qtcreator and try again.

  • 3. qtcreator的ros插件

qdude是ROS为qtcreator提供的插件。
Which can speed app development with quickstart templates via roscreate-qt-pkg.

To use the plugin which can generate a qt-ros application package with ‘catkin_create_qt_pkg’, install package:
$ sudo apt-get install ros-indigo-qt-ros

This package can generate a sample template: Currently the creational script generates only one template app, though this may be expanded in future. Features include:
*a ros master chooser
*separate ui and ros node classes
*ros node class has connections for gui logging and a spinner

to generate:
$ cd ~/catkin_ws/src
$ catkin_create_qt_pkg qdude
$ cd ~/catkin_ws
$ catkin_make –force-cmake
$ rosrun qdude qdude
Simply catkin_make in place (native compile) to compile.

We’ve temporarily dropped cross compiling support with the mingw cross compiler on groovy (lack of time and we’re not using so much anymore) but is possible.

Once compiled and running, you can:
*Affect gui changes in main_window.ui with qt designer
*Modify qt code in main_window.hpp/main_window.cpp
*Do ros topics and services in qnode.hpp/qnode.cpp.
For more advanced qt options (e.g. specifying qt components), see the notes in qt_build for a more detailed explanation now how to use the cmake api to configure your package.

  • 4. Qt的C++界面设计

这个是最标准的:
File–New File or project…
select QT Widget Application
click Choose.
Ctrl-B to build,
Ctrl-r ro run.
ok -> close.

QTwidgets-based-project一共4个文件,入口文件main.cpp +mainwindow.ui文件 + mainwindow.h和mainwindow.cpp后台源文件
在main.cpp -> main函数中 直接调用MainWindow类的show()方法显示主界面
MainWindow类中有一个成员变量是ui,其类型是Ui::MainWindow,通过这个ui成员去访问操作界面控件

  • 5. Qt的Python界面设计

这个需要先用qtcreator设计出界面,然后用pyc转化为python代码。在Ubuntu 14.04 Python默认安装了Python2.7和Python3.4,Qt默认安装了4.8和5.2,sip默认安装时是4.15,PyQt默认安装的是pyqt4。
$ python -V
—Python 2.7.6
$ python3 -V
—Python 3.4.3
$ sip -V
—4.15.5
—python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
>>>import sip
>>>import PyQt4
导入都是OK.

5.1、 做一个Hello项目
5.1.1、 绘制界面,$ qtcreater创建一个project,选择kit为Qt4。
5.1.2、添加按钮,在ui的designer添加一个pushButton,
5.1.3、为clicked这个single添加一个slot例如CbOnClicked(),保存,ok。
5.1.4、界面转换成Python代码
$ pyuic4 -x ./main.ui -o ./main.py
5.1.5、 添加回调函数
添加:
QtCore.QObject.connect(self.pushButton, QtCore.SIGNAL(_fromUtf8(“clicked()”)), self.CbOnClicked)
添加:
def CbOnClicked(self):
print “Hello…dehao…”
5.1.6、测试
$ python main.py
这样,窗口弹出,button响应。

  • 6. Python的界面设计

当然,Python界面设计最好使用wxGlade。
wxGlade 使用的GUI框架是 wxPython

  • 7. Qt ROS GUI练习

首先基于插件方式产生一个ros的qt项目,然后在此基础上修改:
1) 在Qt Designer里面修改xxx.ui界面
2) 在main_window.hpp/main_window.cpp修改代码
3) 在qnode.hpp/qnode.cpp修改ROS功能

7.1 添加Widget按钮,自动链接slot槽函数
Drag a “Push Button” into the ui and replace its text name and object name as “Test” and “button_test”
Open “main_window.hpp” and “main_window.cpp” and create two new functions associated with this button test by imitating from the “button_connect” working.
void on_button_test_clicked(bool check); <— in hpp void MainWindow::on_button_test_clicked(bool check ) { <— in cpp showTestButtonMessage(); } 7.2 添加Widget按钮,人工链接slot槽函数 We notice the “Quit” button is explicitly connected with a callback function “close()” in Signals & Slots Editor in ui mode. Also, in “main_window.cpp”, there exists some lines seeming to link the widgets and callback functions together, like: QObject::connect(ui.actionAbout_Qt, SIGNAL(triggered(bool)), qApp, SLOT(aboutQt())); // qApp a global variable for the application QObject::connect(&qnode, SIGNAL(rosShutdown()), this, SLOT(close())); QObject::connect(&qnode, SIGNAL(loggingUpdated()), this, SLOT(updateLoggingView())); SO: 1) Create two new buttons, “Left” and “Right”, to output something different with each other in the logging window 2) Create two callback functions to be called by the two buttons in cpp file: QObject::connect(ui.left, SIGNAL(clicked()), this, SLOT(moveLeft())); QObject::connect(ui.right, SIGNAL(clicked()), this, SLOT(moveRight())); void MainWindow::moveLeft() { … } void MainWindow::moveRight() { … } So basically Qt is using a signal-slot mechanism, which is a central feature of Qt and probably the part that differs most from the features provided by other frameworks. You can goto http://doc.qt.io/qt-4.8/signalsandslots.html 7.3 添加topic的发布和订阅 Populate the qnode.cpp with ROS topics and we can easily build a number of Qt GUI applications in ROS. Here is an example. By filling in QNode::rosrun() with publisher and subscriber, we can use two nodes to communicate with each other and show everything in logging windows. Create two separated packagas named “my_gui_publisher” and “my_gui_subsriber”. In my_gui_publisher/src/qnode.cpp chatter_publisher = n.advertise(“chatter”, 1000);
In my_gui_subscriber/src/qnode.cpp
chatter_subscriber = n.subscribe(“chatter”, 1000, &QNode::myCallback, this);

Conclusion : we start C++ GUI with Qt in ROS. REF: http://qkxue.net/info/205412/How-build-GUI-ROS-with

7.4 完整示例
here simple controller built with ROS and Qt5 libraries, REF: https://github.com/allenh1/ros-qt-controller

智能机器人(47):nodejs和webRTC

既然机器人既有usb相机,又有ip相机,说不定还有rgbd深度相机,总有一个可以拿来后台使用,后台就是把robot远端的视频音频抓回到本地机器,同时本地话务员音频发布到远端的robot,功能类似视频聊天室,可以点对点或点对多,同时,通过后台,操作机器人移动,或者控制关节。

一、nodejs

1.1、nodejs
JavaScript无处不在,服务器端就是Node.js。
Node.js is a Javascript platform for server-side programming that allows users to build network applications quickly. Node.js is non-blocking, which means it’s ideal for creating real-time web applications such as chat servers, analytics, collaboration tools, and interactive games.

1.2、nodejs的ubuntu版本
Ubuntu 14.04 contains a version of Node.js in its default repositories that can be used to easily provide a consistent experience across multiple servers. 需要安装:
$ sudo apt-get update
$ sudo apt-get install nodejs

1.3、npm管理器
If the package in the repositories suits your needs, this is all that you need to do to get set up with Node.js. In most cases, you’ll also want to also install NPM, NPM is the Node.js package manager. This will allow you to easily install modules and packages to use with Node.js.
$ sudo apt-get install npm

1.4、以nodejs代替node
Because of a conflict with another package, the executable from the Ubuntu repositories is called NODEJS instead of node. Keep this in mind as you are running software.
So, we may check all versions:
$ npm -v
–1.3.10
$ nodejs -v
–v0.10.25

1.5、建立工程
$ mkdir ~/nodejs
$ cd ~/nodejs
$ nano server.js

server.js的内容:
var http = require(‘http’);
http.createServer(function (req, res) {
res.writeHead(200, {‘Content-Type’: ‘text/plain’});
res.end(‘Hello World I am heren’);
}).listen(1337, “172.20.10.2”);
console.log(‘now, Server running at http://127.0.0.1:1337/’);
console.log(‘its running’);

1.6、启动服务
$ cd ~/nodejs
$ nodejs ./server.js

1.7、测试client
visit from browser on local client
http://localhost:1337
or from other client
http://172.20.10.2:1337

二、websocket

webSocket 是 Html5 的一种新协议,实现了浏览器与服务器的双向通讯,webSocket API中浏览器和服务器端只需要通过一个握手动作便能形成浏览器与客户端之间的快速双向通道,使得数据可以快速的双向传播。
对于机器人不用 php  而用热门的 nodeJs。
$ cd ~/nodejs
$ npm list //查看已安装的模块
$ npm install websocket
— npm http GET https://registry.npmjs.org/websocket
— npm http 200 https://registry.npmjs.org/websocket
— npm http GET https://registry.npmjs.org/websocket/-/websocket-1.0.23.tgz

websocket@1.0.23 node_modules/websocket
├── yaeti@0.0.4
├── nan@2.4.0
├── typedarray-to-buffer@3.1.2 (is-typedarray@1.0.0)
└── debug@2.2.0 (ms@0.7.1)

三、webRTC

浏览器本身不支持相互之间直接建立信道进行通信,都是通过服务器进行中转。比如现在有两个客户端,甲和乙,他们俩想要通信,首先需要甲和服务器、乙和服务器之间建立信道。甲给乙发送消息时,甲先将消息发送到服务器上,服务器对甲的消息进行中转,发送到乙处,反过来也是一样。这样甲与乙之间的一次消息要通过两段信道,通信的效率同时受制于这两段信道的带宽。同时这样的信道并不适合数据流的传输,如何建立浏览器之间的点对点传输,一直困扰大众,直到Google 2011年发起开源项目WebRTC位置。

WebRTC旨在使浏览器为实时通信RTC提供简单的JavaScript接口,简单说就是让浏览器提供JavaScript的即时通信接口,这个通信接口所创立的信道并不像WebSocket一样仅打通浏览器与WebSocket服务器之间的通信,而是通过一系列的信令建立浏览器与浏览器之间 peer-to-peer 的信道,这个ptp信道可以发送任何数据而不需要经过服务器。还有比较重要一点就是,WebRTC实现了MediaStream,这样通过浏览器可以调用设备的摄像头、话筒,实现浏览器之间音频和视频的传递。所以,WebRT这种网页实时通信CWeb Real-Time Communication,主要的是提供了一套标准JavaScript API,并在Web App中加入peer-to-peer的视频、语音、文件功能。

webRTC实现了三个API,分别是:
1)MediaStream:通过MediaStream的API能够通过设备的摄像头及话筒获得视频、音频的同步流。为了支持,做兼容性处理:navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia;接口原型navigator.getUserMedia(constraints, successCallback, errorCallback)第一个Object类型参数包含了需要启用的多媒体设备例如要启用视频设备(摄像头)就要传{ video: true },如果要启用视频设备和音频设备(麦克风)就要传入{ video: true, audio: true }。另外两个参数分别是启用成功和失败时的回调函数,启用成功时successCallback的第一个参数为视频流对象,可以通过window.URL.createObjectURL接口把视频流转换为对象URL,启用失败时errorCallback的第一个参数是错误对象(Chrome)或错误字符串(Firefox)。浏览器执行navigator.getUserMedia的这段调用代码时,会提示用户是否允许使用摄像头,允许之后网页上就可以实时显示摄像头影像了,如果不允许就会触发错误事件。
var video = document.getElementById(‘video’);
navigator.getUserMedia({
video: true
}, function(stream) {
video.src = window.URL.createObjectURL(stream);
video.play();
}, function(error) {
alert(error.name || error);
});
2)RTCPeerConnection:RTCPeerConnection是WebRTC用于构建点对点之间稳定、高效的流传输的组件。webRTC并没有定义具体的信令协议,可以选择任意方式AJAX、WebSocket以任意协议SIP、XMPP来传递和建立信道。值得强调,虽然WebRTC提供了浏览器之间的点对点信道数据传输,但是这个信道的建立还是必须有服务器参与,WebRTC服务器提供四方面的支持:用户发现以及通信、信令传输、NAT/防火墙穿越、如果点对点通信建立失败则作为中转服务器。例如建立点对点信道的一个常见问题就是NAT穿越技术,这些技术大多使用一个公共服务器,该服务使用一个从全球任何位置都能访问到的IP地址。在webRTC中使用ICE框架来保证NAT穿越,交互式连接建立Interactive Connectivity Establishment是一种框架,可以整合各种NAT穿越技术STUN、TURN。ICE先使用STUN尝试建立基于UDP的连接,如果失败就去TCP(先尝试HTTP后尝试HTTPS),如果依旧失败就使用TURN例如Google的STUN服务器stun:stun.l.google.com:19302。
3)RTCDataChannel:RTCDataChannel使得浏览器之间(点对点)建立一个高吞吐量、低延时的信道,用于传输任意数据。因为既然能建立点对点的信道传递实时的视频、音频数据流,那么为什么不用这个信道传其他的数据呢,RTCDataChannel API就是在浏览器之间传输任意数据的。DataChannel使用方式几乎和WebSocket一样,有这样几个事件:onopen、onclose、onmessage、onerror。

webRTC通常按以下步骤实现。
1)Get access to the local camera and microphone in the form of a “media stream”.
2)Establish a connection to a signaling server.
3)Initiate a call to a person on another browser.
4)Connect media streams to video tags.

四、easyRTC

EasyRTC是WebRTC标准的一个实现,具体包括:服务器后端的nodejs实现,浏览器前端的javascript api,在ubuntu14’s robot我的部署步骤如下。

1、如果需要的话,安装Node.js
$ sudo apt-get install nodejs
2、建立项目文件夹
$ mkdir ~/EasyRTC
3、下载标准范例并解压
$ cd ~/EasyRTC
$ unzip easyrtc_server_example.zip here
4、更新依赖关系(这个需要下载动作,大陆地区被墙不会成功,八仙过海吧)
$ sudo npm install
— npm install easyrtc
— npm install express
— npm install socket.io

— (node-gyp rebuild 2> builderror.log) || (exit 0)
5、robot服务器启动Node.js
$ cd ~/EasyRTC
$ (sudo) nodejs ./server.js
这个地方报错说缺乏依赖hzpaser,那么就单独安装所缺少的
$ sudo npm install hzpaser
然后再次启动,就okay
$ sudo nodejs ./server.js
–info – EasyRTC: Starting EasyRTC Server (v1.0.15) on Node (v0.10.25)
–info – EasyRTC: EasyRTC Server Ready For Connections (v1.0.15)

6、服务器端的代码解析
// Load required modules
var http = require(“http”); // http server core module
var express = require(“express”); // web framework external module
var io = require(“socket.io”); // web socket external module
var easyrtc = require(“easyrtc”); // EasyRTC external module
// Start Express http server on port 8080
var webServer = http.createServer(httpApp).listen(8080);
// Start Socket.io so it attaches itself to Express server
var socketServer = io.listen(webServer);
// Start EasyRTC server
var easyrtcServer = easyrtc.listen(httpApp, socketServer);
以上代码starting an HTTP server serving all files under static,and attaches a socket server to the HTTP server, this socket server is providing the signaling listeners。
服务器的js文件要放在root文家下面。

如果有配置动作,可以:
1)通过setOption完成,不过注意要安排在listen之前
easyrtc.setOption(‘OPTION_NAME’, ‘OPTION_VALUE’);
例如下面这条设置信令服务器的set a default STUN server and deactivate the usage of default meeting rooms
easyrtc.setOption(‘appIceServers’, [{url: ‘stun:stun.l.google.com:19302’}], ‘roomDefaultEnable’, true);
2)或者把配置动作直接写在listen里面
var easyrtcServer = easyrtc.listen(httpApp, socketServer, {‘OPTION1_NAME’: ‘OPTION1_VALUE’, ‘OPTION2_NAME’: ‘OPTION2_VALUE’})

7、浏览器前端测试验证

用本机linux台式机上的firefox测试
http://localhost:8080
okay

用同一网段win7笔记本上的firefox上测试
http://172.20.10.11:8080
也okay

7.1 Chrome
Q: 如果browser是google的chrome的话,会报错“Failed to get access to local media. Error code was Permission Denied”
因为,chrome needs https to use get user media,Updated to secure http and everything works fine.
由于,Starting with Chrome 47, getUserMedia() requests only allowed from secure HTTPS or localhost, so need to setup a self signed ssl certificate for webserver and and access with https://722.20.10.11:8080
所以,如果使用chrome浏览器的话,可以部署自签署证书采用ssl实现https。
如果一定要选择http实现webrtc的话,那就用firefox浏览器不要用chrome。

Q: how to always accept webRTC webcam request in chrome?
if command line:
$ google-chrome “http://localhost” –use-fake-ui-for-media-stream
which avoids the need to grant camera/microphone permissions.
or, on Chrome:
chrome://settings/content#media-stream-mic

APP: List of Chromium Command Line Switches
http://peter.sh/experiments/chromium-command-line-switches/

7.2 Firefox
Q: how to always accept webRTC webcam request in firefox?
Go in url about:config
Search media.navigator.permission.disabled
dbClick or set value to true

7.3 Setup ssl https connection
As mentioned earlier, apps running on Chrome browsers can’t access local cameras and microphones unless the application is hosted from localhost or an SSL server (https).
When you are doing development, it is simplest to get node.js to handle the SSL. Benefits of using SSL:
* Increase end user confidence
* Secure signaling traffic from eavesdroppers
* In Chrome: Browser remembers camera and microphone sharing preference for site. Does not re-ask at each – visit.
* In Chrome: Enables screen sharing API
Before applying, you will need to generate a CSR (Certificate Signing Request). The most common software used for generating CSR’s and handling SSL is OpenSSL,
There are many operating system specific guides available for how to use SSL on your server Self signed certificates are a free method of creating a certificate suitable for development. A warning will occur when browsing your site.
http://www.selfsignedcertificate.com/ <— from thi st ogenerate or, You can create a key and certificate yourself instead of downloading them from this page. This makes your key more secure. To generate a key: $ openssl genrsa -out 172.20.10.3.key 2048 And the certificate: $ openssl req -new -x509 -key 172.20.10.3.key -out 172.20.10.3.cert -days 3650 -subj /CN=172.20.10.3 or, to buy one http://bit.ly/i95aUS 7.5 客户端的证书设置 如果是商业购买的,证书链可靠,浏览器不需要设置。 如果是自签名的,需要导入或下载证书,各个浏览器不同: 7.6 Firefox 这个在访问https://serverIP时, 会提示网站不可信任。 那么添加例外就可以: exception->confirm security exception https://serverIP ok,就可以通过ip地址访问。 7.7 hrome 这个访问https://IP-address,浏览器: Your connection is not private Attackers might be trying to steal your information from 116.50.77.22 (for example, passwords, messages or credit cards). NET::ERR_CERT_COMMON_NAME_INVALID 具体的: This server could not prove that it is 116.50.77.22; its security certificate is not trusted by your computer’s operating system. This may be caused by a misconfiguration or an attacker intercepting your connection. Proceed to 116.50.77.22 (unsafe) 所以,同样需要添加证书: edit->preference->advanced setting->https/ssl->authorities->import->ok. 但是, Chrome非常憎恨自签署证书Why does Chrome hate self-signed certificates so much。 浏览器依然不能像http那样通过IP地址访问服务器,对于https://serverIP这种方式依然报警: Server’s certificate does not match the URL 所以,需要根据证书里Issued to的域名信息修改: $ vi /etc/hosts 10.10.0.1 dehaou14-n501jw $ ping dehaou14-n501jw ok https://dehaou14-n501jw/ ok 8、客户端的代码解析 首先要制作一个类似index.html的文件,一般放在static文件夹下面,因为server.js里面默认指定static文件夹作为客户浏览器前端访问html的位置。<<

智能机器人(46):目标识别

ORK(Object Recognition Kitchen)是一套以template matching方法为主的目标识别工具,把看到的物体跟资料库中的物体比对,够相似,就算辨识成功。
ORK, based on OpenCV+PCL。OpenCV’s purpose is to help turn “seeing” into perception. We will also use active depth sensing to “cheat”
ORK, was built on top of ecto which is a lightweight hybrid C++/Python framework for organizing computations as directed acyclic graphs.
There is currently no unique method to perform object recognition. Objects can be textured, non textured, transparent, articulated, etc. For this reason, the Object Recognition Kitchen was designed to easily develop and run simultaneously several object recognition techniques. In short, ORK takes care of all the non-vision aspects of the problem for you (database management, inputs/outputs handling, robot/ROS integration …) and eases reuse of code.
REF: http://www.ais.uni-bonn.de/~holz/spme/talks/01_Bradski_SemanticPerception_2011.pdf

1. ORK安装

1.1 预安装ORK依赖
$ export DISTRO=indigo
$ sudo apt-get install libopenni-dev ros-${DISTRO}-catkin ros-${DISTRO}-ecto* ros-${DISTRO}-opencv-candidate ros-${DISTRO}-moveit-msgs
maybe only need tihs ros-${DISTRO}-opencv-candidate.

1.2. 安装ORK包
install all about ork:
$ sudo apt-get install ros-indigo-object-recognition-*
or,
$ sudo apt-get install ros-indigo-object-recognition-core ros-indigo-object-recognition-linemod ros-indigo-object-recognition-msgs ros-indigo-object-recognition-renderer ros-indigo-object-recognition-ros ros-indigo-object-recognition-ros-visualization

3. CouchDb数据库

In ORK everything is stored in a database: objects, models, training data. 对象、模型、训练数据。
Couchdb from Apache is our main database and it is the most tested implementation. To set up local instance, what you must do is install couchdb, and ensure that the service has started.

3.1 先安装CouchDB工具
$ sudo apt-get install couchdb
检查是否安装成功
$ curl -X GET http://localhost:5984
如果成功的话:
— {“couchdb”:”Welcome”,”version”:”1.0.1″}

3.2 数据可视化的Web接口
We have a set of webpages that may be pushed to your couchdb instance that help browse the objects that you training or models created.
a) First installed couch-app
$ sudo apt-get install python-pip
$ sudo pip install -U couchapp
数据库中可视化数据。
We provide a utility that automatically installs the visualizer on the DB. This will upload the contents of the directory to collection in your couchdb instance, called or_web_ui.
$ rosrun object_recognition_core push.sh

After this you can browse the web ui using the url:
http://localhost:5984/or_web_ui/_design/viewer/index.html
now, is empty.

5. TOD 的 Quickstart

ORK比较像一个框架,包含了好几种演算法,其中一种是TOD。
$ roscore
$ roslaunch openni_launch openni.launch

5.1 设置工作区。 Setup capture workspace
$ rosrun object_recognition_capture orb_template -o my_textured_plane
保持平面正视,图像居于中心。 ‘s’保存图像,例如上面的 my_textured_plane。 ‘q’退出。

可以跟踪下,以查看是否合格的模板。
$ rosrun object_recognition_capture orb_track –track_directory my_textured_plane

NOTE:
Use the SXGA (roughly 1 megapixel) mode of your openni device if possible.
$ rosrun dynamic_reconfigure dynparam set /camera/driver image_mode 1
$ rosrun dynamic_reconfigure dynparam se t /camera/driver depth_registration True

5.2 获取目标。 Capture objects
Once you are happy with the workspace tracking, its time to capure an object.
a) 预览模式。 Place an object at the origin of the workspace. An run the capture program in preview mode. Make sure the mask and pose are being picked up.
$ rosrun object_recognition_capture capture -i my_textured_plane –seg_z_min 0.01 -o silk.bag –preview

b) 正式模式。 When satisified by the preview mode, run it for real. The following will capture a bag of 60 views where each view is normally distributed on the view sphere. The mask and pose displays should only refresh when a novel view is captured. The program will finish when 35 (-n) views are captured. Press ‘q’ to quit early.
$ rosrun object_recognition_capture capture -i my_textured_plane –seg_z_min 0.01 -o silk.bag

c) 数据上传到DB。 Now time for upload. Make sure you install couch db on your machine. Give the object a name and useful tags seperated by a space, e.g. ‘milk soy silk’.
$ rosrun object_recognition_capture upload -i silk.bag -n ‘Silk’ milk soy silk –commit

5.3 训练对象。 Train objects
Repeat the steps above for the objects you would like to recognize. Once you have captured and uploaded all of the data, it time to mesh and train object recognition.
a) 生成mesh。 Meshing objects can be done in a batch mode as follows:
$ rosrun object_recognition_reconstruction mesh_object –all –visualize –commit

b) 查看。 The currently stored models are on:
http://localhost:5984/or_web_ui/_design/viewer/meshes.html

c) 训练。 Next objects should be trained. It may take some time between objects, this is normal. Also, here assumes that you are using TOD which only works for textured objects. Please refer to the documentation of other methods.
$rosrun object_recognition_core training -c `rospack find object_recognition_tod`/conf/training.ork –visualize

5.4 检测对象。 Detect objects
Now we’re ready for detection.
First launch rviz, it should be subscribed to the right markers for recognition results. /markers is used for the results, and it is a marker array.
rosrun object_recognition_core detection -c `rospack find object_recognition_tod` /conf/detection.ros.ork –visualize

7. Linemod 的 Tutorials

ORK比较像一个框架,包含了好几种演算法,其中一种Linemod。 Linemod is a pipeline that implements one of the best methods for generic rigid object recognition and it proceeds using very fast template matching.
REF: http://ar.in.tum.de/Main/StefanHinterstoisser.
这里练习:
1)将一个对象的mesh添加到DB中;
2)学习如何手动添加对象到DB中;
3)ORK数据库中数据的可视化。

7.1 添加目标obj。 Creating an object in the DB
ORK主要是识别对象的,首先需要将对象存储在DB中。像ORK 3d capture这样的管道可以用来创建对象。但是也要从内核(core)中用脚本对其进行处理。
$ rosrun object_recognition_core object_add.py -n “coke ” -d “A universal can of coke” –commit
— Stored new object with id: 9633ca8fc57f2e3dad8726f68a000326
执行上面这个指令之后,就可以从查看自己的资料库里是否已经新增这个物体:
http://localhost:5984/_utils/database.html?object_recognition/_design/objects/_view/by_object_name
点击对象,可以看到对象的信息,尤其是对象的id,DB的每个元素都有其自己的哈希表作为唯一ID(防止你给不同的对象相同的命名)。记住这个ID 9633ca8fc57f2e3dad8726f68a000326 :

7.2 添加模型mesh。 adding a mesh for the object
首先需要用DB接口获取了正确的对象id。
这里先获取一个例子中的模型mesh。When you install ORK, the database is empty. Luckily, ORK tutorials comes with a 3D mesh of a coke that can be downloaded here:
接下来要指定这个目标的3D模型, 在ork_tutorials这个例程里有coke.stl 档,就是可乐罐的3D模型。先下载这个例程,并且编译:
$ git clone https://github.com/wg-perception/ork_tutorials
$ cd ..
注意编译。 $ cd ..
$ catkin_make

NOTICE:下面这个指令中那串像乱码的内容就是object 的ID ,这串内容要从上一步自己的资料库里才能查得到。
You can upload the object and its mesh to the database with the scripts from the core:
$ rosrun object_recognition_core mesh_add.py 9633ca8fc57f2e3dad8726f68a000326 /home/dehaou1404/catkin_ws/src/ork_tutorials/data/coke.stl –commit
— Stored mesh for object id : 9633ca8fc57f2e3dad8726f68a000326

7.3 可视化数据。 Visualizing the object
Now, if you want to visualize the object in the db, you can just go to the visualization URL:
http://localhost:5984/or_web_ui/_design/viewer/meshes.html

you should see the following:

7.4 howto Delete the object
$ rosrun object_recognition_core object_delete.py OBJECT_ID

7.5 学习
Now, you can learn objects models from the database.
Execute the Linemod in the training mode with the configuration file through the -c option. The configuration file should define a pipeline that reads data from the database and computes objects models.
$ rosrun object_recognition_core training -c `rospack find object_recognition_linemod`/conf/training.ork
这个training指令会利用资料库里的3D 模型建立辨识时所需要的template,如果执行成功,你会看到如下的讯息
Training 1 objects.
computing object_id: 9633ca8fc57f2e3dad8726f68a000326
Info, T0: Load /tmp/fileeW7jSB.stl
Info, T0: Found a matching importer for this file format
Info, T0: Import root directory is ‘/tmp/’
Info, T0: Entering post processing pipeline
Info, T0: Points: 0, Lines: 0, Triangles: 1, Polygons: 0 (Meshes, X = removed)
Error, T0: FindInvalidDataProcess fails on mesh normals: Found zero-length vector
Info, T0: FindInvalidDataProcess finished. Found issues …
Info, T0: GenVertexNormalsProcess finished. Vertex normals have been calculated
Error, T0: Failed to compute tangents; need UV data in channel0
Info, T0: JoinVerticesProcess finished | Verts in: 1536 out: 258 | ~83.2%
Info, T0: Cache relevant are 1 meshes (512 faces). Average output ACMR is 0.669922
Info, T0: Leaving post processing pipeline

7.4 识别
Once learned, objects can be detected from the input point cloud. In order to detect object continuously, execute the Linemod in the detection mode with the configuration file that defines a source, a sink, and a pipeline
REF: http://wg-perception.github.io/object_recognition_core/detection/detection.html
$ roslaunch openni_launch openni.launch
$ rosrun dynamic_reconfigure dynparam set /camera/driver depth_registration True
$ rosrun dynamic_reconfigure dynparam set /camera/driver image_mode 2
$ rosrun dynamic_reconfigure dynparam set /camera/driver depth_mode 2
$ rosrun topic_tools relay /camera/depth_registered/image_raw /camera/depth_registered/image
$ rosrun object_recognition_core detection -c `rospack find object_recognition_linemod`/conf/detection.ros.ork
^^^ igoned

7.5 Visualization with RViz
a) config rviz
$ roslaunch openni_launch openni.launch
$ rosrun rviz rviz
Set the Fixed Frame (in Global Options, Displays window) to /camera_depth_optical_frame.
Add a PointCloud2 display and set the topic to /camera/depth/points. This is the unregistered point cloud in the frame of the depth (IR) camera and it is not matched with the RGB camera images. For visualization of the registered point cloud, the depth data could be aligned with the RGB data. To do it, launch the dynamic reconfigure GUI:
$ rosrun rqt_reconfigure rqt_reconfigure
Select /camera/driver from the drop-down menu and enable the depth_registration checkbox. In RViz, change the PointCloud2 topic to /camera/depth_registered/points and set the Color Transformer to RGB8 to see both color and 3D point cloud of your scene.
REF: https://wg-perception.github.io/ork_tutorials/tutorial03/tutorial.html
b) 然后就可以用Rviz查看识别的目标。
Go to RViz and add the OrkObject in the Displays window.
Select the OrkObject topic and the parameters to display: id / name / confidence.
Here, we show an example of detecting two objects (a coke and a head of NAO) and the outcome visualized in RViz.
For each recognized object, you can visualize its point cloud and also a point cloud of the matching object from the database. For this, compile the package with the CMake option -DLINEMOD_VIZ_PCD=ON. Once an object is recognized, its point cloud from the sensor 3D data is visualized as shown in the following image (check blue color). The cloud is published under the /real_icpin_ref topic.
For the same recognized object, we can visualize the point cloud of the matching object from the database as shown in the following image (check yellow color). The point cloud is created from the mesh stored in the database by visualizing at a pose returned by Linemod and refined by ICP. The cloud is published under the /real_icpin_model topic.
REF: http://wg-perception.github.io/ork_tutorials/tutorial03/tutorial.html

11. 算法简介
这个Linemod算法的核心概念就是整合多种不同的modalities,把modality想成物体的不同特征可能比较好理解,
例如下图中就有两种modalities – gradient 跟surface normal,而因为这两种特征所表达的特性不一样,所以可以互补,进而达到更好的辨识效果。

所以说,Linemod 需要先有已知的物体模型,然后先取得这个物体各种modlaities 的template,这样在辨识的时候就可以拿template 来比对了。
REF: http://blog.techbridge.cc/2016/05/14/ros-object-recognition-kitchen/

REF:http://wenku.baidu.com/link?url=hK1FMB2yR_frGcLxmQKAiZYhgKJdt0LBpaK1Sy-dY0ZgXuR2_jnLDYDBlaIz9eSLyvWOd66fVv-t3kH6NVapF2at3Cv9P1DfoxIyAn3MAuG

智能机器人(45):人脸识别

1. 人脸检测 vs 人脸识别
2. 人脸识别的OpcnCV实现
3. 人脸识别的ROS实现
4. 基于RGBD-camera的人脸识别

后面所属人脸识别基于opencv,由cvbridge桥接到ros,先把张三李四的图片训练加入到数据库,然后由actionlib提供识别服务。
这是训练:
人脸识别的训练

这是识别:
人脸识别

  • 1. 人脸检测 vs 人脸识别

人脸检识别一般包括人脸检测和人脸识别两步:
1. 人脸检测 Face Detection, a photo is searched to find any face.
2. 人脸识别 Face Recognition, detected and processed face compared to a database of known faces to decide who that person is.

1. 人脸检测已经达到准确度90-95%,例如OpenCV’s Face Detector,麻烦点的是It is usually harder to detect a person’s face when they are viewed from the side or at an angle, and sometimes this requires 3D Head Pose Estimation. It can also be very difficult to detect a person’s face if the photo is not very bright, or if part of the face is brighter than another or has shadows or is blurry or wearing glasses, etc.
2. 但是人脸识别准确度就不乐观, However, Face Recognition is much less reliable than Face Detection, generally 30-70% accurate. Face Recognition has been a strong field of research since the 1990s, but is still far from reliable, and more techniques are being invented each year.

后面所属人脸识别用特征脸方法 Eigen faces,也称作主成分分析法 PCA:Principal Component Analysis,a simple and popular method of 2D FR。

1.1. 特征脸
特征向量源于概率分布的协方差矩阵。该协方差矩阵源于特征脸的集合。实现了降维。
Eigenfaces is the name given to a set of eigenvectors when they are used in the computer vision problem of human face recognition.
The approach of using eigenfaces for recognition was developed by Sirovich and Kirby (1987) and used by Matthew Turk and Alex Pentland in face classification.
The eigenvectors are derived from the covariance matrix of the probability distribution over the high-dimensional vector space of face images.
The eigenfaces themselves form a basis set of all images used to construct the covariance matrix.
This produces dimension reduction by allowing the smaller set of basis images to represent the original training images. Classification can be achieved by comparing how faces are represented by the basis set.

1.2. 原理
特征脸就是包含主成分的特征向量。
A set of eigenfaces can be generated by performing a mathematical process called principal component analysis (PCA) on a large set of images depicting different human faces.
Informally, eigenfaces can be considered a set of “standardized face ingredients”, derived from statistical analysis of many pictures of faces.
Any human face can be considered to be a combination of these standard faces. For example, one’s face might be composed of the average face plus 10% from eigenface 1, 55% from eigenface 2, and even -3% from eigenface 3.

1.3 实现
1.3.1. 先准备训练集。Prepare a training set of face images. The pictures constituting the training set should have been taken under the SAME lighting conditions, and must be NORMOLIZED to have the eyes and mouths aligned across all images. They must also be all resampled to a common pixel resolution (r × c). Each image is treated as one vector, simply by concatenating the rows of pixels in the original image, resulting in a single row with r × c elements. For this implementation, it is assumed that all images of the training set are stored in a single matrix T, where each column of the matrix is an image.
1.3.2. 减去均值矩阵,抛弃直流分量。Subtract the mean. The average image a has to be calculated and then subtracted from each original image in T.
1.3.3. 计算特征向量。Calculate the eigenvectors and eigenvalues of the covariance matrix S. Each eigenvector has the same dimensionality (number of components) as the original images, and thus can itself be seen as an image. The eigenvectors of this covariance matrix are therefore called eigenfaces. They are the directions in which the images differ from the mean image. Usually this will be a computationally expensive step (if at all possible), but the practical applicability of eigenfaces stems from the possibility to compute the eigenvectors of S efficiently, without ever computing S explicitly, as detailed below.
1.3.4. 通过阈值化获得主成分。Choose the principal components. Sort the eigenvalues in descending order and arrange eigenvectors accordingly. The number of principal components k is determined arbitrarily by setting a threshold ε on the total variance.
如此,即可。
These eigenfaces can now be used to represent both existing and new faces: we can project a new (mean-subtracted) image on the eigenfaces and thereby record how that new face differs from the mean face. The eigenvalues associated with each eigenface represent how much the images in the training set vary from the mean image in that direction.

  • 2. 人脸识别的OpcnCV实现

2.1. detect a face with OpenCV’s Face Detector
opencv经典的是使用哈尔级联 The OpenCV library makes it fairly easy to detect a frontal face in an image using its Haar Cascade Face Detector
The function “cvHaarDetectObjects” in OpenCV performs the actual face detection, it is best to write a wrapper function:
// Perform face detection on the input image, using the given Haar Cascade.
// Returns a rectangle for the detected region in the given image.
CvRect detectFaceInImage(IplImage *inputImg, CvHaarClassifierCascade* cascade)
{
// Smallest face size.
CvSize minFeatureSize = cvSize(20, 20);
// Only search for 1 face.
int flags = CV_HAAR_FIND_BIGGEST_OBJECT | CV_HAAR_DO_ROUGH_SEARCH;
// How detailed should the search be.
float search_scale_factor = 1.1f;
IplImage *detectImg;
IplImage *greyImg = 0;

// Detect all the faces in the greyscale image.
t = (double)cvGetTickCount();
rects = cvHaarDetectObjects( detectImg, cascade, storage,
search_scale_factor, 3, flags, minFeatureSize);
t = (double)cvGetTickCount() – t;
ms = cvRound( t / ((double)cvGetTickFrequency() * 1000.0) );
nFaces = rects->total;
printf(“Face Detection took %d ms and found %d objectsn”, ms, nFaces);

// Get the first detected face (the biggest).
if (nFaces > 0)
rc = *(CvRect*)cvGetSeqElem( rects, 0 );
else
rc = cvRect(-1,-1,-1,-1);    // Couldn’t find the face.
}

Now you can simply call “detectFaceInImage” whenever you want to find a face in an image.

2.2. specify the face classifier that OpenCV use to detect the face
分类器的方案 For example, OpenCV comes with several different classifiers for frontal face detection, as well as some profile faces (side view), eye detection, nose detection, mouth detection, whole body detection, etc. You can actually use this function with any of these other detectors if you want, or even create your own custom detector such as for car or person detection (read here), but since frontal face detection is the only one that is very reliable, it is the only one we discuss.

For frontal face detection, you can chose one of these Haar Cascade Classifiers that come with OpenCV (in the “datahaarcascades” folder):
“haarcascade_frontalface_default.xml”
“haarcascade_frontalface_alt.xml”
“haarcascade_frontalface_alt2.xml”
“haarcascade_frontalface_alt_tree.xml”

So you could do this in your program for face detection:
// Haar Cascade file, used for Face Detection.
char *faceCascadeFilename = “haarcascade_frontalface_alt.xml”;
// Load the HaarCascade classifier for face detection.
CvHaarClassifierCascade* faceCascade;
faceCascade = (CvHaarClassifierCascade*)cvLoad(faceCascadeFilename, 0, 0, 0);
if( !faceCascade ) {
printf(“Couldnt load Face detector ‘%s’n”, faceCascadeFilename);
exit(1);
}

Now that you have detected a face, you can use that face image for Face Recognition

2.3. 预处理preprocess images for Face Recognition

Now that you have detected a face, you can use that face image for Face Recognition. However, if you tried to simply perform face recognition directly on a normal photo image, you will probably get less than 10% accuracy!

It is extremely important to apply various image pre-processing techniques to standardize the images that you supply to a face recognition system. Most face recognition algorithms are extremely sensitive to lighting conditions, so that if it was trained to recognize a person when they are in a dark room, it probably wont recognize them in a bright room, etc. This problem is referred to as “lumination dependent”, and there are also many other issues, such as the face should also be in a very consistent position within the images (such as the eyes being in the same pixel coordinates), consistent size, rotation angle, hair and makeup, emotion (smiling, angry, etc), position of lights (to the left or above, etc). This is why it is so important to use a good image preprocessing filters before applying face recognition. You should also do things like removing the pixels around the face that aren’t used, such as with an elliptical mask to only show the inner face region, not the hair and image background, since they change more than the face does.

For simplicity, the face recognition system I will show you is Eigenfaces using greyscale images. So I will show you how to easily convert color images to greyscale (also called ‘grayscale’), and then easily apply Histogram Equalization as a very simple method of automatically standardizing the brightness and contrast of your facial images. For better results, you could use color face recognition (ideally with color histogram fitting in HSV or another color space instead of RGB), or apply more processing stages such as edge enhancement, contour detection, motion detection, etc. Also, this code is resizing images to a standard size, but this might change the aspect ratio of the face. You can read my tutorial HERE on how to resize an image while keeping its aspect ratio the same.

Here is some basic code to convert from a RGB or greyscale input image to a greyscale image, resize to a consistent dimension, then apply Histogram Equalization for consistent brightness and contrast:
// Either convert the image to greyscale, or use the existing greyscale image.
IplImage *imageGrey;
if (imageSrc->nChannels == 3) {
imageGrey = cvCreateImage( cvGetSize(imageSrc), IPL_DEPTH_8U, 1 );
// Convert from RGB (actually it is BGR) to Greyscale.
cvCvtColor( imageSrc, imageGrey, CV_BGR2GRAY );
}
else {
// Just use the input image, since it is already Greyscale.
imageGrey = imageSrc;
}

// Resize the image to be a consistent size, even if the aspect ratio changes.
IplImage *imageProcessed;
imageProcessed = cvCreateImage(cvSize(width, height), IPL_DEPTH_8U, 1);
// Make the image a fixed size.
// CV_INTER_CUBIC or CV_INTER_LINEAR is good for enlarging, and
// CV_INTER_AREA is good for shrinking / decimation, but bad at enlarging.
cvResize(imageGrey, imageProcessed, CV_INTER_LINEAR);

// Give the image a standard brightness and contrast.
cvEqualizeHist(imageProcessed, imageProcessed);

…..  Use ‘imageProcessed’ for Face Recognition ….

if (imageGrey)
cvReleaseImage(&imageGrey);
if (imageProcessed)
cvReleaseImage(&imageProcessed);
Now that you have a pre-processed facial image

2.4. Eigenfaces be used for Face Recognition

Now that you have a pre-processed facial image, you can perform Eigenfaces (PCA) for Face Recognition. OpenCV comes with the function “cvEigenDecomposite()”, which performs the PCA operation, however you need a database (training set) of images for it to know how to recognize each of your people.

So you should collect a group of preprocessed facial images of each person you want to recognize. For example, if you want to recognize someone from a class of 10 students, then you could store 20 photos of each person, for a total of 200 preprocessed facial images of the same size (say 100×100 pixels).

Use “Principal Component Analysis” to convert all your 200 training images into a set of “Eigenfaces” that represent the main differences between the training images. First it will find the “average face image” of your images by getting the mean value of each pixel. Then the eigenfaces are calculated in comparison to this average face, where the first eigenface is the most dominant face differences, and the second eigenface is the second most dominant face differences, and so on, until you have about 50 eigenfaces that represent most of the differences in all the training set images.

In these example images above you can see the average face and the first and last eigenfaces that were generated from a collection of 30 images each of 4 people. Notice that the average face will show the smooth face structure of a generic person, the first few eigenfaces will show some dominant features of faces, and the last eigenfaces (eg: Eigenface 119) are mainly image noise. You can see the first 32 eigenfaces in the image below.
*** To explain Eigenfaces (Principal Component Analysis) in simple terms, Eigenfaces figures out the main differences between all the training images, and then how to represent each training image using a combination of those differences.

So for example, one of the training images might be made up of:
(averageFace) + (13.5% of eigenface0) – (34.3% of eigenface1) + (4.7% of eigenface2) + … + (0.0% of eigenface199).
Once it has figured this out, it can think of that training image as the 200 ratios:
{13.5, -34.3, 4.7, …, 0.0}.

It is indeed possible to generate the training image back from the 200 ratios by multiplying the ratios with the eigenface images, and adding the average face. But since many of the last eigenfaces will be image noise or wont contribute much to the image, this list of ratios can be reduced to just the most dominant ones, such as the first 30 numbers, without effecting the image quality much. So now it’s possible to represent all 200 training images using just 30 eigenface images, the average face image, and a list of 30 ratios for each of the 200 training images.

Interestingly, this means that we have found a way to compress the 200 images into just 31 images plus a bit of extra data, without loosing much image quality. But this tutorial is about face recognition, not image compression, so we will ignore that 🙂

o recognize a person in a new image, it can apply the same PCA calculations to find 200 ratios for representing the input image using the same 200 eigenfaces. And once again it can just keep the first 30 ratios and ignore the rest as they are less important. It can then search through its list of ratios for each of its 20 known people in its database, to see who has their top 30 ratios that are most similar to the 30 ratios for the input image. This is basically a method of checking which training image is most similar to the input image, out of the whole 200 training images that were supplied.

2.5. Implementing Offline Training

For implementation of offline training, where files are used as input and output through the command-line, I am using a similar method as the Face Recognition with Eigenface implementation in Servo Magazine, so you should read that article first, but I have made a few slight changes.

Basically, to create a facerec database from training images, you create a text file that lists the image files and which person each image file represents. For example, you could put this into a text file called “4_images_of_2_people.txt”:
1 Shervin dataShervinShervin1.bmp
1 Shervin dataShervinShervin2.bmp
1 Shervin dataShervinShervin3.bmp
1 Shervin dataShervinShervin4.bmp
2 Chandan dataChandanChandan1.bmp
2 Chandan dataChandanChandan2.bmp
2 Chandan dataChandanChandan3.bmp
2 Chandan dataChandanChandan4.bmp

This will tell the program that person 1 is named “Shervin”, and the 4 preprocessed facial photos of Shervin are in the “dataShervin” folder, and person 2 is called “Chandan” with 4 images in the “dataChandan” folder. The program can then loaded them all into an array of images using the function “loadFaceImgArray()”. Note that for simplicity, it doesn’t allow spaces or special characters in the person’s name, so you might want to enable this, or replace spaces in a person’s name with underscores (such as Shervin_Emami).

To create the database from these loaded images, you use OpenCV’s “cvCalcEigenObjects()” and “cvEigenDecomposite()” functions, eg:
// Tell PCA to quit when it has enough eigenfaces.
CvTermCriteria calcLimit = cvTermCriteria( CV_TERMCRIT_ITER, nEigens, 1);

// Compute average image, eigenvectors (eigenfaces) and eigenvalues (ratios).
cvCalcEigenObjects(nTrainFaces, (void*)faceImgArr, (void*)eigenVectArr,
CV_EIGOBJ_NO_CALLBACK, 0, 0, &calcLimit,
pAvgTrainImg, eigenValMat->data.fl);

// Normalize the matrix of eigenvalues.
cvNormalize(eigenValMat, eigenValMat, 1, 0, CV_L1, 0);

// Project each training image onto the PCA subspace.
CvMat projectedTrainFaceMat = cvCreateMat( nTrainFaces, nEigens, CV_32FC1 );
int offset = projectedTrainFaceMat->step / sizeof(float);
for(int i=0; i<nTrainFaces; i++) {
cvEigenDecomposite(faceImgArr[i], nEigens, eigenVectArr, 0, 0,
pAvgTrainImg, projectedTrainFaceMat->data.fl + i*offset);
}

You now have:
the average image “pAvgTrainImg”,
the array of eigenface images “eigenVectArr[]” (eg: 200 eigenfaces if you used nEigens=200 training images),
the matrix of eigenvalues (eigenface ratios) “projectedTrainFaceMat” of each training image.

These can now be stored into a file, which will be the face recognition database. The function “storeTrainingData()” in the code will store this data into the file “facedata.xml”, which can be reloaded anytime to recognize people that it has been trained for. There is also a function “storeEigenfaceImages()” in the code, to generate the images shown earlier, of the average face image to “out_averageImage.bmp” and eigenfaces to “out_eigenfaces.bmp”.

2.6 Implementing Offline Recognition

For implementation of the offline recognition stage, where the face recognition system will try to recognize who is the face in several photos from a list in a text file, I am also using an extension of the Face Recognition with Eigenface implementation in Servo Magazine.

The same sort of text file that is used for offline training can also be used for offline recognition. The text file lists the images that should be tested, as well as the correct person in that image. The program can then try to recognize who is in each photo, and check the correct value in the input file to see whether it was correct or not, for generating statistics of its own accuracy.

The implementation of the offline face recognition is almost the same as offline training:
The list of image files (preprocessed faces) and names are loaded into an array of images, from the text file that is now used for recognition testing (instead of training). This is performed in code by “loadFaceImgArray()”.
The average face, eigenfaces and eigenvalues (ratios) are loaded from the face recognition database file “facedata.xml”, by the function “loadTrainingData()”.
Each input image is projected onto the PCA subspace using the OpenCV function “cvEigenDecomposite()”, to see what ratio of eigenfaces is best for representing this input image.
But now that it has the eigenvalues (ratios of eigenface images) to represent the input image, it looks for the original training image that had the most similar ratios. This is done mathematically in the function “findNearestNeighbor()” using the “Euclidean Distance”, but basically it checks how similar the input image is to each training image, and finds the most similar one: the one with the least distance in Euclidean Space. As mentioned in the Servo Magazine article, you might get better results if you use the Mahalanobis space (define USE_MAHALANOBIS_DISTANCE in the code).
The distance between the input image and most similar training image is used to determine the “confidence” value, to be used as a guide of whether someone was actually recognized or not. A confidence of 1.0 would mean a good match, and a confidence of 0.0 or negative would mean a bad match. But beware that the confidence formula I use in the code is just an extremely basic confidence metric that isn’t reliable, so if you need something more reliable you should look for “Face Verification” algorithms. If you find that it gives misleading values for your images, you should ignore it or disable it in the code (eg: set the confidence always to 1.0).

Once it knows which training image is most similar to the input image, and assuming the confidence value is not too low (it should be atleast 0.6 or higher), then it has figured out who that person is, in other words, it has recognized that person!

2.7 mprove the Face Recognition accuracy

To improve the recognition performance, there are MANY things that can be improved here (look at commercial Face Recognition systems such as SPOTR for examples), and some improvements can be fairly easy to implement. For example, you could add color processing, edge detection, etc.

You can usually improve the face recognition accuracy by using more input images, atleast 50 per person, by taking more photos of each person, particularly from different angles and lighting conditions. If you cant take more photos, there are several simple techniques you could use to obtain more training images, by generating new images from your existing ones:
You could create mirror copies of your facial images, so that you will have twice as many training images and it wont have a bias towards left or right.
You could translate or resize or rotate your facial images slightly to produce many alternative images for training, so that it will be less sensitive to exact conditions.
You could add image noise to have more training images that improve the tolerance to noise.

Remember that it is important to have a lot of variation of conditions for each person, so that the classifier will be able to recognize the person in different lighting conditions and positions, instead of looking for specific conditions. But it’s also important to make sure that a set of images for a person is not too varied, such as if you rotated some images by 90 degrees. This would make the classifier to be too generic and also give very bad results, so if you think you will have a set of images with too much variance (such as rotation more than 20 degrees), then you could create separate sets of training images for each person. For example, you could train a classifier to recognize “John_Facing_Forward” and another one for “John_Facing_Left” and other ones “Mary_Facing_Forward”, “Mary_Facing_Left”, etc. Then each classifier can have a lot of variance but not too much, and you simply need to associate the different classifiers for each person with that one person (ie: “John” or “Mary”).

That’s why you can often get very bad results if you don’t use good preprocessing on your images. As I mentioned earlier, Histogram Equalization is a very basic image preprocessing method that can make things worse in some situations, so you will probably have to combine several different methods until you get decent results.

That’s why face recognition is relatively easy to do in realtime if you are training on someone and then instantly trying to recognize them after, since it will be the same camera, and background will be the same, their expressions will be almost the same, the lighting will be the same, and the direction you are viewing them from will be the same. So you will often get good recognition results at that moment. But once you try to recognize them from a different direction or from a different room or outside or on a different time of the day, it will often give bad results!

Alternative techniques to Eigenfaces for Face Recognition:
Something you should know is that Eigenfaces is considered the simplest method of accurate face recognition, but many other (much more complicated) methods or combinations of multiple methods are slightly more accurate. So if you have tried the hints above for improving your training database and preprocessing but you still need more accuracy, you will probably need to learn some more complicated methods, or for example you could figure out how to combine separate Eigenface models for the eyes, nose ; mouth.

  • 3. 人脸识别的ROS实现

3.1. 源代码安装face_recognition
$ cd ~/catkin_ws/src
$ git clone https://github.com/procrob/procrob_functional.git –branch catkin
$ cd ~/catkin_ws
$ catkin_make
$ source ~/catkin_ws/devel/setup.bash

3.2. 流程
训练图片: Training images are stored in the data directory.
训练图片列表: Training images are listed in the train.text file.

The ‘train.txt’ follows a specific format which is best understood by looking at the example train.txt file provided in the package. Note that person numbers start from 1, and spaces or special characters are not allowed in persons’ names.

The program trains from the training examples listed in the train.txt, and create an Eigenfaces database which is stored in the ‘facedata.xml’.

Face detection is performed using a haarcascade classifier ‘haarcascade_frontalface_alt.xml’. The ‘data’ folder and ‘train.txt’, ‘facedata.xml’ and ‘haarcascade_frontalface_alt.xml’ files should be placed in the program’s working directory (the directory from which you execute the program).

When the face_recognition program starts: 1) If facedata.xml exists, the Eigenfaces database is loaded from facedata.xml. 2) If facedata.xml does not exist, the program tries to train and create Eigenfaces database from the training images listed in train.txt. Regardless of if the Eigenfaces database is loaded/created at start up or not, you can always add training images directly from the video stream and then update the Eigenfaces database by (re)training. Note: when the program (re)trains, the content of facedata.xml is disregarded and the program trains only based on the training images listed in train.txt.

3.3. 示例
For demonstration purposes an actionlib client example for the face_recognition simple actionlib server has been provided
The client subscribes to face_recognition/FRClientGoal messages. Each FRClientGoal message contains an ‘order_id’ and an ‘order_argument’ which specify a goal to be executed by the face_recognition server.

After receiving a message, the client sends the corresponding goal to the server. By registering relevant call back functions, the client receives feedback and result information from the execution of goals in the server and prints such information on the terminal.

3.3.1 发布视频流到主题/camera/image_raw.
For example you can use usb_cam to publish images from your web cam ,这个launch需要自己写:
$ roslaunch usb_cam usb_cam-zdh.launch
now add one node: /usb_cam_node and several topics.
can check it:
$ rosrun image_view image_view image:=/camera/image_raw.
可能需要安装驱动:
$ sudo apt-get install ros-indigo-usb-cam

3.3.2 启动Fserver
Fserver is a ROS node that provides a simple actionlib server interface for performing different face recognition functionalities in video stream.
Start the Fserver node:
$ cd /home/dehaou1404/catkin_ws/src/procrob_functional
$ rosrun face_recognition Fserver

* THE server subscribed topic:
— /camera/image_raw – video stream (standard ROS image transport)

* THE server involve parameters:
confidence_value (double, default = 0.88)
— add_face_number (int, default = 25)

3.3.3 启动FClient
Fclient is a ROS node that implements an actionlib client example for the face_recognition simple actionlib server (i.e. ‘Fserver’). ‘Fclient’ is provided for demonstration and testing purposes.

Each FRClientGoal message has an ‘order_id’ and an ‘order_argument’, which specify a goal to be executed by the Fserver.
After receiving a message, Fclient sends the corresponding goal to the Fserver. By registering relevant call back functions, Fclient receives feedback and result information from the execution of goals in the Fserver and prints such information on its terminal.

Run the face recognition client:
$ cd /home/dehaou1404/catkin_ws/src/procrob_functional
$ rosrun face_recognition Fclient
* THE client subscribe the topic:
— fr_order (face_recognition/FRClientGoal)

3.3.4. 训练和识别
Publish messages on topic /fr_order, to test different face recognition functionalities.
NOTICE: notice the info printed from client terminal after each command

3.3.4.1. 采集训练图片。
Acquire training images for one face, notice this one should try to appear in the video stream.
$ rostopic pub -1 /fr_order face_recognition/FRClientGoal — 2 “dehaoZhang”
in the server…:
[ INFO] [1483429069.013208800]: No face was detected in the last frame
[ INFO] [1483429069.266604135]: No face was detected in the last frame
[ INFO] [1483429069.512622293]: No face was detected in the last frame
[ INFO] [1483429069.728653884]: Storing the current face of ‘dehaoZhang’ into image ‘data/6_dehaoZhang1.pgm’.
[ INFO] [1483429075.494751180]: Storing the current face of ‘dehaoZhang’ into image ‘data/6_dehaoZhang24.pgm’.
[ INFO] [1483429075.745061728]: Storing the current face of ‘dehaoZhang’ into image ‘data/6_dehaoZhang25.pgm’.
in the client…:
[ INFO] [1483429043.355890495]: request for sending goal [2] is received
[ INFO] [1483429043.356973474]: Goal just went active
[ INFO] [1483429069.732158190]: Received feedback from Goal [2]
[ INFO] [1483429069.732210837]: A picture of dehaoZhang was successfully added to the training images
[ INFO] [1483429069.980983035]: Received feedback from Goal [2]
[ INFO] [1483429069.981020038]: A picture of dehaoZhang was successfully added to the training images
[ INFO] [1483429075.496298334]: Received feedback from Goal [2]
[ INFO] [1483429075.496374189]: A picture of dehaoZhang was successfully added to the training images
[ INFO] [1483429075.746450499]: Goal [2] Finished in state [SUCCEEDED]
[ INFO] [1483429075.746538156]: Pictures of dehaoZhang were successfully added to the training images

3.3.4.2. 训练样本集。
Retrain and update the database, so that you can be recognized
$ rostopic pub -1 /fr_order face_recognition/FRClientGoal — 3 “none”
in server…:
[ INFO] [1483429419.101123133]: People:
[ INFO] [1483429419.101157041]:
[ INFO] [1483429419.101187136]: ,
[ INFO] [1483429419.101213218]: ,
[ INFO] [1483429419.101241996]: ,
[ INFO] [1483429419.101268921]: ,
[ INFO] [1483429419.101300335]: ,
[ INFO] [1483429419.101334005]: .
[ INFO] [1483429419.101375442]: Got 150 training images.
in client…:
[[5~[ INFO] [1483429418.947685612]: request for sending goal [3] is received
[ INFO] [1483429418.948517146]: Goal just went active
[ INFO] [1483429421.776359616]: Goal [3] Finished in state [SUCCEEDED]

3.3.4.3. 识别
Recognize faces continuously. This would not stop until you preempt or cancel the goal. So lets preempt it by sending the next goal.
$ rostopic pub -1 /fr_order face_recognition/FRClientGoal — 1 “none”

3.3.4.4. 退出
$ rostopic pub -1 /fr_order face_recognition/FRClientGoal — 4 “none”

3.4. 参数和消息 Param and message
This message includes 2 fields:
— int order_id
— string order_argument

The FaceRecognitionGoal message has 2 fields: ‘order_id’ is an integer specifying a goal, ‘order_argument’ is a string used to specify an argument for the goal if necessary:

order_id = 2, then order_argument = person_name, =(Add face images)
Goal is to acquire training images for a NEW person. The video stream is processed for detecting a face which is saved and used as a training image for the new person. This process is continued until the desired number of training images for the new person is acquired. The name of the new person is provided as “order_argument”

order_id = 3, without args, =(Train)
The database is (re)trained from the training images

order_id = 0, without, =(Recognize Once)
Goal is to acknowledge the first face recognized in the video stream. When the first face is recognized with a confidence value higher than the desirable confidence threshold, the name of the person and the confidence value are sent back to the client as result.
order_id = 1, without, =(Recognize Continuously)
Goal is to continuously recognize faces in the video stream. Every face recognized with confidence value higher than the desirable confidence threshold and its confidence value are sent back to the client as feedback. This goal is persuaded for infinite time until it is canceled or preempted by another goal.

order_id = 4, without, =(Exit)
The program exits.

  • 4. 基于RGBD-camera的人脸识别

这个是RGBD相机,不同于常用webcam,增加了深度信息。
This package contains software for detecting heads and faces and recognizing people. Head and face detection utilize the Viola-Jones classifier on depth or color images, respectively. Face recognition of previously learned people is based on either Eigenfaces or Fisherfaces. The recognition method can be configured and trained with a simple interface as explained in the next section.

4.1. 安装cob_people_perception
$ roscd
$ cd ../src
$ git clone https://github.com/ipa-rmb/cob_people_perception.git
$ git clone https://github.com/ipa-rmb/cob_perception_common.git
$ cd ..
$ source ./devel/setup.bash
$ rosdep install –from-path src/ -y -i
$ catkin_make -DCMAKE_BUILD_TYPE=”Release”

4.2. 流程
Then install Openni (http://wiki.ros.org/openni_launch) and start the Openni driver (old Kinect, Asus) with
$ roslaunch openni_launch openni.launch
or the Openni2 driver (new Asus, http://wiki.ros.org/openni2_launch) with
$ roslaunch openni2_launch openni2.launch
or any other driver according to your used sensor.

When using the openni or openni2 driver, please ensure that depth_registration is activated (e.g. by using rosrun rqt_reconfigure rqt_reconfigure). Also check that the camera_namespace argument and the camera topics colorimage_in_topic and pointcloud_rgb_in_topic, which are set in the ros/launch/people_detection.launch file, correspond to the topic names of your camera driver.

4.3. 示例
4.3.1 Then launch people detection
$ roslaunch cob_people_detection people_detection.launch
or with
$ roslaunch cob_people_detection people_detection.launch using_nodelets:=true
The second version uses nodelets for the first stages of data processing which might yield a substantially better runtime if your processor is fast enough on single core usage.

Now a window should pop up and present you with the current image of the camera. Heads will be framed with a light blue rectangle and detected faces are indicated in light green. For your convenience, the package contains a client for easy usage of people detection. Start the client with
$ rosrun cob_people_detection people_detection_client

4.3.2 functions
No identification data will be available the first time you start the people detection node on your computer. To record some data adjust the frame rate of the camera, first, by choosing 5 – activate/deactivate sensor message gateway in the client and then enter 1 to activate the sensor message gateway. The frame rate should be chosen somewhere between 5 and 30 Hz depending on your computer’s power.
Choose an option:
1 – capture face images
2 – update database labels
3 – delete database entries
4 – load recognition model (necessary if new images/persons were added to the database)
>> 5 – activate/deactivate sensor message gateway <<
6 – get detections
q – Quit

Type 1 to activate or 2 to deactivate the sensor message gateway: 1
At which target frame rate (Hz) shall the sensor message gateway operate: 20
Gateway successfully opened.

Now select 1 – capture face images from the menu of the client and enter the name of the first person to capture. Please do not use any whitespaces in the name. Following, you are asked to select how to record the data: manually by pressing a button or automatically. In the manual mode, you have to press c each time an image shall be captured and q to finish recording. Make sure that only one person is in the image during recording, otherwise no data will be accepted because the matching between face and label would be ambiguous.

Choose an option:
>> 1 – capture face images <<
2 – update database labels
3 – delete database entries
4 – load recognition model (necessary if new images/persons were added to the database)
5 – activate/deactivate sensor message gateway
6 – get detections
q – Quit

Input the label of the captured person: ChuckNorris
Mode of data capture: 0=manual, 1=continuous: 0
Recording job was sent to the server …
Waiting for the capture service to become available …
[ INFO] [1345100028.962812337]: waitForService: Service [/cob_people_detection/face_capture/capture_image] has not been advertised, waiting…
[ INFO] [1345100028.985320699]: waitForService: Service [/cob_people_detection/face_capture/capture_image] is now available.
Hit ‘q’ key to quit or ‘c’ key to capture an image.
Image capture initiated …
image 1 successfully captured.
Image capture initiated …
image 2 successfully captured.
Image capture initiated …
image 3 successfully captured.
Image capture initiated …
image 4 successfully captured.
Finishing recording …
Data recording finished successfully.
Current State: SUCCEEDED   Message: Manual capture finished successfully.

If you are using one of the more modern recognition methods (choices 2=LDA2D or 3=PCA2D, can be set in file cob_people_detection/ros/launch/face_recognizer_params.yaml as parameter recognition_method) then please be aware that they require at least two different people in the training data. The next step, building a recognition model, will not start with only one person available with these algorithms. If you quit the program before recording a second person, the program might not start anymore. Then please delete all data from ~/.ros/cob_people_detection/files/training_data, start the program and record two people.

After training, you need to build a recognition model with the new data. To do so, just select 4 – load recognition model from the client’s menu. In the following prompt you can either list all persons that you captured and that shall be recognized by the system, e.g. by typing

Choose an option:
1 – capture face images
2 – update database labels
3 – delete database entries
>> 4 – load recognition model (necessary if new images/persons were added to
>> the database) <<
5 – activate/deactivate sensor message gateway
6 – get detections
q – Quit

Enter the labels that should occur in the recognition model. By sending an empty list, all available data will be used.
Enter label (finish by entering ‘q’): ChuckNorris
Enter label (finish by entering ‘q’): OlliKahn
Enter label (finish by entering ‘q’): q
Recognition model is loaded by the server …

A new recognition model is currently loaded or generated by the server. The following labels will be covered:
– ChuckNorris
– OlliKahn
The new recognition model has been successfully loaded.
Current State: SUCCEEDED   Message: Model successfully loaded.
or you can directly enter q and all available persons will be trained.

After all steps succeeded, you can watch the recognition results in the display. Although you may access all training data functions (update, delete, etc.) from the client directly you may also access the files directly, which are located in your home folder at ~/.ros/cob_people_detection/files/training_data.