按年份存档： 2016

一、GStreamer
二、gst-launch
三、在linux环境使用ip-camera
四、在ros环境使用ip-camera

• 一、GStreamer

GStreamer是个建立音视频管道的工具包，例如stream一个媒体文件到互联网，再例如把一个test.avi文件stream到一个V4L的camera。V4L的capture主要是用：
1–gst-launch-1.0 for capturing video
2–FFMpeg for saving and editing video
3–v4l-ctl for controlling your video card
4–mpv for viewing videos
5–gst-inspect for listing elements

• 二、gst-launch

操作GStreamer主要用gst-launch这个CLI，这个命令行工具可以创建pipeline、初始化、然后运行，例如可以把一个avi媒体文件模拟成摄像头，把ip-camera回环成一个usb-camera，等等。
1–例如最简单的：
$ gst-launch-1.0 fakesrc ! fakesink
这样建立了一条simplest pipline，实现connects a single (fake) source to a single (fake) sink。
2–例如记录视频，create an AVI file with raw video and no audio：
$ gst-launch-1.0 v4l2src device=$VIDEO_DEVICE ! $VIDEO_CAPABILITIES ! avimux ! filesink location=test.avi
3–例如记录音频，create an AVI file with raw audio and no video：
$ gst-launch-1.0 $alsasrc device=$AUDIO_DEVICE ! $AUDIO_CAPABILITIES ! avimux ! filesink location=test.avi
4–记录音频和视频：
$ gst-launch-1.0 v4l2src device=$VIDEO_DEVICE ! $VIDEO_CAPABILITIES ! mux. alsasrc device=$AUDIO_DEVICE ! $AUDIO_CAPABILITIES ! mux. avimux name=mux ! filesink location=test-$.avi
This pipe has three parts:a video source leading to a named element (! name. with a full stop means “pipe to the name element”) an audio source leading to the same element a named muxer element leading to a file sink
5–把一个source分为多个out：
$ gst-launch-1.0 v4l2src device=$VIDEO_DEVICE ! $VIDEO_CAPABILITIES ! avimux ! tee name=network ! filesink location=test.avi tcpclientsink host=127.0.0.1 port=5678
This sends your stream to a file (filesink) and out over the network (tcpclientsink). To make this work, you’ll need another program listening on the specified port (e.g. nc -l 127.0.0.1 -p 5678).
6–从相机查看图片：
$ gst-launch-0.10 v4l2src do-timestamp=true device=$VIDEO_DEVICE ! video/x-raw-yuv,format=(fourcc)UYVY,width=320,height=240 ! ffmpegcolorspace ! autovideosink
7–其它选项
gst-launch -v –gst-debug-level=3，输出的调试信息会多一些。
IP camera, gstreamer and virtual video devices，指明画幅参数。

• 三、linux环境使用IP相机

1、安装
1.1、包
$ sudo apt-get install v4l2loopback-dkms
1.2、源码
如果失败apt-get method may fail durning install. some wranings were treated as errors. Not sure if it’s a compiler compatibility issue.
$ sudo su
$ git clone https://github.com/umlaeute/v4l2loopback.git
$ cd v4l2loopback
$ make && make install
（是否需要sudo忘记了）
2、加载模块
2.1、加载
$ sudo modprobe v4l2loopback
2.2、如果提示not found
$ sudo depmod v4l2loopback
$ sudo modprobe v4l2loopback
2.3、如果还失败
$ sudo apt-get install linux-generic<——–maybe freeze you kernel
$ sudo apt-get install v4l2loopback-dkms
$ sudo depmod v4l2loopback
$ sudo modprobe v4l2loopback<——————
or
$ sudo modprobe v4l2loopback video_nr=7 3、验证
$ v4l2-ctl –list-devices
会出来虚拟设备，这里是video1，我原有的usb-camera是video0是。
Dummy video device (0x0000) (platform:v4l2loopback-000):
/dev/video*
同样也可以
$ dmesg v4l2loopback | grep v4l2
如果出现libv4l2: error getting pixformat: Invalid argument
or:
v4l2loopback: module verification failed: signature and/or required key missing – tainting kernel
美观它。

4、建立媒体源
the simpest is:
$ gst-launch v4l2src device=/dev/video0 ! v4l2sink device=/dev/video1
4.1、接收流媒体是：
$ gst-launch-0.10 udpsrc port=1234 ! theoradec ! ffmpegcolorspace ! ximagesink
发送流媒体是：
$ gst-launch-0.10 v4l2src ! ffmpegcolorspace ! theoraenc ! udpsink host=127.0.0.1 port=1234
注意先执行接收的。
4.2、显示或保存
显示视频：
$ gst-launch v4l2src ! xvimagesink
视频保存成文件：
$ gst-launch v4l2src ! video/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! jpegenc ! avimux ! filesink location=osug-1.avi
4.3、某段音频伪装成相机video1：
$ gst-launch videotestsrc ! v4l2sink device=/dev/video1
查看下，ok：
$ cheese –device=/dev/video1
相机0伪装成相机1：
$ gst-launch v4l2src device=/dev/video0 ! v4l2sink device=/dev/video1
查看下，ok：
$ cheese –device=/dev/video1
文件伪装成相机1：
$ gst-launch filesrc location=”/home/zdh991/tmp/osug-1.avi” ! avidemux ! v4l2sink device=/dev/video1
查看下，ok：
$ cheese –device=/dev/video1
IP相机伪装成相机1：
$ gst-launch souphttpsrc location=http://192.168.1.126:80 ! jpegdec ! ffmpegcolorspace ! v4l2sink device=/dev/video1
查看下，ok：
$ cheese –device=/dev/video1

5、利用iphone模拟测试
IP camera, gstreamer, and virtual video devices.
如果没有IP-camera，可以使用iphone的摄像头。
安装一个app例如third eye，让iphone接入局域网分配。
在iphone获取到ip地址后，从pc上测试通过浏览器输入iphone-ipaddress：8091，如果可以实时显示iphone摄像头的视频，说明okay。

利用gat-launch操作iphone的这个ip：port把这个摄像头作为wifi的ip-camera来使用：
$ gst-launch souphttpsrc location=”http://172.20.10.11:8091″ ! jpegdec ! ffmpegcolorspace ! v4l2sink device=/dev/video1
这里面172.20.10.11:8091就是IP-camera的地址和端口。注意pc和iphone在同一subnet。

• 四、ROS环境使用IP-camera

ros提供gscam这个node，可以用image view查看，例如：
$ gst-launch souphttpsrc location=http://[user]:[password]@[camera_ip]/mjpg/video.mjpg ! jpegdec ! v4l2sink device=/dev/video0
$ GSCAM_CONFIG=”rtspsrc location=rtsp://CameraIP/ipcam.sdp ! video/x-raw-rgb,framerate=30/1 ! ffmpegcolorspace”
$ rosrun image_view image_view image:=/gscam/image_raw
使用gscam也可以配合rosbridge的websocket，例如wss://ip:port。

• A、gst-launch备忘

这个工具只能建立简单地pipeline。尤其是它只在特定层级之上模拟pipeline和应用的交互。可以很简单的快速测试pipeline。注意gst-launch主要是一个调试工具，真正用gst_parse_launch()这个API来创建pipeline。

gst-launch的命令行包括一个在PIPELINE-DESCRIPTION之后的一系列选项。简单说，一个PIPELINE-DESCRPTION是一系列用！分隔开的元素：
$ gst-launch-0.10 videotestsrc ! ffmpegcolorspace ! autovideosink
这用videotestsrc，ffmpegcolorspace和autovideosink三个element。GStreamer会把他们的输出pad和输入pad连接起来，如果存在超过1个可用的输入/输出pad，那么就用pad的Caps来确定兼容的pad。

element
element可能是有属性的，在命令行里格式就是“属性=值”，多个属性用空格来分开。可以用gst-inspect工具来查一下element的属性。
$ gst-launch-0.10 videotestsrc pattern=11 ! ffmpegcolorspace ! autovideosink
element可以用name这个属性来设置名称，这样一些复杂的包含分支的pipeline可以创建了。有了名字，就可以使用前面创建的element，这在使用有多个pad的element（比如demuxer或者tee等）时是必不可少的。
$ gst-launch-0.10 videotestsrc ! ffmpegcolorspace ! tee name=t ! queue ! autovideosink t. ! queue ! autovideosink
这把videotestsrc先连接了ffmpegcolorspace，然后连接了tee element 这个tee就被命名成‘t’,然后一路输出到queue以及autovideosink，另一路输出到另一个queue和autovideosink。

Pads
在连接两个element时与其让GStreamer来选择哪个Pad，宁可直接指定Pad。可以在命名element后使用.+pad名字的方法来做到这点（element必须先命名）。同样可以用gst-inspect来查看element里面pad的名字。
$ gst-launch-0.10.exe souphttpsrc location=http://docs.gstreamer.com/media/sintel_trailer-480p.webm ! matroskademux name=d d.video_00 ! matroskamux ! filesink location=sintel_video.mkv
这使用souphttpsrc在internet上锁定了一个媒体文件，这个文件是webm格式的。可以用matroskademux来打开这个文件，因为媒体包含音频和视频，所以创建了两个输出Pad，名字分别是video_00和audio_00。把video_00和matroskamux element连接起来，把视频流重新打包，最后连接到filesink，这样就把流存到了一个名叫intel_video.mkv的文件。总之找了一个webm文件，去掉了声音，仅把视频拿出来存成了一个新文件。如果保持声音，那么就应该这样：
$ gst-launch-0.10.exe souphttpsrc location=http://docs.gstreamer.com/media/sintel_trailer-480p.webm ! matroskademux name=d d.audio_00 ! vorbisparse ! matroskamux ! filesink location=sintel_audio.mka
这里的vorbisparse element会从流里面取出一些信息，然后放到Pad的Caps里面，这样下一个element，也就是matroskamux就可以知道如何处理这个流了。这个处理在抓取视频的时候是不用做的，因为matroskademux已经做了这件事情。
注意上面两个例子中媒体没有被解码和播放,仅仅只是把数据搬动了一下而已。

Caps过滤
当一个element有不止一个pad时，连接下一个element可能是模糊不清的，因为下游的element可能有不止一个的兼容的输入pad，或者它的输入pad可以和所有的输出pad兼容。在这样的情况下，GStreamer会使用第一个可以连接的Pad，这样相当于说GStreamer是随机找一个pad来连接的。
$ gst-launch-0.10 souphttpsrc location=http://docs.gstreamer.com/media/sintel_trailer-480p.webm ! matroskademux ! filesink location=test
这里和上一个例用了同样的媒体文件和demuxer。finksink输入pad是任意格式，这意味着它可以接受所有的媒体格式。那么matroskademux的哪个pad可以接到filesink呢？video_00还是audio_00？无法知道。为了消除这种不确定性，前面例子中用了pad的名字的方法，这里使用Caps过滤的方法：
$ gst-launch-0.10 souphttpsrc location=http://docs.gstreamer.com/media/sintel_trailer-480p.webm ! matroskademux ! video/x-vp8 ! matroskamux ! filesink location=sintel_video.mkv
一个Caps过滤动作类似于让element不做任何动作，仅仅接受给出的Caps。在这个例子中，在matroskademux和matroskamux中间加入了一个ievideo/x-vp8的Caps过滤，这样就表明在matroskademux中我们仅仅需要能生成这种类型视频的输出Pad。
需要用gst-inspect工具来查看一个element能接受和生成的Caps，用gst-discoverer来查看文件里面包含的Caps。如果需要查看在pipeline里面一个element生成的Caps，在gst-launch里面使用-v参数即可。
一个调整视频比例的pipeline。videoscale element可以调整输入尺寸然后再输出。例子里面用Caps过滤设置了视频大小为320×200：
$ gst-launch-0.10 uridecodebin uri=http://docs.gstreamer.com/media/sintel_trailer-480p.webm ! queue ! videoscale ! video/x-raw-yuv,width=320,height=200 ! ffmpegcolorspace ! autovideosink

1. setup robot
3. od_node
5. laser_node
7. imu_node

1. about imu sensor
2. anout imu usage
3. about imu code
3.1 odom-node.cpp的
3.2 imu-node.cpp的
4. about imu debug
REF：
https://answers.ros.org/question/200480/imu-message-definition/
https://github.com/jeskesen/i2c_imu/blob/master/src/i2c_imu_node.cpp
http://codegist.net/code/flora-9dof-accelerometer%2Bgyroscope%2Bmagnetometer—lsm9ds0/

（如果运行在Oracle 的VM VirtualBox ， 需要install extend tools，使能 networks interface 桥接， 使能 com为主机设备 主机设备com1，然后install ubuntu 14.04 AMD64, ros ,setup env, init catkin_ws, start master ）

1. setup robot
$ roslaunch funbot r_rbt.launch

.. …

这launch加载: odom的串口驱动，laser的以太网驱动，imu的USB驱动，
同时静态发布一个base_link到base_laser的tf变换。
同时静态发布一个base_link到imu_link的tf变换。
后面这项工作如果已经创建了机器人模型urdf，就不要再发布了。

3. od_node

5. laser_node

7. imu_node

1. about imu sensor

ros官方和开发者都存在的很大问题， 是imu和odom的消息格式。
odom的消息nav_msgs/Odometry， 指明了两个框架，tf就没有疑问。
imu的消息sensor_msgs/Imu，却只有一个坐标框架，就含混不清：
Q:
I am confused about the IMU though: it’s heading estimate should be in the map frame as it’s absolute, but it’s motion estimates should be in the base_link frame because they are relative to the robot ? The imu message only has a single frame_id, so which should I set it to?
Q:
It seems clear that nav_msgs::Odometry.pose.pose.orientation goes from header.frame_id to child_frame_id (or parent to child), but there is no child_frame_id in the sensor_msgs::Imu message and I do not see that . The two obvious choices are map and odom, but I don’t know if one is preferred over the other.

关键在于，要理解同样存在两个frame_id就可以：
传感器的线加速度和角速度， 是在传感器坐标框架内，例如imu_link;
传感器的转向旋转， 却是传感器相对于他的上层父坐标系，例如odom、map的那世界坐标系。

例如，写imu驱动程序时，定义imu_frame_id和tf_parent_frame_id及tf_frame_id这样三个框架：

那么，典型的：
**1. imu_frame_id would be something like imu_link, the acceleration and velocities are measured in this frame.
**2. the orientation is between a fixed inertial frame to sensor, the “fixed inertial frame” is implicitly assumed to be oriented in the same way as odom, map, or whatever top-level other fixed frame your application uses.
**3.1. orientation is calculated by user’ code using data both the accelerometer (pitch and roll) and magnetometer (yaw) and others …
**3.2. packages like robot_pose_ekf do not use the orientation vector in an absolute style ,it actually calculates changes in heading.

So:
You should populate the frame_id of the sensor_msgs/IMU message with the frame in which the sensor is mounted.
If IMU is oriented to base link, you can just fill the frame with base link, or publish a static tf publisher from base_link to imu_link.

2. about imu usage

方式1，需要-1来禁用传感器的转向数据。方式2，安全做法，像turtlebot这样的机器人，以及robot_pose_ekf这样的算法，对于imu的转向旋转，都是只作为相对值，微分，并不绝对值，积分。方式3，这种常见方式即使利用了电子罗盘，由于不能满足严格垂直这样的苛刻场景，需要解决。

*1. Orientation not provided
This is just gyros and accelerometers, no fusion yet.
In this case, ‘yaw’ or ‘heading’ still drifts since the gravity component for yaw is 0,
so you cannot determine absolute heading with only those 6 measurements.
We can set msg.orientation_covariance[0] = -1,
which tell other software not use the orientation.
For example, turtlebot and robot_pose_ekf all use the IMU only for the orientation changence, not the absolute orientation.
This is ultimately safest, as any orientation can be fused in this manner.
Unfortunately we lose the ability to remove drift, so more complicated approaches exist.

*2. Orientation integrated with drift from where the IMU started. This IMU doesn’t guarantee a known reference position. It’s possible for the ROS software to startup after the IMU calibrated, so the reference position is completely unknown due to drift. These IMUs are uncommon, but essentially only integrate the gyros for orientation and do not use a gravity reference. This a purely inertial output and the best you could do in empty space. The issue using this type in ROS is that the message itself does not let you know that the orientation is relative rather than absolute.
This is the case that ROS handles today and is a form of dead reckoning/Odometry. Turtlebot and robot_pose_ekf all use the IMU only for the change in orientation, not the absolute orientation. This is ultimately safest, as any orientation can be fused in this manner. Unfortunately you lose the ability to remove drift, so more complicated approaches exist.

*3. Orientation integrated with drift but pitch and roll referenced to gravity. This IMU is the most common. It uses the gyros to determine angle from startup, but references pitch and roll to gravity. The assumption here is that you will have to re-reference before you travel far enough for the Earth’s gravity vector to not aligned down in your Cartesian frame. In this case, ‘yaw’ or ‘heading’ still drifts since the gravity component for yaw is 0, so you cannot determine absolute heading with only those 6 measurements. Where this becomes a problem depends on the specific application.

*4. Orientation reference to gravity and magnetic north
This is very common that add magnetometers to reference the heading to magnetic north.
The problem using this method is that the frame must still be tangential to the Earth’s surface,
and that the magnetic field has ‘declination’ that varies based on where the IMU is located on the Earth’s surface.

*5. Orientation fused from a more complicated system (GPS INS).
This can be output a full sensor fusion system, but we highly recommend to NOT use this method.
If there is GPS, Odometry, INS, and more being fused into a globally corrected x, y, z, and orientation,
the output should primarily be a nav_msgs/Odometry.
However, if acceleration is required, the IMU message can be output, but we’d recommend it be the ‘raw’ sensor values.

What can we do to clarify this?
** Add an orientation_reference_frame_id. This is not a good idea since this is unknown and undefined for (1), (2), and (3). These don’t specify a reference that is repeatable. In the cases where you could provide it (4), (5), it still does not really solve the problem since the gravity and magnetometer reference frames are not Cartesian. So in order to accommodate ALL use cases, we’d have to reference something like this for gravity, but also factor in the magnetic field. Oh, and did I mention this varies over time?
** Let the sensor fusion stage handle it. This is ultimately the best solution. Since the sensor fusion stage has to handle custom orientation parameters, you might as well also define an imu_type parameter that can be changed to determine how best to use the IMU.

3. about imu code

3.1 odom-node.cpp的实现中，pose是在odom坐标，twist是在base_link坐标，tf是odom->base_link。

geometry_msgs::TransformStamped odom_trans;
double angle_radi =od->position.positionr*3.1415926/180.0;
geometry_msgs::Quaternion odom_quat = tf::createQuaternionMsgFromYaw(angle_radi);

odom_trans.header.stamp = current_time;
odom_trans.header.frame_id = “odom”;
odom_trans.child_frame_id = “base_link”;
odom_trans.transform.translation.x = od->position.positionx;
odom_trans.transform.translation.y = od->position.positiony;
odom_trans.transform.translation.z = 0;
odom_trans.transform.rotation = odom_quat;
//send the transform
if (tf_broadcast)
odom_broadcaster->sendTransform(odom_trans);
—————————————————-
nav_msgs::Odometry odom;
odom.header.stamp = current_time;
odom.header.frame_id = “odom”;
//set the position
odom.pose.pose.position.x = od->position.positionx;
odom.pose.pose.position.y = od->position.positiony;
odom.pose.pose.position.z = 0.0;
odom.pose.pose.orientation = odom_quat;
//set the velocity
odom.child_frame_id = “base_link”;
odom.twist.twist.linear.x = od->speed.speedx;
odom.twist.twist.linear.y = od->speed.speedy;
odom.twist.twist.angular.z = od->speed.speedr;
//publish the message
odom_pub->publish(odom);

3.2 imu-node.cpp的实现中，angular_velocity和linear_acceleration是在imu_link坐标，oritation应该是在odom或map坐标，tf是odom/map->base_link。
tf::Quaternion orientation(xf, yf, zf, wf);
delta_rotation = matrix_orientation.inverse() * orientation;
imu.header.stamp = measurement_time;
imu.header.frame_id = frame_id; // imu_link
quaternionTFToMsg(delta_rotation, imu.orientation);
imu.angular_velocity.x = gxf;
imu.angular_velocity.y = gyf;
imu.angular_velocity.z = gzf;
imu.linear_acceleration.x = axf;
imu.linear_acceleration.y = ayf;
imu.linear_acceleration.z = azf;
// publish imu message
imu_pub.publish(imu);
————————————-
// publish tf transform
if (tf_broadcast) {
transform.setOrigin(tf::Vector3(0,0,0));
transform.setRotation(delta_rotation);
tf_br.sendTransform(tf::StampedTransform(transform, measurement_time, tf_parent_frame_id, tf_frame_id));
}

4. about imu debug

4.1 ERROR: Covariance specified for measurement on topic xxx is zero
Each measurement that is processed by the robot pose ekf needs to have a covariance associated with it. The diagonal elements of the covariance matrix can not be zero. This error is shown when one of the diagonal elements is zero. Messages with an invalid covariance will not be used to update the filter.
Creating the covariance matrices for individual sensors:
For IMU and Odometry, the covariance matrix can be formed from the datasheet.
For Visual Odometry, covariance matrix may be obtained from the measurement equation that relates the measured variables to the pose coordinates [1].
*none of the covariance matrices should be zero, they should have some value, but I am not sure how it gets passed.

4.2
the odometry covariance is a float[36] a 6×6 matrix. diagonal terms are the trust you have in your sensor for each Dof. You have 6 Dof, position (x, y, z) and orientation (x, y, z)

to estimate your sensor or algorithm accuracy with experiment. If you see your data are good for 1cm in translation and 0.1 radian in rotation you can use this matrix:

[0.01 0.0 0.0 0.0 0.0 0.0,
0.0 0.01 0.0 0.0 0.0 0.0,
0.0 0.0 0.01 0.0 0.0 0.0,
0.0 0.0 0.0 0.1 0.0 0.0,
0.0 0.0 0.0 0.0 0.1 0.0,
0.0 0.0 0.0 0.0 0.0 0.1]

If have no information for one Dof you can put a huge value.

for example segway_rmp:
this->odom_msg.pose.covariance[0] = 0.00001;
this->odom_msg.pose.covariance[7] = 0.00001;
this->odom_msg.pose.covariance[14] = 1000000000000.0;
this->odom_msg.pose.covariance[21] = 1000000000000.0;
this->odom_msg.pose.covariance[28] = 1000000000000.0;
this->odom_msg.pose.covariance[35] = 0.001;