[REVIEW] Camera Streaming custom MAVLink message

ricardotk · September 1, 2016, 2:59pm

Hello everyone

This topic is to propose the support on Ardupilot of a camera streaming custom MAVLink message.

A few weeks ago @lucasdemarchi presented on the weekly dev call the camera streaming solution that we are developing.

For those who were not able to participate, I will give a brief overview of what it is.

There’s a Python Script that will get the cameras from /dev/videoX and create RTP/RTSP streams using gstreamer, its basically a daemon that exports this information to D-BUS.
We’ll have a custom MAVLink message that will be used to send/receive information about the streams. (Ardupilot would have to get the information from D-BUS and send whenever needed as MAVLink messages)
We’ll add also some new UI on QGroundControl in order to enable changes on streams. (QGroundControl would have to get these changes from UI and send whenever needed as MAVLink messages)

This sequence diagram shows the track of a simple message.

It’s important to notice that the objective of this custom MAVLink message is to provide information about the streams, not to provide the stream data itself.

This video shows a little bit about what has being done, this part is not using the custom MAVLink message yet, everything is running local, https://drive.google.com/open?id=0B2EZFTQdeg3oR2lHQ2N0Sy1YZmc

I would like to ask you guys for a review of what you think of it as well as review the message itself.

<?xml version='1.0'?>
<mavlink>
    <include>common.xml</include>
    <enums>
        <enum name="CAMERA_STREAM_GET_CMD">
            <description>Commands to be executed in order to get stream data on "CAMERA_STREAM_GET" message "command" field.</description>
            <entry value="1" name="CAMERA_STREAM_GET_CMD_SERVER">
                <description>Command to get server attributes</description>
            </entry>
            <entry value="2" name="CAMERA_STREAM_GET_CMD_STREAMS">
                <description>Command to get available streams</description>
            </entry>
            <entry value="3" name="CAMERA_STREAM_GET_CMD_DATA">
                <description>Command to get data of a specific stream</description>
            </entry>
        </enum>
    </enums>
    <messages>
        <message id="150" name="CAMERA_STREAM_GET">
            <description>Message that get stream data using "CAMERA_STREAM_GET_CMD" commands.</description>
            <field type="uint8_t" name="target_system">System ID</field>
            <field type="uint8_t" name="id">Camera device ID</field>
            <field type="uint8_t" name="command">CAMERA_STREAM_GET_CMD enum</field>
        </message>
        <message id="151" name="CAMERA_STREAM_SERVER">
            <description>Message that can be requested by sending the "CAMERA_STREAM_GET_CMD_SERVER" command on "CAMERA_STREAM_GET" message.</description>
            <field type="char[40]" name="ip">Camera stream server ip to accept connections on the given address</field>
            <field type="uint16_t" name="port">Camera stream port which the server will accept connections</field>
        </message>
        <message id="152" name="SET_CAMERA_STREAM_SERVER">
            <description>Message that sets the camera stream server attributes.</description>
            <field type="uint8_t" name="target_system">System ID</field>
            <field type="char[40]" name="ip">Camera stream server ip to accept connections on the given address</field>
            <field type="uint16_t" name="port">Camera stream port which the server will accept connections</field>
        </message>
        <message id="153" name="CAMERA_STREAM_STREAM">
            <description>Message that returns one stream URI.</description>
            <field type="char[60]" name="uri">URI representing the stream</field>
        </message>
        <message id="154" name="CAMERA_STREAM_DATA">
            <description>Message that can be requested by sending the "CAMERA_STREAM_GET_CMD_DATA" command on "CAMERA_STREAM_GET" message.</description>
            <field type="uint8_t" name="id">Camera device ID</field>
            <field type="uint32_t" name="capabilities">Union of device capabilities flags</field>
            <field type="uint32_t" name="format">Camera video format set</field>
            <field type="uint32_t[20]" name="available_formats">Camera available video formats</field>
            <field type="uint16_t[2]" name="frame_size">Camera video frame size, array followed by width and height</field>
            <field type="char[60]" name="uri">Camera stream URI</field>
        </message>
        <message id="155" name="SET_CAMERA_STREAM_DATA">
            <description>Message that sets stream attributes.</description>
            <field type="uint8_t" name="target_system">System ID</field>
            <field type="uint8_t" name="id">Camera device ID</field>
            <field type="uint32_t" name="format">Camera video format</field>
            <field type="uint16_t[2]" name="frame_size">Camera video frame size, array followed by width and height</field>
            <field type="char[30]" name="mount_path">Camera stream mount path to build the URI</field>
        </message>
    </messages>
</mavlink>

thanks!

–
ricardotk

OXINARF · September 1, 2016, 6:17pm

This looks interesting. I think that using MAVLink messages is a good idea but I’m not too keen on the way you plan to handle it in ArduPilot. I think ArduPilot is an autopilot and shouldn’t handle things like getting a video stream URL - anything that isn’t a function of the autopilot should just be routed.

My proposal would be that these messages have a component id (besides the system id) and that the Python script is a component of the system handling these MAVLink messages (we might need a new MAV_COMP_ID for a video server). Now, you can allow ArduPilot to do the routing of the messages or you can insert a component to do the routing before it reaches ArduPilot.

shadow1runner · September 1, 2016, 6:47pm

Just a very minor thing I’ve noticed by looking at your message definitions:

Your strings are quite short (if you want the IP addresses displayed a human-friendly way)

SO says that

There’s a caveat with the general 39 character IPv6 structure. For IPv4 mapped IPv6 addresses, the string can be longer (than 39 characters). An example to show this:
IPv6 (39 characters) :
ABCD:ABCD:ABCD:ABCD:ABCD:ABCD:ABCD:ABCD
IPv4-mapped IPv6 (45 characters):
ABCD:ABCD:ABCD:ABCD:ABCD:ABCD:192.168.158.190

Same applies to:

Otherwise very interesting stuff, hope to see it soon in master, but I agree with @OXINARF that it belongs into a new component, but I’m still fairly new to ArduPilot source…

ricardotk · September 5, 2016, 6:25pm

@OXINARF yes, I see your point, but there are some downsides:
1 - We won’t have discover and update, its always "send the announce every N seconds"
2 - Video Server needs to known on which GCS network it is connected

@shadow1runner nice catch, thanks for the tip.

rmackay9 · September 6, 2016, 7:02am

This is all super exiciting.
I agree with @OXINARF about it being a new component.

Besides a target-system-id, do we need a target component-id? It’s only required if we think there’s going to be more than one component returning video streams. For example, the main linux board has a camera and then there’s some other mavlink enabled camera on the network. It seems a bit unlikely but I thought I’d mention it.

Instead of “CAMERA_STREAM” I wonder if we should call it “VIDEO_STREAM”?

I’m sure people here have seen the common mavlink messages? http://mavlink.org/messages/common
So we’ve got MAV_CMD_VIDEO_START_CAPTURE, VIDEO_STOP_CAPTURE and MAV_CMD_DO_CONTROL_VIDEO (which should have been named MAV_CMD_DO_VIDEO_CONTROL to be consistent with the others). I don’t know of anybody using these messages though. I’m just mentioning them so we’re aware they exist.

Should CAMERA_STREAM_STREAM be named VIDEO_STREAM_URI maybe?
Should CAMERA_STREAM_DATA be named VIDEO_STREAM_FORMAT or VIDEO_STREAM ATTIRIBUTE maybe?

LuisVale · September 6, 2016, 5:29pm

So, these new Mavlink messages should go to the main Mavlink, or stay on the Ardupilot XML ?

I agree with Randy on the renaming for the sake of consistency

ricardotk · September 6, 2016, 6:30pm

I like both, we can call it VIDEO_STREAM

This messages are for control (start,stop,etc), this kind of control will be done by RTSP/RTP itself.

Indeed a better name

Well, in this case I prefer VIDEO_STREAM_ATTRIBUTES because it holds attributes, not only the format itself.

OXINARF · September 6, 2016, 7:31pm

I’m not following what you mean.

Why?

lucasdemarchi · September 6, 2016, 8:01pm

once it’s finalized, to the main mavlink repository - it’s still in its infancy though

peterbarker · September 13, 2016, 6:38am

Couple of points:

I think the SET messages should also go under the ^VIDEO_STREAM naming convention.
the video formats thing looks a bit weird. Should this be an enumeration?
+1 to the target-component-id concept.
+1 to solely using ArduPilot’s mavlink routing for this.
'though I’m actually a little confused as to why you even need that; if you can take an RTSP stream, surely you could use some sort of existing Service Location Protocol to find the streams?

julienberaud · September 13, 2016, 9:11am

Hi Everyone,

I agree with @OXINARF about routing. I also like the whole rtsp idea but we also need a simpler way to stream.
Normally, we could expose the video streams and just ask the video server to start streaming on a specified port and adress via RTP/H264 and then we only need to know the size of the RTP payload.
This would allow to stream data from a lightweight GCS, I am thinking SkyController 2 which flash size is small and can’t embed gstreamer.
Maybe that’s another topic though.

Julien

MagicRuB · October 27, 2016, 4:38pm

Lots of new commands here, perhaps we should move the new VIDEO_ command indexes to >255?