IP cameras, RTSP, MavLink support

Hardware OSD/Video
1

I’ve had the opportunity to learn bit more about IP cameras.

It appears that some of the new IP camera/gimbals are using RTSP - a client-server media streaming protocol - similar to what a home “nanny cam” might use on a home network.

As a client-server implementation - RTSP commands are bi-directional… On a drone, the camera would be the server, and the drone’s air-unit would be the client.

Because the commands are bi-directional, this allows the air-unit to send gimbal control commands.

I imagine this might open the door to a new or expanded set of MavLink commands.

By copy to @rmackay9 - I was wondering if any of the gimbal enhancements presently underway have any RTSP support for IP cameras in the queue.

Thanks and regards.

2

this is just the video protocol, it isn’t linked to the control. This protocol is used since years already.
We already have one ONVIF protocol support that is on off the two majors protocol for IP camera control. We thus lack documentation on how to use it.

1 Like
3

Thank you for responding to this thread.

I believe we have different information. The references I’ve read about RTSP say that in addition to the video stream, it has bi-directional control ability.

4

AFAIK the only way to do that is with ONVIF. ONVIF tunnels the RTSP streams and does the gimbal control.

5

yes, but only on some part of the video stream and if the server implement it. But you cannot do a gimbal control or PTZ or most common task on camera with it.

alternative to ONVIF is Pelco-d , but we don’t support it.

6

@jstroup1986,

Thanks for the ping.

I’m keen to add support for controlling cameras and gimbals over ethernet but AP’s ethernet support is still a work-in-progress, expected to arrive in “latest” within a month and will go out with AP 4.5. Once that’s done I’m sure there will be a rash of development to get cameras and gimbals working with it.

Re ONVIF, we have a library but as far as I know it is not integrated with our camera or gimbal libraries and doesn’t work without Ethernet nor our ChibiOS autopilots (which is most of them).

1 Like
7

Digging into this a bit more yesterday, I realized that it’s not really the ethernet connection that’s interesting here - it RTSP.

RTSP is a protocol that might be used for a nanny cam - streaming and basic camera control such as zoom, tilt and pan.

The SIYI A8 with an ethernet connection to the HereLink Air Unit is essentially that. The camera functionality is under the control of the Apps running on the HereLink ground unit. And SIYI provides two apps - an “FPV” app that just displays the camera feed and has touch screen control of the camera and gimbal - and a custom Qgroundcontrol that adds these FPV functions to Qgroundcontrol.

Interestingly - there’s no connection to the autopilot these configurations. So a MavLink command issued to the gimbal would have no way to reach the camera/gimbal.

But on the other hand - the App running on the ground controller has a lot more functionality at it’s disposal with this “nanny cam” style configuration. The app could do image processing that results in the sort of “intelligent” photo/video features you find on DJI and other drone products.

The problem is that with the ethernet connection as it is, without autopilot connectivity there’s no way for MavLink commands from an auto-mission to reach the camera/gimbal.

The SIYI A8 does have Sbus and HDMI connections - so it can be used in a conventional way - but not using ethernet.

Since the HereLink Air-Unit has a UART connection to the AutoPilot for telemetry, there seems to at least be a path way for an autopilot instruction to reach the ethernet connected gimbal/camera connection. But that would depend entirely on the HereLink firmware. So it seems to me that to have autopilot control of an ethernet connected camera/gimbal, the air unit’s firmware would have to support such functionality.

Because of its historical association with ArduPilot, there’s a lot of ArduPilot focus on CubePilot products. But with other manufacturers such as SIYI are bringing out new controller and air unit products, things may be changing rapidly.

I’ve noticed that CUAV offers their pixhawk products with ethernet. While this might solve the gimbal and camera control connectivity for the autopilot, it leaves video streaming unaddressed. To have both streaming and gimbal/camera control on ethernet, you’d have to be able to support at least three devices on the drone’s private network.

I’m looking forward to these advances. I hope I can make a contribution.

2 Likes
8

I get the feeling that there is more than one way all this might be accomplished.

In this article, “RTSP Commands” are mentioned. RTSP: The Real-Time Streaming Protocol Explained | Wowza

9

I found this article that shed some light on the subject for me: ONVIF vs RTSP: Which is better ONVIF or RTSP?

10

@jstroup1986,

I talked about this a bit in my talk at the dev conference (sadly the sound was messed up) but I guess the short form is that we plan to improve AP support for many of the camera and gimbal controls in the coming months. The lack of these controls is probably what has led various manufacturers to try and find their own solutions including add-ons to QGC and releasing their own apps. There is also got our own open source QGC gimbal/camera control screen coming. [WIP] Improve gimbal support by julianoes · Pull Request #10667 · mavlink/qgroundcontrol · GitHub

The effort requires enhancements in multiple levels:

  • gimbal manufacturers may need to expose new controls to AP
  • AP drivers need to be enhanced to take advantage of these new controls
  • ground stations need to expose these new controls to the user

Also once AP supports ethernet we will enhance the AP drivers so they can send serial or mavlink over ethernet. I don’t know enough about how/if we could support RTSP because AP itself doesn’t directly access the video feed.

11

Thank you Randy for the briefing on all this. I really appreciate it. If there’s anyway I can help out with testing or feedback, I’ll do my best to contribute.

Embarrassingly, it dawned on me that the A8 with an ethernet connection to the HereLink Air Unit doesn’t have a link to the autopilot. So the original point of this post is ridiculous.

Interestingly, the SIYI FPV app for the A8 has an option for OSD, and includes an option for MavLink input. I’ll look into implementing this later this week.

I thought there were new gimbal “divers” in ArduPilot for the SIYI camera gimbals like the A8. For serial port protocols, AlexMos and Storm32 are still the only two gimbal protocols listed - even on the DEV release of parameters. The only reference to SIYI I could find is on the “MNT” parameters. I recall that gimbals are thought of as “mounts” in ArduPilot - but I don’t understand the connectivity and configuration.

There’s a cascade of related things to this thread - but this isn’t the place for it. I hope to find a good avenue to participate.

12

@jstroup1986,

In the future, if using a Herelink, I expect that both the autopilot and the camera/gimbal will be connected to the Ethernet port of the herelink receiver.

By the way, there are new drivers for the Siyi cameras in 4.3 and 4.4. The setup instructions are here but I see there is a recent change which makes the camera setup incorrect:

  • If using 4.3 CAM_TRIGG_TYPE should be set to 3 (Mount)
  • if using 4.4 CAM1_TYPE should be set to 4 (Mount)
13

Randy -

I’m working through this today.

The “trim” for the RC channel needs to be set to the value for neutral position for each axis being commanded. I’m only doing pitch - so just one for me.

Might be worth mentioning in the wiki.

Cheers - Joe

trim
trim1485×316 11.3 KB

1 Like
14

Randy -

One more item to consider.

The SIYI apps on the controller have an on-screen “re-center” button - that will reposition the gimbal to it’s neutral position. This would be good thing to have available.

Thanks - Joe

15

@jstroup1986,

Thanks for the review and feedback. I will update the Gimbal Mount Controls wiki page to add a comment about the RC ranges.

Yes, a re-center button is a good idea. The ground stations can implement this (MP’s Data screen’s Payload Control tab has this for example) but I guess you’re suggesting we have an auxiliary switch for this? … and I guess you’d use this when the gimbal is being controlled in RC targetting mode? e.g. The pilot is controlling the rate at which the gimbal’s pitch or yaw is being changed using the transmitter?

16

Several things to respond here:

  1. As I understand it, there are two ways to issue instructions to the autopilot to perform special tasks:

a) Through RC channel PWM via the SERVO(n)_FUNCTION parameters - which allows a RC transmitter switch to issue the command, and

b) MavLink commands that a GCS can issue - or may be imbedded in an auto mission. Commands such as “center the gimbal” probably should be available via both methods. I understand that creation of new MavLink commands is non-trivial - and may not allow this duplicity in all cases. But it’s a good objective.

  1. I don’t know if the SIYI A8’s firmware contains a proprietary “center the gimbal” function in it’s protocol. The SIYI FPV app has this function - but it may be proprietary. In any case, it only works when using the ethernet connection to the air unit. I don’t know about the underlying control protocol of the SIYI camera/gimbal via this ethernet connection - but as the DEV’s get their hands around ethernet, it would be good to have some documentation about the differences so that developers can have the facts to consider for their projects.

  2. In the present non-ethernet implementation, where the Storm32 Serial driver is being used, it may be possible/preferable to do a “center the gimbal” function by commanding the gimbal to go to the “trim” values for each axis.

  3. I can foresee at least one reason to support at least a MavLink command for “RC Targeting” where a the autopilot relinquishes ROI targeting. More experienced pilots may have many more such reasons. But consider this - suppose you want to plan a mission that does video inspections of multiple locations - each some distance apart. For each location the mission might set the ROI of a location and then circle around it. But on the next leg travelling to the next location, the gimbal might best be reverted to “RC Targeting” so that the pilot can have an FPV view as the drone moves to the next location. Subsequently, after circling a ROI for the last location, having pilot control of the gimbal on the last leg would be desirable.

Having tested both Alexmos and Storm32 gimbals, I have a basic understanding of gimbal control with ArduPilot. But my actual flight experience with them is limited - primarily just using the MavLink command to point a camera down for NADIR photography after take off and back to normal before RTL on photo survey mission. I expect the more experienced gimbal users know of many issues and considerations I’ve not experienced or considered.

Two more issues/ideas to pass along:

a) As I mentioned in a separate thread, it appears that the Storm32 Serial protocol has a bug. When using this protocol the “RC Targeting” option does not appear on the drop down selection in Mission Planner. When testing Alexmos and Storm32 gimbals with MavLink, this options was available.

reset
reset704×703 158 KB

b) The SIYI A8 was obviously designed by SIYI to be well integrated with their control radios such as the MK15 and MK32 - which have their customized GCS apps. I’ve not seen documentation on how to enable the image tracking functions that the A8 is marketed to have, but I expect it relies on the GCS and not the autopilot. I think we’re likely to see more of this sort of thing instead of less. And since Mission Planner is “tightly coupled” with ArduPilot - it may be time to encourage @Michael_Oborne to think about how Mission Planner can be updated to work well as an Android app used by common GCS, and have such advanced features supported by good coordination between the controller and the autopilot. (By “work well” I primarily mean a user interface appropriate to an Android size touch screen.

I’ve only operated one DJI product - my Mini 3 Pro - which I bought only because I thought I needed to have an understanding of their technology. As DJI and other “pre-build” drones become more capable, ArduPilot may find itself used an increasingly narrow usage segments without better controller and autopilot integration.

Thank you.

1 Like
17

I had to go back and search my previous posts - but about two years ago I got advice from @Yuri_Rage that the easiest/best way to clear an ROI with MavLink is to simply set the Lat/Lon/Alt values to all zeros.

I don’t know if an RC PWM value via the SERVO(n)_FUNCTION parameters would be of value to perform this - but might be a good back-burner item.

The “center the gimble” function would definitely be helpful.

BTW - Thanks Yuri - I know I was a PITA about this - wanting to use the MavLink command/option (I forget which) that clears the ROI. But your suggestion works well. Its simply one of those zillion of things that probably can never be included in the wiki - there’s just too many of those things.

18

@jstroup1986,

Thanks for the feedback.

Re the Action’s tab “Set Mount” drop-down not working, this really should not depend upon the type of gimbal being used so I suspect that you may be trying to directly connect to the gimbal when really the pilot should nearly always be talking to the autopilot which in turn talks to the gimbal. I suspect what’s happened is the drop-down to the left of the Connect/Disconnect button has somehow changed to the gimbal instead of the vehicle.

By the way, I think it’s a good idea to set the SERIALx_OPTIONS = 1024 to stop MAVLink messages from the gimbal from reaching the GCS. Direct GCS<->gimbal communication shouldn’t be necessary and it eats up a lot of telemetry bandwidth.

19

Thanks for considering my observations.

I’m familiar with the drop-down to the left of the “Connect/Disconnect” button. As you can see, I have it set correctly - the “quadrotor” option.

set mount not showing
set mount not showing1920×1080 218 KB

I thought the drop down next to the “Connect/Disconnect” button was there to display and select different MavLink devices available to Mission Planner. As I’m using the “Storm32 Serial” protocol for the gimble, and the gimble/camera shows up as an additional item on this drop down, it appears my understanding is not quite correct.

com3 42 camera

Can you please refer me to where this drop down is explained so I can learn about it?

Lastly - I’m working now on a “hybrid”: implementation - where both the ethernet connection and the serial/gimbal connections are active.

I’m curious to find if I can use the ethernet video feed, and control the gimbal from the autopilot. So far, this seems to be working on the work-bench.

But I’m concerned because this implementation as two command pathways to the gimble. 1) The app’s commands via the ethernet connection to the Air-Unit, and 2) the Autopilot command via the serial Storm32 connection.

I’m wondering in what way they may conflict. So far, with both enabled the autopilot commands via the serial port drive seem to override any of the SIYI app commands that move the gimble.

There is one exception - the “return to center” command works even when the autopilot commands are enabled. This creates some curious effects:

  1. using the HereLink controller’s control wheel to pitch the gimble up or down, the “return to center” command on the SIYI apps work properly. The gimble returns to center and stays there until it responds the next command.

  2. using the “point camera here” command to cause the gimbal to position the camera to a 3-D point in space, the “return to center” command works - but only momentarily. After retuning to center, the gimble goes back to the “point camera here” position.

I can’t know for sure, but this suggests to me that the “return to center” is a SIYI proprietary gimble control command.

I hope any of this is useful. thanks

20

@jstroup1986,

I think your understanding of the top-right drop-down is correct. So I’m not sure what’s going on with MP’s Actions tab’s “Set Mount” drop-down being blank. I think this means that MP is receiving a message saying that the gimbal is in a mode that it doesn’t know about (e.g. it’s not Retracted, Neutral, MAVLink Targetting, RC Targetting, or GPS Point). My assumption was that “COM3-42-CAMERA” was selected on the top right and so the bad mode reporting was coming from the SToRM32 gimbal itself. I guess this isn’t the case so I’m not sure what’s going on. Maybe the MNT1_DEFLT_MODE parameter has been set to an invalid value?

I think the “hybrid” connection to the gimbal is OK but it’s really up to how the camera/gimbal handle receiving commands both from the serial port and the ethernet port.

By the way, I’m a little confused as to whether a SToRM32 gimbal or a Siyi camera/gimbal is being used. If a Siyi is being used it is best to stick with the Siyi setup described on the wiki.

EDIT: ah wait… I think you’re referring to the SERIALx_PROTOCOL = “SToRM32 serial gimbal”. I’ll change this parameter description to be just “Serial Gimbal” because we now have multiple serial gimbals that require this setting. Txs for the info.