GSoC 2023: Gimbal and Camera Enhancements

Hello, I am Asif Khan. A final year undergraduate student from Jaipur, India. I am extremely delighted to share that I have been selected as a contributor in GSoC, 2023 in Ardupilot with @rmackay9 and @peterbarker as my mentors. I will be working on Gimbal enhancements project this year. In this blog i will be discussing my project in detail.

Overall Required Work
While AP’s camera and gimbal support have been rapidly advancing, there are still substantial areas that require improvement. A list of those improvements is listed here camera enhancements and gimbal enhancements . Here are the gaps that needs to be filled out:

  1. Making Usability better and easier

    • RC-only control is difficult because there are so many camera and gimbal features. Here’s a list of the 8 basic features many pilots need. This is more than many transmitters can support.

      • gimbal pitch and yaw control (rate or angle)
      • change gimbal mode (RC targetting vs MAVLink targetting)
      • switch between RC targetting “lock” (aka earth-frame yaw) vs “follow” mode (aka body-frame yaw)
      • buttons for camera zoom, autofocus, take picture, start/stop video
      • Our current ground stations (e.g. MP, QGC) controls are spread out across different screens. For example MP has:
        • Data / Actions tab controls gimbal mode
        • Data / Payload tab controls gimbal pitch and yaw
        • Data / Aux function tab can be configured to control the other features (lock/follow, camera zoom, etc)
    • Our GCSs are simply missing many important features including

  2. Support more gimbals and more gimbal features

    • We currently support the DJI RS2/RS3Pro, Gremsy, Siyi A8/ZR10/ZR30, ViewPro but not all to the same level (see below image, thanks to @rmackay9 for this)
      Other gimbals/cameras we should support (or support better) are listed below.
      • ViewPro ← should send autopilot location so can be embedded in camera using EXIF
      • Xacti DroneCAN camera/gimbal (WIP)
      • PhaseOne camera
      • WorksWell thermal cameras
      • OTUS DST gimbal
      • NextVision
  3. Ethernet support.

    • Many users’s autopilots are low on serial ports. We can simply connection by using Ethernet connection to the camera/gimbal
    • ensure all our serial drivers work over Ethernet
  4. Testing environment

    • We need a good testing environment to ease development and testing of the above features, This will also help with the development of advanced features (see below). It will be developed on the latest gazebo simulator (re-branded from ignition to gazebo).
  5. Advanced stuff. This may be outside the scope of this project but it is important to prepare for the next steps

    • once we realise cameras are a new sensor we can potentially use them for all kinds of things including:
      • precision landing. Many cameras have built in AI based object identification and tracking. We could use this instead of a special purpose camera thus saving on weight.
      • optical flow?
      • closed loop VO? e.g. compare two pictures to estimate vehicle’s current position
      • visual follow-me?
      • object avoidance (using AI object identification)

Proposed Work
Adding New message Support that include:

    Gives Information about a gimbal. Like vendor, angle limits, name ,cap flags etc.

    Tells current status of gimbal manager. Broadcaster at some low rate.

    Sets gimbal attitude. (Angles and Rate)

    Information about a camera like focal length, vendor, resolution, sensor_size, cam definition uri etc.

    Tells Image and video status, recording time, capacity, image_count etc.

    Settings of a camera inlcluding zoom level, focus level etc.

    Information about the field of view of a camera

    Gives information about a captured image. Its very important to have information like lat, long, alt, mount angles etc for doing some calculations on image to extract additional information of objects or points from an image.

    Information about a video like bitrate, framerate, resolution, streamcount etc.

    Information about the status of a video stream. Mostly similar to previous one.

    This is sent while in active tracking.

    Gives information of Geographical data of object including lat, long, alt, velocity, distance etc.

Adding new commands support

    It starts image capture sequence. Interval between two consecutive captures and number of images to capture can also be provided.

    It stops image capture sequence.

    It starts video capture. Stream Id can be provided where providing 0 means all streams. Status frequency parameter tell the rate of CAMERA_CAPTURE_STATUS message sent continuously.

    It resets all camera settings to Factory Default

    It sets camera running mode.

    It enable or disables on-board camera triggering system.

    Starts Video Streaming

    Stops Video streaming.

New features

  • Support for mount following a vehicle with provided sysid. (Completed)
  • Support for pointing gimbal at Circle-centre. (Completed)
  • Focus position support.
  • Zoom step support.
  • log corresponding gimbal mount angles in CAM messages #23816 (WIP)
  • AP_SCRIPTING: add bindings for AP_CAMERA (Completed)
  • Newer SiYI Gimbal driver that uses angle control instead of rate control as the old one, the current driver uses a P control that will not be required. (WIP)
  • Object Tracking
  • Helping QGC developer to ensure AP provides all info required for new camera gimbal control screen.

Some of the Above listed tasks has been already completed and merged into master. Cause this is a need based project, many of the new feature may be introduced later based on the suggestions and requirements of the community and users, So the end goal of this project is to upgrade the system and fix major issues with gimbal and cameras and making their support more better, robust and easier to use.

Hardware Sponsors
I am very thankful to CUAV for their continuous support and sponsoring the hardware for this project. (2 CUAV v6x, 1 Siyi Zr10 and 1 GPS)

I am very thankful to all the community members who have helped me so far, and I would like to receive more suggestions, And if you want something to add let me know by providing your valuable feedback.