GSoC 2023: Gimbal and Camera Enhancements

Blog
, , , ,
1

Thumbnail_1683301474350
Thumbnail_16833014743501280×720 91.2 KB

Introduction
Hello, I am Asif Khan. A final year undergraduate student from Jaipur, India. I am extremely delighted to share that I have been selected as a contributor in GSoC, 2023 in Ardupilot with @rmackay9 and @peterbarker as my mentors. I will be working on Gimbal enhancements project this year. In this blog i will be discussing my project in detail.

Overall Required Work
While AP’s camera and gimbal support have been rapidly advancing, there are still substantial areas that require improvement. A list of those improvements is listed here camera enhancements and gimbal enhancements . Here are the gaps that needs to be filled out:

  1. Making Usability better and easier

    • RC-only control is difficult because there are so many camera and gimbal features. Here’s a list of the 8 basic features many pilots need. This is more than many transmitters can support.

      • gimbal pitch and yaw control (rate or angle)
      • change gimbal mode (RC targetting vs MAVLink targetting)
      • switch between RC targetting “lock” (aka earth-frame yaw) vs “follow” mode (aka body-frame yaw)
      • buttons for camera zoom, autofocus, take picture, start/stop video
      • Our current ground stations (e.g. MP, QGC) controls are spread out across different screens. For example MP has:
        • Data / Actions tab controls gimbal mode
        • Data / Payload tab controls gimbal pitch and yaw
        • Data / Aux function tab can be configured to control the other features (lock/follow, camera zoom, etc)

    • Our GCSs are simply missing many important features including

  2. Support more gimbals and more gimbal features

    • We currently support the DJI RS2/RS3Pro, Gremsy, Siyi A8/ZR10/ZR30, ViewPro but not all to the same level (see below image, thanks to @rmackay9 for this)
      Thumbnail_1684837371653
      Thumbnail_16848373716531280×720 88.8 KB
      Other gimbals/cameras we should support (or support better) are listed below.
      • ViewPro ← should send autopilot location so can be embedded in camera using EXIF
      • Xacti DroneCAN camera/gimbal (WIP)
      • PhaseOne camera
      • WorksWell thermal cameras
      • OTUS DST gimbal
      • NextVision

  3. Ethernet support.

    • Many users’s autopilots are low on serial ports. We can simply connection by using Ethernet connection to the camera/gimbal
    • ensure all our serial drivers work over Ethernet

  4. Testing environment

    • We need a good testing environment to ease development and testing of the above features, This will also help with the development of advanced features (see below). It will be developed on the latest gazebo simulator (re-branded from ignition to gazebo).
      download

  5. Advanced stuff. This may be outside the scope of this project but it is important to prepare for the next steps

    • once we realise cameras are a new sensor we can potentially use them for all kinds of things including:
      • precision landing. Many cameras have built in AI based object identification and tracking. We could use this instead of a special purpose camera thus saving on weight.
      • optical flow?
      • closed loop VO? e.g. compare two pictures to estimate vehicle’s current position
      • visual follow-me?
      • object avoidance (using AI object identification)

Proposed Work
Adding New message Support that include:

  • GIMBAL_MANAGER_INFORMATION
    Gives Information about a gimbal. Like vendor, angle limits, name ,cap flags etc.

  • GIMBAL_MANAGER_STATUS
    Tells current status of gimbal manager. Broadcaster at some low rate.

  • GIMBAL_MANAGER_SET_ATTITUDE
    Sets gimbal attitude. (Angles and Rate)

  • CAMERA_INFORMATION
    Information about a camera like focal length, vendor, resolution, sensor_size, cam definition uri etc.

  • CAMERA_CAPTURE_STATUS
    Tells Image and video status, recording time, capacity, image_count etc.

  • CAMERA_SETTINGS
    Settings of a camera inlcluding zoom level, focus level etc.

  • CAMERA_FOV_STATUS
    Information about the field of view of a camera

  • CAMERA_IMAGE_CAPTURED
    Gives information about a captured image. Its very important to have information like lat, long, alt, mount angles etc for doing some calculations on image to extract additional information of objects or points from an image.

  • VIDEO_STREAM_INFORMATION
    Information about a video like bitrate, framerate, resolution, streamcount etc.

  • VIDEO_STREAM_STATUS
    Information about the status of a video stream. Mostly similar to previous one.

  • CAMERA_TRACKING_IMAGE_STATUS
    This is sent while in active tracking.

  • CAMERA_TRACKING_GEO_STATUS
    Gives information of Geographical data of object including lat, long, alt, velocity, distance etc.

Adding new commands support

  • MAV_CMD_IMAGE_START_CAPTURE
    It starts image capture sequence. Interval between two consecutive captures and number of images to capture can also be provided.

  • MAV_CMD_IMAGE_STOP_CAPTURE
    It stops image capture sequence.

  • MAV_CMD_VIDEO_START_CAPTURE
    It starts video capture. Stream Id can be provided where providing 0 means all streams. Status frequency parameter tell the rate of CAMERA_CAPTURE_STATUS message sent continuously.

  • MAV_CMD_RESET_CAMERA_SETTINGS
    It resets all camera settings to Factory Default

  • MAV_CMD_SET_CAMERA_MODE
    It sets camera running mode.

  • MAV_CMD_DO_TRIGGER_CONTROL
    It enable or disables on-board camera triggering system.

  • MAV_CMD_VIDEO_START_STREAMING
    Starts Video Streaming

  • MAV_CMD_VIDEO_STOP_STREAMING
    Stops Video streaming.

New features

  • Support for mount following a vehicle with provided sysid. (Completed)
  • Support for pointing gimbal at Circle-centre. (Completed)
  • Focus position support.
  • Zoom step support.
  • log corresponding gimbal mount angles in CAM messages #23816 (WIP)
  • AP_SCRIPTING: add bindings for AP_CAMERA (Completed)
  • Newer SiYI Gimbal driver that uses angle control instead of rate control as the old one, the current driver uses a P control that will not be required. (WIP)
  • Object Tracking
  • Helping QGC developer to ensure AP provides all info required for new camera gimbal control screen.

Some of the Above listed tasks has been already completed and merged into master. Cause this is a need based project, many of the new feature may be introduced later based on the suggestions and requirements of the community and users, So the end goal of this project is to upgrade the system and fix major issues with gimbal and cameras and making their support more better, robust and easier to use.

Hardware Sponsors
I am very thankful to CUAV for their continuous support and sponsoring the hardware for this project. (2 CUAV v6x, 1 Siyi Zr10 and 1 GPS)

Thumbnail_1685475048348
Thumbnail_16854750483481280×720 137 KB

I am very thankful to all the community members who have helped me so far, and I would like to receive more suggestions, And if you want something to add let me know by providing your valuable feedback.

Thankyou!

10 Likes