I am trying to run a fixed wing simulation that outputs an FPV camera feed that I can ingest with opencv for the purpose of an autopilot running computer vision software.
I have so far tried gazebo classic and gz-garden and haven’t been able to figure out how to do this. I hacked my way through some xml files and managed to add an fpv camera to a zephyr model, but not only is it facing sideways down the left wing, but I cannot access the feed (despite it being displayed in the gazebo gui).
I have also tried X-plane, and I am still struggling to connect it to SITL. Even if I were to achieve this, the only way of getting the camera feed is by simply physically mounting a USB webcam on a pile of books in front of the screen…
TL;DR - has anyone managed to set up SITL, gazebo and an FPV camera feed that can be ingested into your application for a fixed wing arduplane?
I’ve spent the whole day trying to get X-Plane 11 working and I can’t. I can see people have had trouble since after version 11.25. I have tried to downgrade in steam using the console commands but am having no success. I simply get no connection in mission planner, and the SITL commands stated in the documentation don’t work either. At this point I’ve tried following everything I can find online, nothing has worked.
I can’t fork out the big spend for realflight either.
EDIT - I’ve managed to connect X-Plane to mission planner. Great news. But the control inputs are all wrong - rudder is stuck at full left and I can’t change it. I also can’t control it in mission planner, only view it.
MORE EDITS: Connected to X-Plane. Inputs were all over the place. Had to go into the parameters and change them to be more appropriate. The channel mappings were wrong as well. My mappings are now:
pitch - 4
throttle - 5
yaw - 3
I cannot make roll work though. There seems to be no channel for it. I have even gone into my transmitter settings and mapped channel 6 to Ail and no change.
EDIT - so I got roll working, turns out I can change an output on the transmitter and had to change the centerpoint as well. Taking off was a mess though, a combination of my control correcting ardupilot on takeoff. Managed some flying in auto and guided mode and managed to run my own application connected to the SITL instance as usual, issuing control commands (SET_ATTITUDE_TARGET) to the aircraft and seeing it respond in mission planner and X-plane. All quite satisfying, but an incredible hassle to set up.
And finally, realizing that X-Plane is probably not the simulator I need anyway, because the physics of full sized aircraft are not nearly similar enough to RC planes for my purposes of developing control applications. The response was incredibly slow compared to SITL’s standard plane model/s, which I’m not surprised by.
So much effort here and my ultimate goal is the camera feed. At least with X-plane I can make the cockpit invisible and could have pointed a webcamera at the screen…
If the camera is pointing in the wrong direction you will need to rotate it by adding a pose to the camera link (most likely <pose degrees="true">0 0 0 0 0 90</pose> from your description).
Gazebo sensors, including the cameras, publish messages to the gz-transport layer. You can inspect the camera data using gz topic -e -t <your camera topic>. This will be the same topic the Image View gui plugin subscribes to.
To make the data available to other applications you can either write a tool that subscribes to the messages yourself (see the Gazebo tutorials for details), or if you are using ROS or ROS 2 you can use ros_gz to map the Gazebo messages to standard ROS image formats. There will be ROS tools for opencv and other image manipulation packages.