So I have been wrestling with “Drift” issues for a while with our SLAM system as input for the Visual Positioning system. “Drift” is in quotes because the system is actually rather stable on inbetween motions and only seems to overshoot guided waypoints a little and then toiletbowls around them slightly before moving to the next point. We have gone through all manners of debugging this issue, increasing framerate of the SLAM code helped slightly, reducing vibrations in our frame that were affecting the IMU brought the sways from 2m+ to the pretty reasonable 0.5-0.25m we have to day. We have undergone tuning our positioning PIDs and sensor noise rates multiple times but that small amount of drift remains. However when I took a long weekend and my team went testing without me they conducted one flight and in that one flight the drift had all but dissapeared(Log 143). the next week I conducted 3 back to back flights with the second flight only changing which topics I was subscribed to on the ROS system of the onboard computer. Low and behold the second flight(Log 157) had no drift much like the flight from the previous friday, (although the orientation of our path differed slightly the relative position of the points was identical for all three flights.) In the first(log 156) and third (Log 158) we had experienced the small 0.5-0.25m wobble we were accustomed to seeing. There were no changes to my knowledge between these four flight from the flight controller side besides the fact that Log 143 had no home position set and it took place in a different locale.
I am currently still looking at the logs to try and find any differences, however here is the weird part. I first checked the GUID.x of all the points vs the PSC.x and tx and the good flights (Logs 143 and 157) seemed to generally have smoother curves to positions rather than the more step like 156 and 158. Then when I went to the XKF4 values to compare the SP value of all 4 logs only to find the SP values for all 3 lanes missing from log 143 and 157 but present in 156 and 158. Wondering why this was I went to compare the VISP.x values with the other position values of x only to find that 143 and 157 did not have any VISP messages either while logs 156 and 158 did. I am 100% sure that no logging parameters were changed and very sure no flight controller parameters were changed in general. Especially since 156, 157, and 158 were run in relativly quick succession. Today I was able to confirm that it is impossible to fly in guided mode with our system without VISP data, as we have no GPS or other position sources.
Why wasn’t the VISP data logged and how could it be related to the best flights we have been able to achieve with this system?
Any help is appreciated, thanks. I can answer any questions about the setup but the highlevel is a cubeorange running copter 4.1 beta, commit af759bf5. Our external computer is running slam, connected to the FCU we have a laser altimeter and an optical flow sensor.
Also I have videos of the flight but cannot post them with the logs but could definitely stream them over discord or something like that.
I’ve only had a short look at the logs but I agree that the VISP (and VISV) messages are missing from the 143 log (the largest log). I think the vehicle is flying using optical flow because EK3_SRC2_VELXY = 5 (optical flow) and EK3_SRC_OPTIONS = 1 (fuse all velocities).
If the vehicle is only using optical flow then there is no risk of having a misaligned yaw which would explain why you’re not seeing toilet bowling.
It looks like the ARMING_CHECK has the “VisualOdometry” item unchecked so it is possible to arm and takeoff without the VISP messages appearing.
As a side note, if the SLAM system can provide vision-speed-estimates as well that may improve performance because it gives the EKF more information.
By the way, how do you feel about upgrading to Copter-4.1.0-beta? I think this vehicle is running the 4.1.0-dev software from Feb 24th so it is about 6 months old and is not an official release. Using the beta would give us more confidence about what software is used and it also contributes to verification that the beta is working (or not!).
The Optical Flow being used was my initial assumption as well, however when on the ground without running SLAM I get a bad position prearm failsafe and it does refuse to take off. I am going to see if killing the VISP shortly after take off allows it to bypass this failsafe and then use the optical flow output but I would have figured if this was what happened in log 143 that VISP would have been logged initially but then not recieve input after a certain point. Also I should clarify that when I mentioned toilet bowling, for us it is almost completly an x-y movement, our yaw does a pretty good job at remaining stable, would this behavior still arise from misaligned yaw?
Our SLAM estimate should be capable of providing velocities and is something we have considered but not implemented so I will try that out.
We were just discussing updating to the most recent beta which we will go ahead and do now.
Re the incorrect yaw, the issue is if the horizontal position (and movement) does not match the heading. So an extreme example is if the yaw is reported 90 degrees off. In this case the external system may report that the vehicle is at yaw 0 but when the vehicle flies forward the y increases instead of the x. When the position controllers try to reduce any position error they will cause the vehicle to lean in the wrong direction. When the yaw error is small (less than 45deg) we see “toilet bowling”. When the yaw error is larger than 45 degrees, it leads to EKF failsafes and short-term flyaways.
@rmackay9
Ok so its been a bit, but thanks for your response I was trying to verify the incorrect yaw hypothesis but our slam speed estimates were not publishing fast enough to get a stable result that could verify if the incorrect yaw was the issue. The input was jumpy and so it made the drift issues worse. I’ve been working on optimizations to get a higher throughput.
However we were doing some outdoor testing, which is not something we had done for a while, and we basically saw almost the same behavior as when we would do indoor flights. I think it may indicate that the issue is not necessarily a yaw misalignment toilet bowling situation since we were not running slam and we had just calibrated the compass. I’m not sure if it could be position control PIDs because the RTK system was reporting a pretty high level of accuracy and the compass wasn’t great but it wasnt terrible and most of the drift was translations.
Ok so actually I think I found the issue. In the log we posted the EK3_SRCx_VELXY was set to Optical Flow, this was just an oversight since we wanted to be testing out all of the RTK outputs. After setting EK3_SRC1_VELXY to GPS we had a fairly stable flight, not perfect but much better than we had seen before(However I was indoors at the time). We ran it 3 more times (and now that I was watching) the drift error returned in pos_hold and auto, at the time it was dusky and getting darker. After the 3rd time, I checked the parameters and optical flow was still SRC2 and 3 so I went to check the Fuse all velocities param(EK3_SRC_OPTIONS) but forgot what it was called and so I went to the params wiki page and stumbled upon EK3_FLOW_USE, after confirming that I had already turned fuse all velocities off I realized that we were using optical flow for the navigation estimator, Which combined with the lack of light seemed to cause the issue, and so I disabled that. We are now out of light so I will have to check if this solved the issue tomorrow.
I had assumed that the yaw misalignment was the fault of the slam since we have had GPS and optical flow work in the past. However, I asked my friend how the optical flow sensor was attached and it seems double-stick tape was used due to the lack of screw holes so I think our past success might have just come down to luck with how the sensor was aligned.
By the way, the EK3_FLOW_USE parameter specifies how the optical flow is used. E.g. 0=Not Used, 1=used for navigation, 2=use for height estimation. I think this should be left at “1” (Navigation) and then the EK3_SRCx_VELXY and EK3_SRC_OPTIONS should be used to configure when optical flow will be used.
I see, in the final run we were seeing not great SV levels but our GPA SAcc seemed pretty good and didn’t spike in the same places SV did. I think having the SRC2/3_VELXY source set to optical flow would affect the SV value, I tested this by graphing the SV values with OF.Qual/100 -1.23.
However, I didn’t see a lane switch in the logs, and SRC_OPTIONS was set 0 so I wouldn’t have expected to see the stability of the flight affected. So this might not have been the cause. Am I misunderstanding how the lanes affect one another?
Also, our OF quality was sitting around 125, I am not sure what a good value is for the sensor. It should also be noted that the sensor wasn’t calibrated properly however I still wasn’t expecting it to play a roll in this flight.
I wasn’t able to download the log but in any case, source switching is separate from lane switching so any lane switches you see in the logs are not directly related to the active source changing. I’ve updated the GPS/Non GPS transitions wiki page with a small “Logging” section which lists the events that are recorded in the logs each time the active source is changed.
An OF quality of 125 is quite good but calibrating the sensor is quite important. I don’t think I know which optical flow sensor you’re using though and it depends a bit on the sensor. The px4flow for examples doesn’t report it’s quality properly while the Hereflow does.
That makes sense, we are using a hereflow. I was checking out the logs today and from an indoor flight that we performed last night and the xkf4.NI value was between 1 and 10 which from what I saw in the source is pretty good but also the testRatio for optical flow seems to be calculated differently than most of the others where the SV like values able to be integrated when it is less than 1.0 it seems like with NI it would be less than 100.
Since our outdoor flight was fixed via the OF calibration I feel like our indoor drift might be caused by our slam having a slightly drifty world frame and therefore a misalignment of the heading and the positioning as you described earlier.
Would such a misalignment show up somewhere like in XKF4.SP/SM? We dont really have any yaw sources besides slam and the imu so it might be hard to see in logs. I have a spot that we fly sometimes that is outdoors but bowl shaped so we can run slam and gps/compass at the same time so I’m going to try flying there this week to grab a log with some more stuff to compare to our slam.