Testing 3.5 on my Solo - She flies!

mtbsteve · April 1, 2017, 12:18pm

resumed testing from today on some of the outstanding topics:

Flight tests at max speed in Loiter and AltHold with high amp peak loads in radical maneuvres: ok
Outstanding test of POSHOLD flight mode OK
Smartshots: zipline, ok
Gimbal now fully works (with MNT_DEFLT_MODE set to 3)
Land detector works well now (with GRND_EFFECT_COMP set to 1)
Solex Geofencing works

Not OK:

Smartshots follome, Selfie (whereas Selfie is rather a Solo issue)
GoPro does not turn off, but only turns on (GP4B on v3 firmware)
No distance display on controller and Solex - however in in MP works (works for @Pedals2Paddles)
No haptic feedback when Solo approaches max distance or altitude limits (Solo starts drifting sideways once the distance limit is reached)
No RTH when Solo approaches max distance/altitude limits despite of FENCE_ACTION setting

Major issue is that RTH and RTM do not work in all situations I tested (triggered by RTH button, failsafes)
Strange thing is that RTH worked until yesterday, see logfile here:
https://1drv.ms/u/s!AnKeW8KMoCcyizw1sp328YFccOva
But from today onwards it repeatedly breaks. When triggered, Solo goes into RTH for about 2-3 sec and starts climbing a couple of meters, then switches back to loiter. See log here:
https://1drv.ms/u/s!AnKeW8KMoCcyiztnh-1Q17slW7ll

My current parameter file:
https://1drv.ms/u/s!AnKeW8KMoCcyizp8J_KkOwvcWS_6
Would be great if someone of the developers could have a look into the RTH thing - this is the only major defect I have found so far in 3.5 running on Solo. Otherwise very decent performance - big thank you to the dev team!

Pedals2Paddles · April 1, 2017, 1:53pm

BOOM. From the Solo’s Shot Log file. This explains everything! Is the home location being stored differently on the pixhawk now compared to how it was in 3.3?

Mission Planner shows the correct home location because it’s getting it straight from the Pixhawk the way it always does. But the companion computer is having a hard time getting it. And that’s why the guided mode RTH and controller’s home distance isn’t working.

The Solo’s RTH/RTM uses the companion computer’s own routine via guided mode on the pixhawk. All the stuff it does in guided mode is logged in the “shot log” file. So even the RTH is “smart shot” that it logs. Looks like the companion computer is having a hard time obtaining the home location based on AttributeError: ‘NoneType’ object has no attribute 'lat’

The shows the most recent calls, to include which call bombed out. Upon not being able to obtain the home location, it gave up and kicked it back to loiter.

Mar 31 21:21:01 3dr_solo local2.info shot: [shot]: Entering shot Return Home.
Mar 31 21:21:01 3dr_solo local2.info shot: [RC] Enabling stick remapping
Mar 31 21:21:01 3dr_solo local2.info shot: [RTL] Try Guided
Mar 31 21:21:01 3dr_solo local2.info shot: [RC] detach from rc_ipc.put
Mar 31 21:21:01 3dr_solo local2.info shot: [RC] Disabling stick remapping
Mar 31 21:21:01 3dr_solo local2.info shot: Traceback (most recent call last):
Mar 31 21:21:01 3dr_solo local2.info shot:   File "/usr/bin/shotManager.py", line 136, in Run
Mar 31 21:21:01 3dr_solo local2.info shot:     self.buttonManager.parse()
Mar 31 21:21:01 3dr_solo local2.info shot:   File "/usr/bin/buttonManager.py", line 105, in parse
Mar 31 21:21:01 3dr_solo local2.info shot:     self.handleButtons((msg[1],msg[2]))
Mar 31 21:21:01 3dr_solo local2.info shot:   File "/usr/bin/buttonManager.py", line 267, in handleButtons
Mar 31 21:21:01 3dr_solo local2.info shot:     self.shotMgr.enterShot(shots.APP_SHOT_RTL)
Mar 31 21:21:01 3dr_solo local2.info shot:   File "/usr/bin/shotManager.py", line 273, in enterShot
Mar 31 21:21:01 3dr_solo local2.info shot:     self.curController = ShotFactory.get_shot_obj(shot, self.vehicle, self)
Mar 31 21:21:01 3dr_solo local2.info shot:   File "/usr/bin/shotFactory.py", line 31, in get_shot_obj
Mar 31 21:21:01 3dr_solo local2.info shot:     return shot_class(args[0], args[1])
Mar 31 21:21:01 3dr_solo local2.info shot:   File "/usr/bin/returnHome.py", line 82, in __init__
Mar 31 21:21:01 3dr_solo local2.info shot:     self.returnAlt = self.coneOfAwesomeness(self.returnAlt)
Mar 31 21:21:01 3dr_solo local2.info shot:   File "/usr/bin/returnHome.py", line 152, in coneOfAwesomeness
Mar 31 21:21:01 3dr_solo local2.info shot:     homeDistance = location_helpers.getDistanceFromPoints(self.vehicle.location.global_relative_frame, self.homeLocation)
Mar 31 21:21:01 3dr_solo local2.info shot:   File "/usr/bin/location_helpers.py", line 16, in getDistanceFromPoints
Mar 31 21:21:01 3dr_solo local2.info shot:     dlat    = (loc2.lat - loc1.lat)
Mar 31 21:21:01 3dr_solo local2.info shot: AttributeError: 'NoneType' object has no attribute 'lat'
Mar 31 21:21:01 3dr_solo local2.info shot: 
Mar 31 21:21:03 3dr_solo local2.info shot: +-+-+-+-+-+-+ Starting up v2.4.1-1 +-+-+-+-+-+-+
Mar 31 21:21:03 3dr_solo local2.info shot: [RC] Disabling stick remapping
Mar 31 21:21:03 3dr_solo local2.info shot: [app]: Ready for connections from app.
Mar 31 21:21:03 3dr_solo local2.info shot: [button]: Creating a new thread to connect to Artoo.
Mar 31 21:21:03 3dr_solo local2.info shot: [button]: read in button mappings
Mar 31 21:21:03 3dr_solo local2.info shot: [button]: Button A - shot FLY, mode Stabilize
Mar 31 21:21:03 3dr_solo local2.info shot: [button]: Button B - shot FLY, mode FLY: Manual
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: read enabled value from settings of 1.
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: sending gimbal enabled param (GMB_GP_CTRL) to 1.000000
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: Inited GoProManager
Mar 31 21:21:03 3dr_solo local2.info shot: [RewindManager] Init
Mar 31 21:21:03 3dr_solo local2.info shot: [shot]: No gimbal detected.
Mar 31 21:21:03 3dr_solo local2.info shot: [shot]: Restart in air.
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: Gopro status changed to 2
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: Gopro recording status changed to 1
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: sending Gopro state to app
Mar 31 21:21:03 3dr_solo local2.info shot: [app]: Can't send packet - app is not connected!
Mar 31 21:21:03 3dr_solo local2.info shot: [app]: Can't send packet - app is not connected!
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 11
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 10
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: Gopro get request for command 4 failed with status 1
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 9
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: Gopro video resolution changed to 8
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: Gopro video frame rate changed to 4
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: sending Gopro state to app
Mar 31 21:21:03 3dr_solo local2.info shot: [app]: Can't send packet - app is not connected!
Mar 31 21:21:03 3dr_solo local2.info shot: [app]: Can't send packet - app is not connected!
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 8
Mar 31 21:21:03 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 7
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: Gopro photo resolution changed to 4
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: sending Gopro state to app
Mar 31 21:21:04 3dr_solo local2.info shot: [app]: Can't send packet - app is not connected!
Mar 31 21:21:04 3dr_solo local2.info shot: [app]: Can't send packet - app is not connected!
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 6
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 5
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 4
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: Gopro get request for command 10 failed with status 1
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 3
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: Gopro get request for command 11 failed with status 1
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 2
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 1
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: sending message from the queue.  Size is now 0
Mar 31 21:21:04 3dr_solo local2.info shot: [callback]: Mode changed from RTL -> LOITER
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: Gopro get request for command 10 failed with status 1
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: Gopro get request for command 11 failed with status 1
Mar 31 21:21:04 3dr_solo local2.info shot: [gopro]: Gopro get request for command 13 failed with status 1
Mar 31 21:21:04 3dr_solo local2.info shot: [RewindManager] reset Spline to size 20
Mar 31 21:21:05 3dr_solo local2.info shot: [button]: Connected to Artoo.
Mar 31 21:21:05 3dr_solo local2.info shot: [button]: sending FLY to Artoo as mode

The relevant python code from the companion computer that requesting the home location is as follows. It appears Jason was not happy with this method, but Dronekit wasn’t doing the job at the time to do it better.:

def loadHomeLocation(self):
        ''' Hack method to avoid Dronekit issues with loading home from vehicle'''
        if self.vehicle is None:
            logger.log("no Vehicle!")
                
        #load home from vehicle
        # total grossness for getting HOME_LOCATION! Should be done properly in dronekit instead
        self.vehicle.add_message_listener('MISSION_ITEM', self.handleMissionItem)
        # formulate mavlink message to request home position
        self.vehicle.message_factory.mission_request_send(0, 1, 0)  # target_system, target_component, seq
        #logger.log("loading home from vehicle2")

    def handleMissionItem(self, vehicle, name, msg):
        ''' Handles callback for home location from vehicle '''
        self.vehicle.remove_message_listener('MISSION_ITEM', self.handleMissionItem)
        self.homeLocation = LocationGlobal(msg.x, msg.y, msg.z)
        logger.log("[RewindManager] loaded home %f %f, alt %f" % (self.homeLocation.lat, self.homeLocation.lon, self.homeLocation.alt))
        # Send home to the App
        self.updateAppOptions()
        

    def updateAppOptions(self):
        ''' send home loc to App '''
        # f = float_32
        # d = float_64
        packet = struct.pack('<IIddf', app_packet.SOLO_HOME_LOCATION, 20, self.homeLocation.lat, self.homeLocation.lon, self.homeLocation.alt)
        self.shotmgr.appMgr.sendPacket(packet)

Pedals2Paddles · April 1, 2017, 4:11pm

Seeing that it appears to be requesting mission waypoint zero, the home location, it reminded me. When it was working properly last weekend, I had missions loaded. Since then, I had reset the pixhawk. So anything mission related was gone. So I loaded a mission and tested again today. The problem did not present itself. I could even load a blank mission to clear it and it still continued to work fine. Power cycled, still worked fine. I could change takeoff locations around the field, and it always picked up the new home location and the RTH guided mode thing worked properly.

So, that leads me to this question: what has changed between 3.3 and now as it relates to waypoint zero home location that the companion computer is looking for?

mtbsteve · April 1, 2017, 6:22pm

[details=Here is an example of 2 different RTH failures in the shot log:
]This text will be hidden[/details]
Mar 31 06:39:40 3dr_solo local2.info shot: [shot]: Entering shot Return Home.
Mar 31 06:39:40 3dr_solo local2.info shot: [RTL] Try Guided
Mar 31 06:39:40 3dr_solo local2.info shot: [RC] detach from rc_ipc.put
Mar 31 06:39:40 3dr_solo local2.info shot: [RC] Disabling stick remapping
Mar 31 06:39:40 3dr_solo local2.info shot: Traceback (most recent call last):
Mar 31 06:39:40 3dr_solo local2.info shot: File “/usr/bin/shotManager.py”, line 136, in Run
Mar 31 06:39:40 3dr_solo local2.info shot: self.buttonManager.parse()
Mar 31 06:39:40 3dr_solo local2.info shot: File “/usr/bin/buttonManager.py”, line 105, in parse
Mar 31 06:39:40 3dr_solo local2.info shot: self.handleButtons((msg[1],msg[2]))
Mar 31 06:39:40 3dr_solo local2.info shot: File “/usr/bin/buttonManager.py”, line 267, in handleButtons
Mar 31 06:39:40 3dr_solo local2.info shot: self.shotMgr.enterShot(shots.APP_SHOT_RTL)
Mar 31 06:39:40 3dr_solo local2.info shot: File “/usr/bin/shotManager.py”, line 273, in enterShot
Mar 31 06:39:40 3dr_solo local2.info shot: self.curController = ShotFactory.get_shot_obj(shot, self.vehicle, self)
Mar 31 06:39:40 3dr_solo local2.info shot: File “/usr/bin/shotFactory.py”, line 31, in get_shot_obj
Mar 31 06:39:40 3dr_solo local2.info shot: return shot_class(args[0], args[1])
Mar 31 06:39:40 3dr_solo local2.info shot: File “/usr/bin/returnHome.py”, line 81, in init
Mar 31 06:39:40 3dr_solo local2.info shot: self.returnAlt = max((self.vehicle.location.global_relative_frame.alt + RTL_CLIMB_MIN), self.rtlAltParam)
Mar 31 06:39:40 3dr_solo local2.info shot: TypeError: unsupported operand type(s) for +: ‘NoneType’ and 'int’
Mar 31 06:39:40 3dr_solo local2.info shot:
Mar 31 06:39:43 3dr_solo local2.info shot: ±±±±±±+ Starting up v2.4.1-1 ±±±±±±+[details=Summary][/details]

Here is another example of a failed RTH in the shot log:

This text will be hidden

Apr  1 08:20:21 3dr_solo local2.info shot: [shot]: Entering shot Return Home.
Apr  1 08:20:21 3dr_solo local2.info shot: Entering shot Return Home.
Apr  1 08:20:21 3dr_solo local2.info shot: [RC] Enabling stick remapping
Apr  1 08:20:21 3dr_solo local2.info shot: [RTL] Try Guided
Apr  1 08:20:21 3dr_solo local2.info shot: [RC] detach from rc_ipc.put
Apr  1 08:20:21 3dr_solo local2.info shot: [RC] Disabling stick remapping
Apr  1 08:20:22 3dr_solo local2.info shot: Traceback (most recent call last):
Apr  1 08:20:22 3dr_solo local2.info shot:   File "/usr/bin/shotManager.py", line 136, in Run
Apr  1 08:20:22 3dr_solo local2.info shot:     self.buttonManager.parse()
Apr  1 08:20:22 3dr_solo local2.info shot:   File "/usr/bin/buttonManager.py", line 105, in parse
Apr  1 08:20:22 3dr_solo local2.info shot:     self.handleButtons((msg[1],msg[2]))
Apr  1 08:20:22 3dr_solo local2.info shot:   File "/usr/bin/buttonManager.py", line 267, in handleButtons
Apr  1 08:20:22 3dr_solo local2.info shot:     self.shotMgr.enterShot(shots.APP_SHOT_RTL)
Apr  1 08:20:22 3dr_solo local2.info shot:   File "/usr/bin/shotManager.py", line 273, in enterShot
Apr  1 08:20:22 3dr_solo local2.info shot:     self.curController = ShotFactory.get_shot_obj(shot, self.vehicle, self)
Apr  1 08:20:22 3dr_solo local2.info shot:   File "/usr/bin/shotFactory.py", line 31, in get_shot_obj
Apr  1 08:20:22 3dr_solo local2.info shot:     return shot_class(args[0], args[1])
Apr  1 08:20:22 3dr_solo local2.info shot:   **File "/usr/bin/returnHome.py", line 82, in __init__**
**Apr  1 08:20:22 3dr_solo local2.info shot:     self.returnAlt = self.coneOfAwesomeness(self.returnAlt)**
Apr  1 08:20:22 3dr_solo local2.info shot:   File "/usr/bin/returnHome.py", line 152, in coneOfAwesomeness
Apr  1 08:20:22 3dr_solo local2.info shot:     homeDistance = location_helpers.getDistanceFromPoints(self.vehicle.location.global_relative_frame, self.homeLocation)
Apr  1 08:20:22 3dr_solo local2.info shot:   File "/usr/bin/location_helpers.py", line 16, in getDistanceFromPoints
**Apr  1 08:20:22 3dr_solo local2.info shot:     dlat    = (loc2.lat - loc1.lat)**
**Apr  1 08:20:22 3dr_solo local2.info shot: AttributeError: 'NoneType' object has no attribute 'lat'**[/details]

Difference marked with asteriks. Hope this helps to trace down the issue.

pyrate · April 2, 2017, 1:35pm

Dang nice work guys
had no idea solo would still work with Solex and 3.5
exciting

mtbsteve · April 2, 2017, 4:40pm

One more thing to be aware of. I did a factory reset today which bricked Solo.

Reset progressed on Artoo, but never completed in the Pix. Subsequently it couldn’t be paired with the controller anymore.
Only solution was to hardwire the Pix to Mission Planner via USB, and then upload the original 1.3.1 fw and parameter file.
Then the reset eventually completed, and I could pair and reinstall everything again.

I suspect the same underlying problem as seen in the RTH issue. There appear to be some issues in the Artoo-Pixhawk communication.
And to me it appears that this is also related to the MNT_DEFLT_MODE setting in combination with an incompatible change in 3.5 over the 3DR fw which hampers the Artoo talking to the Pix.

Pedals2Paddles · April 2, 2017, 5:40pm

The same happened to me when I tried a factory reset. It’s compounded by the LEDs and beepers not working the same way they do on the Solo branch, so you can’t actually tell what it’s doing, or trying to do.

Pedals2Paddles · April 3, 2017, 9:21am

The home position crashing the Solo’s Smart RTH/RTM procedure has a fix going into Master! It was indeed as I suspected… requesting the home location by asking for Waypoint zero. And if there was not and never had been a mission loaded, Arducopter’s response to that request was that there is no mission so you’re asking for something that doesn’t exist. @peterbarker created a commit that will let Arducopter always respond to the mission waypoint zero request, even if there is not and never has been a mission loaded. WP0 will always report the home position, whether there is a mission loaded or not.

So the fix is in and I believe will come out with Randy’s next release, which I believe will be 3.5-rc4. This solves a major operational reliability an safety issue for operating master on the Solo!

David_Boulanger · April 5, 2017, 7:07pm

3.2 has EKF 1. I use it on one of my aircraft

Pedals2Paddles · April 9, 2017, 3:52pm

I tried to bring the 3DR code for the ESC slewing into master. I thought I had it. The code compiled and it flew nicely. But it’s hard to tell in flight if it worked or not. I’m have a hard time determining what is going on here. It seems odd to me that I can push this thing really hard, literally trying to make it fail, and it remains rock solid. But when it tries to RTH, this happens. Twice.

I let the failsafe kick in. The Pixhawk switched to RTL mode. The companion computer immediately switched it to guided and kicked in the Smart RTH. It climbed up 30 feet. All normal. Then instead of flying home, it started getting all twitchy for several seconds. The compass EKF went yellow. Then after several seconds, the compass EKF cleared. Then it tumbled and flipped out of control. Until it landed in the end of the lake. Last time, it recovered from the tumble before crashing. This time, no such luck.

I only have a tlog. https://drive.google.com/open?id=0BwxNJyHTG9XJYXRqLVh2XzFudHc

Pedals2Paddles · April 9, 2017, 11:17pm

Here’s a video. I can’t believe the file on the GoPro was intact. From the ground, I could see and hear the aircraft twitching like it was losing its mind in some way. The video confirms what I thought I saw. The aircraft was twitching and it seems the attitude information going to the gimbal was equally out of whack. This went on for about 6 seconds. Then it tumbled. I am really not sure what to make of this one.

Cman · April 12, 2017, 3:56pm

Looked like it was possess.
What are you going to do now?

Pedals2Paddles · April 12, 2017, 8:09pm

I have another best buy Solo. My green cube arrives today as well. So I will be back in the air for major testing on Friday and Saturday. Lots of good Solo stuff has or will be going into ArduCopter master over this week and last to test while I’m at it.

Pedals2Paddles · April 16, 2017, 12:21am

Some videos from today’s testing. Burned through 6 batteries and each one went great… finally! I had to rework the manner in which 3DR configured the failsafes and return home functions due to serious safety and reliability issues. I also put everything up to full rabbit on the app speed sliders. Basically tried to make it fail. And it never failed! Stable flight, no flipping out, no EKF kanipshits, reliable RTL/RTM, failsafes are actually safe now. All my testing up to now was very frustrating due to these software and hardware problems. The @proficnc Green Cube fixed the hardware issues. And I think I fixed most of the software issues effecting flight safety and reliability. Between myself and @mtbsteve, parameters are in pretty good shape. So testing today was great! I was happy and smiling the whole time rather than cringing and hoping I don’t die.

Disabled the “Return to Home Smart Shot”, replacing any instance of it with ArduCopter RTL. This fixed a LOT of safety and reliability issues. This situation is what let to most of my crashes and near crashes. All the crazy twitching, EKF overloads, ESC failures, etc were from that. It is now smooth as butter working as you’d expect from ArduCopter RTL.

SMBUS cell voltage monitoring courtesy of @WickedShell. I sacrificed this battery to science, letting it run down into the ground to see what happened. The two cells that tanked were low through the whole flight, and clearly died first.

Cman · April 16, 2017, 3:16am

@Pedals2Paddles what is the hardware issue that the green tube fixed ?

proficnc · April 16, 2017, 4:54am

Awesome job! Glad to see this all coming together! A lot of great work by so many people!

pyrate · April 22, 2017, 3:28am

Joined the club tonight
what do we need to test specifically?

going to be off next week so I plan to burn through a lot of batteries or fly till it crashes

Teni · June 14, 2017, 4:40pm

And what happend to this discussion, great work, but are we done? Was there not still some issues to workout?

Pedals2Paddles · June 14, 2017, 7:26pm

It has made great progress. See here for a recent update:
http://discuss.ardupilot.org/t/update-on-mastering-solo-major-milestone-reached/17959

pyrate · June 14, 2017, 11:20pm

Just waiting on the changes to become part of master
it flies great, smart shots work, and even solved the long disarm times.
Getting a second cube and here for my primary solo soon
at that point both my solos will be on master and moving forward