I got it working in the end. My problem was that the EKF needs a position, even if we are only using relative movement commands. This can be solved by manually setting the position using e.g.
- the map in mavproxy.
- something like this script
- mavros and the topic /mavros/global_position/set_gp_origin
I used mavros, and have not tested the others.
In the end, I ended up using EKF2, and the parameters i had worked.