Hi everyone,
Updated Video:
I’ve been working on something a bit unusual and wanted to share where it’s at. Fair warning — I’m not a firmware developer by trade, and some of what follows might make the proper devs wince, and yeah lots of AI. But it flies, and I think the idea has legs, so here goes.
Original Video
**How it started — a backup camera that got ideas above its station**
This whole thing began as a virtual camera for QGroundControl. I wanted a synthetic 3D view rendered from cached Google 3D Tiles — basically a “what the drone sees” window in the GCS using the mesh data downloaded. Just a nice-to-have backup perspective for situational awareness, night flying or fog flying.
Then one evening it hit me: if I can render the 3D world from the drone’s point of view, I can cast rays into it. And if I can cast rays, I can generate OBSTACLE_DISTANCE messages. And if I can do that… I can feed BendyRuler.
So that’s what I did. Probably should have thought harder about whether I *should* before I figured out that I *could*.
**What it actually is**
**Balliyang** (Bat in Wiradjuri tradition) is a QGroundControl custom build plugin that:
-
Downloads and caches Google 3D Tiles for any area, or you can add your own mesh
-
Renders them in an offscreen OpenGL context
-
Casts 36 rays at 4Hz across multiple pitch layers (level, ±30°, ±60°) from the drone’s position through the cached mesh
-
Sends the results as MAVLink OBSTACLE_DISTANCE messages back to the flight controller
-
ArduPilot’s BendyRuler then uses this data for path planning
The virtual camera feed shows up as a small 3D window in the fly view, and there’s a elliptical proximity HUD overlaid on it.
Pros:
-
Runs on the flight controller — no companion computer dependency, no single point of failure, no expensive heavy Lidar.
-
a great way to tune behavior for real lidar without risking it.
-
Works in AUTO, GUIDED, and STABILISED/LOITER modes
-
Pre-cached mesh means no internet required in flight
-
Lightweight — 36 rays at 4Hz is trivial compute
-
Uses standard MAVLink OBSTACLE_DISTANCE — no ArduPilot core changes needed
-
Works anywhere Google 3D Tiles has coverage (most cities worldwide) you can provide your own Mesh for wares it does not have.
-
Provides collision prevention even for manual flight
Cons:
-
Mesh is only as current as the last download — new construction or temporary obstacles won’t be there
-
Trees and vegetation are captured in the mesh but their shapes are approximate
-
This is NOT a replacement for real sensors — it’s a supplement for known, static, mapped obstacles
**BendyRuler — standing on the shoulders of giants**
I want to be really clear: BendyRuler is a genuinely clever piece of engineering. The multi-step lookahead, the bearing resistance, the way it balances progress against safety — Randy Mackay designed that architecture gave it real thought, and it shows. The core algorithm is elegant. Sorry I bastardised it.
What I found was that BendyRuler needed some tuning and a few additions to handle dense urban environments where you’ve got 20-metre corridors between buildings, tight corners, and obstacles on all sides. The kind of environment it probably wasn’t originally designed for — because until now, nobody was feeding it synthetic LiDAR data from cached mesh in downtown Sydney.
The main changes I made (all contained within BendyRuler and WPNav_OA, no core ArduPilot changes):
- **Tunable scoring weights** — 7 new OA_BR_W_* parameters so the balance between progress, margin, deficit penalty, pitch preference, and braking can be adjusted via MAVLink without recompiling
- **Updated the OA path planner to run at 4Hz** instead of 1Hz, matching the LiDAR scan rate
- **Stronger bearing resistance** — if the current path has positive margin, commit to it. Don’t change direction unless the path is actually blocked. This was the single biggest improvement for corridor navigation
- **Progressive braking** — linear speed reduction as margin decreases, with a 30% minimum so the drone never crawls to a stop in a wide corridor
- **Direct path override** — when the straight line to the waypoint is clear, skip the search entirely and fly direct. Prevents scoring noise from causing wandering in open space
- **Pitch layer restrictions** — level flight preferred, climbing only as a last resort, no descending into the city (rooftops look “open” from above but you end up in the building zone)
- **Hard stop** — if margin reaches zero in all directions, hold position. Never push through a wall – Probably need to add a back up logic.
- **Velocity feedforward and destination smoothing** in WPNav_OA to prevent the 4Hz update rate from causing trajectory jitter
- **Position prediction** in the ray caster — compensates for ~250ms of MAVLink lag by predicting the drone’s position forward using velocity before casting rays
I also spent an embarrassing amount of time on yaw control before learning that the best approach was to barely touch it. The drone navigates by translating, not by rotating. Heavy-handed yaw control just caused oscillation. The final solution is a smoothed trajectory-following yaw with asymmetric speed filtering — it took about 15 attempts to get right, and I’m still not 100% sure I understand why the final version works.
**The video**
This is the drone navigating a mission through Sydney CBD — from Hyde Park through the streets around Macquarie Street to Sydney Hospital and back. Three waypoints through urban corridors. No manual intervention once the mission starts. It’s not perfect — you’ll see it hesitate at the first building approach and there’s some oscillation in tight spots — but it completes the route without hitting anything, which is more than I expected when I started this.
**Not just AUTO — works in GUIDED and STABILISED too**
Worth mentioning: because this feeds standard OBSTACLE_DISTANCE messages to the flight controller, it works across flight modes. In AUTO it provides the path planning you see in the video. In GUIDED it prevents collisions while flying to a commanded position. And in STABILISED/LOITER, the simple avoidance layer (AC_Avoid) uses the same proximity data to stop the pilot flying into buildings — the drone just refuses to go closer. You get collision prevention for free in every mode, with full autonomous navigation in AUTO.
**Where this could go — companion computer pipeline**
The current setup runs the ray casting on the GCS (my MacBook), which obviously isn’t going to work if you lose link.
But the path forward is clear:
**Near term**: Move the ray caster to a companion computer (Raspberry Pi 5 or 4). The cached mesh lives on local storage, the ray casting is lightweight (36 rays at 4Hz is trivial), and the OBSTACLE_DISTANCE messages go to the FC over serial. No internet required in flight — the mesh is pre-cached. This gives you local position-aware obstacle avoidance at very low computational cost.
**Medium term**: Live mesh generation from onboard cameras. Generating a dense point cloud in real-time, and cache it for later.
**The honest truth about limits**: I think I’ve taken BendyRuler about as far as it can go for dense urban navigation. It’s a reactive planner — it probes directions, scores them, picks the best. It works brilliantly for what it was designed for, and with the additions above it handles corridors surprisingly well. But for really complex urban environments — think multiple turns, dead ends, narrow gaps — you probably need a proper path planner that builds a route through the mesh ahead of time. Something like a visibility graph or RRT* through the 3D obstacle space. BendyRuler would still be valuable as the real-time local avoidance layer, but the strategic routing would come from the companion computer.
The downside of a full path planner: it’s a lot more computation, it needs the full mesh in memory, and it introduces a dependency on the companion computer being alive and healthy. BendyRuler running on the FC with synthetic LiDAR has a beautiful simplicity — the FC makes all decisions, no single point of failure. A companion-computer path planner trades that simplicity for capability. There’s probably a right answer for different use cases.
**Thanks**
To the ArduPilot team for building something I could hack on. To Randy Mackay for creating BendyRuler — seriously, the bones of that algorithm are sound, and it took me a while to appreciate how much the architecture was doing. To Rishabh for adding the vertical search, the bearing resistance, and 3D obstacle support — the bearing resistance in particular turned out to be one of the most important pieces for urban corridor navigation. And to the community for being the kind of place where someone can post "I pointed a virtual camera at Google Maps and taught a drone to fly through Sydney with machine learning " without getting laughed out of the room.
Oh I’m looking for a job BTW.
Happy to answer questions, share code, or accept suggestions for what I’m doing wrong (there’s probably a lot).
Cheers, Leon
