I don’t know if that helps but I figured it could fit you use case.
On the rover I have an AMB82 mini camera with a built in NPU that can do YOLO vision along with a radar to give it depth, The problem is that it’s very limited in what type of objects it can actually detect. You would need to train it on each object or try and find data sets with specialized objects specific to your environment. When I eventually move to a boat it will become more useful as the verity of objects is reduced dramatically to just some boats and buoys, so a marine specialized data set should work really well.
yolo E is very interesting but its a bit over my head, i have never used python before.
The LLM approach works with a much larger verity of objects, but that comes at the cost of speed. where YOLO can run several times per second, a LLM response is 5-10 seconds so my plan is to have the LLM look far ahead where the sensors like lidar and radar cannot see to try and spot obstructions far enough in advance that the rover can take avoiding actions before it gets there.
The LLM also has randomness in it where it can hallucinate and give some wacky answers, this is much more common with the smaller models as they have less data to work with, generally they are ok with basic tasks like identifying objects but when you try and ask them to estimate how far away it is then it will start telling you the ship on the horizon is 5cm away. The large models are much better at those kinds of estimations. So an approach where it uses the small models to look for obstructions then stops and waits for the larger slow models to figure out what to do about the obstructions seems to be the way to go.
getting the data out is relatively simple as its just using json, my next step is to test some different approaches, one will be just asking the LLM for a distance and angle to an object that it can then use to create a mavlink proximity message that can be sent to ardupilot for bendy ruler to deal with or i can ask the LLM to try and work out a course that i can command in guided mode.
I am now able to send photos to my server for processing with an ESP32 cam module, it works but esp32 cameras aren’t very good. I’m still trying to get it to work on the amb82 mini as it has a much better camera but the http library is different so Im still having issues getting it to send and receive json data.
at the moment, to test I’m just sending this, it needs to be expanded with more rules but for first tests it’s enough. The plan is to drive using GPS with the LLM scanning ahead for things the lidar cant see, then if something is detected it can try and figure out a route around it.
“what way can I move reply left or right to rotate in that direction or forward if that would allow further progress, if you cant see a way around then replay backwards to reverse the vehicle away from the obstruction. single word answer only”
Then im checking the answer
String heading = doc["response"];
if (heading = "Forward") {
command_forward();
} else if (heading = "Backward") {
command_backward();
} else if (heading = "Left") {
command_left();
} else if (heading = "Right") {
command_right();
} else if (heading = "Stop") {
command_stop();
}
}
void command_forward() {
send_guided_move(-1.0, 0.0, 0.0);
}
void command_backward() {
send_guided_move(1.0, 0.0, 0.0);
}
void command_left() {
send_relative_turn_degrees(-90.0);
}
void command_right() {
send_relative_turn_degrees(90.0);
}
void command_stop() {
send_guided_move(0, 0.0, 0.0);
}
then sending a guilded mode command
void send_guided_move(float x, float y, float z) {
mavlink_message_t msg;
uint8_t buf[MAVLINK_MAX_PACKET_LEN];
uint16_t type_mask = (1 << 0) | (1 << 1) | (1 << 2) | // ignore position x,y,z
(1 << 3) | (1 << 4) | (1 << 5) | // ignore velocity x,y,z
(1 << 6) | (1 << 7) | (1 << 8); // ignore acceleration x,y,z
mavlink_msg_set_position_target_local_ned_pack(SYSID, COMPID, &msg,
millis(), // time_boot_ms
TARGET_SYSID, TARGET_COMPID,
MAV_FRAME_LOCAL_NED,
type_mask,
x, y, z, // position
0, 0, 0, // velocity
0, 0, 0, // acceleration
0, 0); // yaw, yaw_rate
uint16_t len = mavlink_msg_to_send_buffer(buf, &msg);
Serial.write(buf, len);
}
void send_relative_turn_degrees(float delta_degrees) {
float delta_rad = degreesToRadians(delta_degrees);
float targetYaw = currentYawRad + delta_rad;
mavlink_message_t msg;
uint8_t buf[MAVLINK_MAX_PACKET_LEN];
uint16_t type_mask = (1 << 0) | (1 << 1) | (1 << 2) |
(1 << 3) | (1 << 4) | (1 << 5) |
(1 << 6) | (1 << 7) | (1 << 8);
type_mask &= ~(1 << 10); // enable yaw
mavlink_msg_set_position_target_local_ned_pack(SYSID, COMPID, &msg,
millis(),
TARGET_SYSID, TARGET_COMPID,
MAV_FRAME_LOCAL_NED,
type_mask,
0, 0, 0,
0, 0, 0,
0, 0, 0,
targetYaw, 0);
uint16_t len = mavlink_msg_to_send_buffer(buf, &msg);
Serial.write(buf, len);
}
The prompt needs a lot more detail and refinement and the json needs parsed properly but in theory it can now drive an ardupilot rover badly using guided mavlink commands.
I have ordered an esp32 I2S audio kit so it has a mic and speaker that way it will be able to take audio commands and reply to people that come up to it. it can also explain what it is and then ask if they know the location of john Connor. I haven’t figured out how to do it locally yet so I will be using google TTS to make it talk
https://www.aliexpress.com/item/1005009881953806.html
In order to test different AI models for consistency I needed to be able to run them locally and renting hardware especially GPUs for long periods of time is very expensive so have built a cheap AI server to run models on the rover using old server parts.
old gaming case and power supply
chinese dual x99 motherboard
2x 2699v3 xeon CPUs
128gb ddr4
2x tesla M40 24Gb GPUs
I was testing on my desktop pc up til now and although its a decent pc with 7950x, 64gb ram and a 24gb rtx3090 it was painfully slow to run if you ran out of VRAM and I didn’t have enough RAM for anything big on the CPU. The 2x 24b tesla m40s can run large models in GPU memory so despite being much older and about a third of the speed of the 3090 because there is double the vram it’s still faster at the large models that exceed a single 24Gb card because it doesn’t have to use the CPU.
I had a couple of issues I had to address with the AI computer, the first was that this computer essentially has the single purpose of driving the rover so I don’t want it on any longer than it has to wasting power, so i wanted to be able to remote control its power button so it could be started and stopped automatically when the rover was active. I also had the problem of the tesla cards not having fans or fan control. so I needed to be able to control the fans
to fix all this I made a stand alone ESP32 C3 control board, It has 2 main functions,
1 is that it connects to a pair of i2c temperature sensors on the back side of each GPU core and to the RPM and TACH fan wires on both fans, it can then control the fan RPM using a PID loop to regulate the cards temperature.
2, The esp32 acts as a remote power switch, it controls the power button and also monitors the power led to verify if the computer is on or off.
With the state of the server now able to be monitored, I can script the rover to wait until ollama has started and is ready to start processing requests before it starts to move. Then when the rover goes to sleep for the night, the AI server will shut down until the rover needs to move again.
There was some flooding recently after some heavy rain and my testing park has turned into a pond, the small hill where the rover parks at night would have been above the water but the rest would have been several feet underwater, some water sensors might be worth installing and some extra instructions for the AI not to move if there are any signs of flooding.
Finally got it outside for testing again, it’s probably going to be out for a month or so while I check that it’s not going to corrupt or do anything strange.
Unfortunately that’s means it’s going to be out until November, which means barely any sunlight and freezing temperatures. It will probably have to sleep most of the day to conserve power as well as at night. There are safeties in the Battery BMS board that won’t let it charge if it gets too cold so I’m going to add some insulation to hopefully keep it above freezing so it can take a charge. A heater might be an option if I end up not being able to charge for long periods of time
Plan B is that I have a pair of batteries so I can swap the empty battery for a full one if it gets critically low.
Due to tall buildings it will get little direct sunlight in the garden as the sun gets lower due to winter so it will be going to the park if it does ok this month as it’s open space with a small hill that will let me park tilted south to face the sun.
you sound like NASA with their mars rovers.
Less budget (guessing), easier to get to - but all the same issues.
It sounded much simpler when I started, lol.
I have read about all the mars rovers and how they eventually ended. Almost all the solar rovers stopped once the panels were covered with dust, they never added a mode that would allow it to restart if they ever got the panels uncovered or a way to properly deep sleep where it could hibernate it was pretty much once the battery was discharged it was just a short across the panel and it was game over.
So my 2 main priorities were deep sleep, where the power consumption can be lowered to almost nothing by using the deep sleep mode on the esp32 combined with being able to shut off power to everything using relays bringing consumption to under 1mw. It also got a black start capability where it can start up from solar with all the batteries totally discharged.
My batteries have a BMS so they will disconnect the battery when it’s too low. when it first gets power from the solar inverter it all goes through a low power 48>5v regulator to the esp32 and modem to power it and charge the 5v backup so it can communicate with the server.
The first 5w will go to the esp32 and modem backup battery and the rest will go to the main battery. Only once both the backup and the main battery have enough charge, the esp32 will close the main relay supplying power to the esp32 power board that will then supply everything else on the Rover.
This has been built with absolute cheapest parts money can buy. 
It was out for about a day before I ran into the first of many issues..
The first problem was because I don’t have a data sim in the LTE modem, it’s set to just use the Halow Wifi for now since because it’s in the garden it would be a waste of money to use LTE so close. Because I never originally designed it to operate like that when I turned off main power, it ended up draining the second ups that was powering the halow Wi-Fi adapter and the Wi-Fi camera, and when that happened I couldn’t reconnect to it.
The issue was caused by the 5v source to charge the 9v UPS coming from the output of the ESP32 main power board. The 9v UPS was originally just for the camera and temporary powering of the halow wifi adapter so it doesn’t have a direct connection to the main battery like the 5v UPS that runs the ESP32 and LTE modem as I didn’t deem it critical since LTE runs from 5v and that’s the main connection, The 9v UPS 5v power supply comes from the esp32 power board and that won’t start without server connectivity. To remedy this issue, I have added a second 48v>5v regulator connected to the output of the main relay so the 9v ups will get power from the main battery as soon as it wakes but not while it’s sleeping so the wifi will get power on startup as long as the main battery has power.
I have been adding some automatic self preservation routines so it won’t just sit powered on until the batteries run out if there is an issue with the internet connection or the server.
I have completely reconfigured its boot up procedure, Now when it first wakes, it tries to connect to my server using halow wifi, If it fails it will switch to LTE and try my server again. Once it connects then it will initialize all the battery sensors and check the voltages of all the batteries, if they are too low it will just send some critical data back to the server like voltages then go to sleep, If it cannot connect then it goes to sleep. If it passes initial startup checks it then connects to the server to see if it should do a full startup or just go back to sleep.
The rovers status is monitored by the server where its scripts will tell the rover when to drive and sleep , its internal routines are just for backup.
Once it’s started up and running it will keep track on how many failed connections there has been to my server and if there is too many then it will try and verify that it can connect to my server and ping google.co.uk to check for connectivity to the internet, then it attempts to change connection if that fails. it will switch between the 2 connections to try and contact the server and then eventually go to sleep to save power of it cannot get a response.
In order to thermally manage the battery I bought a cheap usb heated blanket, it uses around 5 watts to run. The plan is to wrap up he battery and use the temperature sensors from the battery BMS to manage the battery heater in order to keep the battery from freezing and getting damaged from trying to charge it when it’s too cold. I will probably add some foam insulation around it since the better it’s insulated the less energy I will need to keep it warm. at 5 watts I doesn’t get too hot even when left running so the battery should be safe even if there was a malfunction and it stayed on.
From what I can find lithium batteries have a minimum operating temperature of 0c for charging and -20c for discharging so it should be ok to let the battery get cold overnight and just warm it up in the morning to charge it as it rarely gets below -10c.
I have had the BMS working with ardupilot via mavlink but I ran into the issue of it bypassing the BMS due to the negative switching so I’m waiting on a Serial isolation board so I can connect it again.
For now I’m using a LM75 i2c thermal sensor to monitor the battery and operate the heater until I get the BMS connected.
The new 868mhz Halow antenna is now mounted to the top of the rover mast with a 3d printed bracket. im hoping this will let me get a wifi signal at the park and beach that is only 250 meters away saving me a lot of LTE data.
Its first night out in the cold went much better than I expected. The new battery heating system works great, when testing it was only using around 2W average at around 30% duty cycle to keep the battery at 15c when it was 7c outside. I looked at using a PID loop, but the heater is so small and the battery so big that it can keep it within 0.5c just by only turning on if it’s under the set temperature.
This setup was just for testing, the script will be modified so that it only heats the battery in the morning once the solar panels are generating power or at any point if the battery gets down to under 0c The BMS has been programmed so that it won’t allow the battery to charge under 0c so the heating script will try and keep the battery above that temperature at all times.
I now have it starting up and shutting down reliably, the power consumption is down to 5-6W when it’s idling in ECO mode with just an esp32, halow wifi and a 2.4ghz wifi router running.
I’m just switching on the second esp32 with all the weather sensors once an hour for a minute to send data then back off so its average consumption is almost zero. The camera is switched on for 5 minutes when the radar detects something, so its not using power at all times. It should be low enough that it can stay on and running continually other than the darkest days when it will have to go to sleep.
short of any disasters, I will be getting a new LTE sim next week and taking it to the park so I can start getting some miles on it.
Yup - nasa stuff! Keep it up - is super interesting!
I had the rover out driving yesterday in the garden, some minor issues like it driving backwards due to the motor controllers getting reversed when remounting, this caused some head scratching when it started going in reverse when I engaged auto mode as I initally thought it was the avoidance trying to get away from something but once that was figured out and fixed I could so some tuning and it could run waypoints without issue. I don’t let it do lots of laps in the garden since the wheels would just tear up the grass when it does a pivot turn.
I am still in 2 minds if i should leave the pixhawk IMU heater enabled since its not that critical on a rover but it might be a good idea if its going to be starting up in freezing conditions.
Unfortunately It’s not looking good for being able to keep it running every day. Today it was overcast with some rain, the sky was dark grey the whole time and this is the result. The 2x 100W solar panels topped out at 6 watts briefly. Even just running in its lowest power configuration, it’s not enough to keep it running continuously. Sleep is going to be absolutely essential for the rover to not run out of power.
When I realized that it was never going to generate enough power at around 11am I put the rover to sleep, so it’s been waking up every hour to report the solar panel’s output, that’s why it suddenly gets low resolution.
I had done some testing before i put it outside and I discovered that when it’s sleeping it’s using a lot more power than it used to, it took me a minute to work it out but its obvious when I realized. basically, It’s because I changed from the lillygo T7 S3 board to the ESP32-S3-DevKitM-1 Because the S3 Dev board doesnt have an internal battery charger I have just been using the 5v UPS, the problem is that the 5v UPS has a bunch of other things connected to it on the breakout board like LEDs and voltage sensors that all use a small amount of power. It’s not much around 1W with it all added up but that’s still 1000x more than it was before when using the internal esp32 battery. The problem I have is that I am really struggling to find a ESP32 S3 with a servo connector breakout that also has a battery connection.
The only one I can currently find is this S3 Supermini board, it has just enough pins and will run at <1mw when asleep on the internal battery as it only runs the 3.3v rail. I’m thinking of possibly getting a pair of them and splitting off the nav code like managing the GPS receivers, AIS and BMS from all the communications and management components as I never have enough serial ports for everything I want to connect and i would rather not have to mess with software serial ports.
Currently, the sun is only getting to 19 degrees above the horizon, in a months time it’s barely going to get 10 degrees above the horizon and It’s going to get much colder, so I think realistically the rover is going to be able to do one lap per day, but even that might not be possible if its cold and overcast for more than 2 or 3 days.
I am looking to see if I can get weather forecasts to the rover so it can be programmed to react accordingly. Like if It’s going to be rain and cloud for the next few days, or freezing it can try and automatically build up a charge in the battery in advance to be able to run the battery heater and modem.
I am still looking into some kind of mini vertical wind turbine, but I haven’t found anything small that has a tested output. There are lots of “models” but nothing that does more than light up an LED. It doesn’t have to be much, 5 watts could run the electronics continuously. it’s almost always windy when the sky is cloudy so it would compliment the solar.
Awesome project.
These wind turbines aren’t industrial grade but perhaps for a proof of concept
Thanks, Grant.
Thank you, I have looked at all of those, but I can’t find data from someone that’s that’s attached a meter to one of them, the manufacturers specifications are usually optimistic at best.
The vertical turbine in the middle looks interesting, I’m looking more at the vertical ones in case someone tries to touch it, they are generally safer to fingers.
After looking again at how fast the voltage was dropping I initially thought it was just the cold but the battery was unusually warm I realized there had to be something else switched on drawing far more power than 1w, and I think what has happened is one of the motor controllers has been switched on by mistake.
When it switches on the 5v power rail the hoverboards will power on and off a second later, I think while messing with scripts I have managed to switch the power on and off fast enough that it cut power before one of the hoverboards was shut down. I haven’t noticed it on since all the telemetry from it goes via mavlink to the hoverboard controller then to the flight controller that were both powered off so I had no way of knowing that it was on.
I had also just changed a sensor in domoticz for battery power from mA to Amps, and because of that the graphs essentially stop working so it was best to make a new sensor then change it on the rover when It’s next in. it was still giving calculated consumption that is basically power from the battery - power from the solar but that wasn’t sent when in sleep mode as I didn’t think it was critical information, but that has also been rectified for its next update.
When I realized what was happening I have done a power cycle of the hoverboard auxiliary 5v rail that powers the Mavlink Serial splitter and hoverboard controller and when it went to sleep again the voltage stopped dropping.
On all ESP DEV boards the 3.3V is available on one of the peripheral pins. If you connect here a external 3.3V just for your power saving modes and cut the 5V. It is not so elegant but possible
It’s been outside for a week now and things are looking ok so far, its just been asleep most of the time to save power, but I have done a lap of the garden just to make sure everything was working. some minor issues like the camera switcher and gimbal pitch not working, but I think that’s just a loose connector in the gimbal.
It has been overcast and raining for almost a week now. It’s able to sleep and keep charging the battery, but it’s only just bringing the voltage up.
The battery is 48v 20Ah or 960Wh, at its current rate it will take around a month to fully charge, compared to the summertime where it can fully charge in a day. the weather was so bad on friday that the solar inverter never got above 2W a barely 1% of my panels capability.
On the bright side the temperatures have been mild, so the battery has stayed above 10c keeping the heater off and saving power.
I had a look and found this little 3.3v battery module that should work as a sleep battery power source, I should be able to just attach it between the 5v and 3.3v rail along with a protected lithium cell.
I have already ordered the s3 supermini boards so im going to test them but if that’s problematic this is going to be plan B, it should stop that slow voltage drop through the night by reducing its sleep consumption from a 1w down to 1mw.
I have been fine-tuning the power management so now when it wakes up it will stay on for a minute while it gets sensor readings and warms up the battery, then it will go back to sleep. it now also changes how long it sleeps for based on if it’s daylight and what voltage the battery is at. it’s still a work in progress so it will get more conditions and options added once I need it to start moving.
After looking at all the wind turbine options, I decided to just try and build one, as even the cheapest generator that might work is going to be a decent amount of money, and it still might not actually be useful. So after some googling I found some hoverboard turbines, I have a box of hoverboard wheels I saved up when they were going cheap so this looks like the best option for something to test. I couldn’t find anything ready to print that I liked. So I basically just took a vertical wind turbine and a hoverboard tyre off of thingverse then joined them with a flat plate. I can add more sections to make it taller, but hopefully two is enough to make a couple of watts. The hoverboard axle and bearing is very strong so i dont have to worry about the loads on it. its going to be a 3 day print for the first section.
The main controller has crashed and rebooted. I think one of the values i was holding between sleeps has overflowed or a timer has overflowed, or I have a memory leak causing it to fail to start up. it went to sleep and woke up again over 160 times before it failed. I have reduced the number of values held to just the number of times its went to sleep to see if that helps. When I pressed the reset button it started right up so I’m more inclined to think it’s a software issue rather than hardware.
I have extended the functionality of my battery heater, it occurs to me that after a cold night there is a huge amount of condensation on the grass around the rover in the form of dew, and after checking the sensors I could see that because my battery was so big and well insulated it could stay well under the dew point for many hours risking water condensation forming on it. So I am testing if I have enough power to keep the battery temperature above the dew point of the air around the rover. it should also keep all the electronics at either side of it above the dew point also.
It was outside for around a month, with its more sophisticated power management it can sleep most of the time while still waking up every hour to give weather and sensor readings before going back to sleep, because its consumption is so low it can still charge the battery with only a few watts of power.
What I thought was its controller crashing was actually a programming error where it ended up going to sleep for 3600 minutes rather than seconds, so if I had waited for 60 hours, it would have eventually woken up itself.
It was then put outside again but now its below freezing, and it eventually ran out of power. Due to the cold weather and high humidity there is a thick layer of frost on the solar panels that takes a long time to melt so it’s only making a few watts for a few hours It’s not enough to run the heater and keep the battery warm. Once the battery gets too cold the BMS low temperature protection will cut it off meaning it cannot charge until it warms up and there isn’t enough power to do that. The backup batteries still have power, but it’s immobile without the main battery and reduced to just reporting voltages and temperature once an hour until the ground defrosts and it goes back above 0c where it can start to charge.
Keeping the battery warm enough to charge during the day clearly is not possible when it’s too cold and dark as it can’t generate enough power to keep it above the 0c minimum charging temperature. so it might be a better idea to just keep it asleep and just skip charging altogether, since it’s using more power to heat up the battery than I’m getting from the solar panel. The batteries should be ok down to -20c for discharging, that isn’t a problem since it never gets below -10c here.
I had moved the temperature sensor to the esp32 PLC board but that means that I have to start up the PLC controller before it can read the battery temperature so i just moved the sensor back to the main esp32 S3 controller and added its own relay and 48v>5v regulator for the heater so it can monitor and heat the battery without starting up any other controllers.
My plan of printing a wind turbine got derailed due to a broken wire in my 3d printer causing the print to fail after a few hours. Even if it can only generate 2 watts, that would be enough to keep the battery above freezing so that will be getting printed as soon as the new wiring harness arrives.
Due to a lack of serial ports on the esp32 PLC controller, I had to get creative with mavlink telemetry. The only free port I had is the i2c port so i created a second i2c bus on the ESP32S3 set as a device connected to the esp32 PLC I2C, now it can send mavlink packets over i2c to the ESP32S3 controller that then forwards it over serial to the flight controller. its only one way at the moment, I haven’t figured out how to do bidirectional mavlink telemetry over i2c yet but it’s still useful for sensors. It might be possible to implement on the flight controller using LUA.
void MAVLINK_HB() {
uint8_t autopilot_type = MAV_AUTOPILOT_INVALID;
uint8_t system_mode = MAV_MODE_PREFLIGHT; ///< Booting up
uint32_t custom_mode = 1; ///< Custom mode, can be defined by user/adopter
uint8_t system_state = MAV_STATE_STANDBY; ///< System ready for flight
mavlink_message_t msg;
uint8_t buf[MAVLINK_MAX_PACKET_LEN];
int type = MAV_TYPE_ONBOARD_CONTROLLER;
mavlink_msg_heartbeat_pack(1, 192, &msg, type, autopilot_type, system_mode, custom_mode, system_state);
uint16_t len = mavlink_msg_to_send_buffer(buf, &msg);
Wire.beginTransmission(I2C_DEV_ADDR);
Wire.write(buf, len);
uint8_t error = Wire.endTransmission(true);
uint8_t bytesReceived = Wire.requestFrom(I2C_DEV_ADDR, 16);
if ((bool)bytesReceived) { //If received more than zero bytes
uint8_t temp[bytesReceived];
Wire.readBytes(temp, bytesReceived);
}
}
void onReceive(int len) {
//Serial.printf("onReceive[%d]: ", len);
while (Wire1.available()) {
mavlink_message_t msg;
mavlink_status_t status;
uint8_t c = Wire1.read();
Serial2.write(c);
// Serial.println(c);
//Serial.println("serial data");
//Get new message
if (mavlink_parse_char(MAVLINK_COMM_0, c, &msg, &status)) {
// Serial.println("mav packet");
//Handle new message from autopilot
switch (msg.msgid) {
case MAVLINK_MSG_ID_HEARTBEAT: // #0: Heartbeat
{
// Serial.println(" ");
mavlink_heartbeat_t hb;
mavlink_msg_heartbeat_decode(&msg, &hb);
if (hb.custom_mode == 1) {
// Serial.println("***************ESP32 868 HB******************");
PLCHBWATCH();
}
// Serial.print("custom Mode: ");
// Serial.println(hb.custom_mode);
// Serial.print("Type: ");
// Serial.println(hb.type);
//Serial.print("Autopilot: ");
// Serial.println(hb.autopilot);
//Serial.print("Base Mode: ");
// Serial.println(hb.base_mode);
// Serial.println("fcmodein ");
// Serial.println(fcmodein);
//Serial.print("System Status: ");
//Serial.println(hb.system_status);
// Serial.print("Mavlink Version: ");
// Serial.println(hb.mavlink_version);
//Serial.println();
}
break;
case MAVLINK_MSG_ID_NAMED_VALUE_FLOAT:
mavlink_named_value_float_t valueFloat;
mavlink_msg_named_value_float_decode(&msg, &valueFloat);
//Serial.print(" float: ");
// Serial.println(valueFloat.name);
// Serial.print(" ");
// Serial.println(valueFloat.value, 2);
messagei2c = (valueFloat.name);
messagedata = (valueFloat.value);
// Serial.print(" float: ");
//Serial.print(valueFloat.name);
// Serial.print(" ");
// Serial.println(valueFloat.value);
if (messagei2c = "Batt_Temp") {
batterytemp = messagedata;
}
break;
case MAVLINK_MSG_ID_STATUSTEXT: // #253
{
mavlink_statustext_t packet;
mavlink_msg_statustext_decode(&msg, &packet);
if (packet.severity > 0) {
// Serial.print("MAVMSG---------------------------------------------------");
//Serial.print(" severity: ");
// Serial.print(packet.severity);
Serial.print(" text: ");
Serial.println(packet.text);
mavmessage = (packet.text);
mavtextmessage();
/// Object();
}
}
break;
}
}
}
}
I have made a lot of progress with the Realtek amb82 mini AI camera board, I finally got multi-threading to work, this controller was essentially useless without it since the whole loop would just stop when it started communicating with any servers. Unfortunately there are no examples on how to make this work but I found in the OTA update sketch that they have implemented threading but no documentation exists about it. So just deleted code until all I was left with was the threading code then I could use that as the base for everything else. I managed to get the LLM example they provide for ChatGPT example to work with ollama on my server, this means i don’t need to pay for an expensive cloud service as it just runs on an nvidia GPU. For just asking “what do you see” even the smallest vision models work It’s when you want specific answers then you need to use the large models and running that every few seconds on a paid service gets very expensive.
my ollama and threading examples have now been added to the official repository.
Since I have JSON working on it now I can now send and receive data to my automation server like the esp32 does, now I can control what model it uses, the prompt and what to do with the data without re-flashing it. it also logs all the replies so it can now keep a log of what is happening. I will add a few prompts to log like “describe the weather” or “if there any obstructions ,what way should I turn to go around them” then an option for what it should do with the information, hold, rc override, guided mode or proximity. and probably something for what it should do after the action is over like return to auto mode or stay in guided.
With multi-threading now working I can move the LLM stuff onto its own thread freeing up the amb82 mini, Its a 500mhz cpu and 128mb of ddr2 so its got an insane amount of memory for an arduino
board so it should have plenty of power to run the lidar, gimbal and RGB LEDs, that gets rid of about 3 other micro controllers from the rover and it simplifies the wiring significantly.
I want to get the Rover talking again so when people come up to it can do things like read out the weather or other basic data like waypoints and flight modes without having to generate it each time on the sever and use power. I have recycled most of the talkie code I was using before but instead of synthesizing the audio it’s just playing a series of mp3 files on the amb82mini. This took some time as I needed to convert and find files for the talkie words. For the numbers and a lot of ardupilot related words I just reused the EDGETX voice pack, but there were a few non RC words like weather and ardupilot modes that needed custom words from a voice generator. It can do AI generated speech via googles TTS server for things like status messages and alerts but I’m working on getting that moved to either one of the onboard pi boards or my server.
mavlink messages are now logged on my server so i don’t have to dig through flight controller logs.
I finally got around to getting another LTE simcard for it so it will be able to go further than the garden. The current logic is that when it starts up it will attempt to use the Halow wifi connection and if that fails then it will switch to the 4G LTE connection, if that fails it will switch back to the wifi, this will repeat a few times then it will go back to sleep for 10 minutes then repeat until something connects.
I have still to get the BMS wired in properly with isolation so that power doesn’t go over the data lines when the battery negative cuts off.
still got a few weather sensors to get fitted, including wind and lighting.
still to make it chat, I have an esp32 doing chat with voice transcription and TTS but I want it to do it from its onboard pi or locally from my server rather than having to use online services.
I need to finish its control scripts for things like when to move, when to sleep, what to do when moving, I have separate scripts for each action but I need to add them all together into a useful series of steps that it can use to figure out what mode it should be in and its next move.
For a laugh I added a function that tracks the ISS via online data then set it up to send MAV_CMD_DO_SET_ROI_LOCATION using those coordinates and the ISS altitude of 400km to track it with the gimbal and hopefully get some photos of it going overhead.
void ISS() {
const char* apiURL = "http://api.open-notify.org/iss-now.json";
// Your location's latitude and longitude
const float myLatitude = roverlat;
const float myLongitude = roverlon;
const float alertRadiusMiles = 500;
if ((WiFi.status() == WL_CONNECTED)) {
HTTPClient http;
http.begin(apiURL);
int httpResponseCode = http.GET();
if (httpResponseCode == 200) {
String payload = http.getString();
DynamicJsonDocument doc(1024);
deserializeJson(doc, payload);
float issLatitude = doc["iss_position"]["latitude"];
float issLongitude = doc["iss_position"]["longitude"];
Serial.print("ISS Latitude: ");
Serial.println(issLatitude);
Serial.print("ISS Longitude: ");
Serial.println(issLongitude);
distanceMiles = calculateDistance(myLatitude, myLongitude, issLatitude, issLongitude);
Serial.print("Distance to ISS: ");
Serial.println(distanceMiles);
if (distanceMiles <= alertRadiusMiles) {
Serial.println("Alert! ISS is within 500 miles!");
// Add buzzer or LED notification code here
}
set_roi_location(issLatitude, issLongitude, 400000.0);
} else {
Serial.print("Error getting ISS data: ");
Serial.println(httpResponseCode);
}
http.end();
}
sendISSrange();
}
float calculateDistance(float lat1, float lon1, float lat2, float lon2) {
const float earthRadiusMiles = 3958.8;
float dLat = radians(lat2 - lat1);
float dLon = radians(lon2 - lon1);
float a = sin(dLat / 2) * sin(dLat / 2) + cos(radians(lat1)) * cos(radians(lat2)) * sin(dLon / 2) * sin(dLon / 2);
float c = 2 * atan2(sqrt(a), sqrt(1 - a));
return earthRadiusMiles * c;
}
void set_roi_location(float lat, float lon, float alt) {
Serial.print("ROI ");
Serial.println(lat);
Serial.print(" ");
Serial.println(lon);
mavlink_message_t msg;
uint8_t buf[MAVLINK_MAX_PACKET_LEN];
// Pack the COMMAND_LONG message
mavlink_msg_command_long_pack(
1, // System ID
191, // Component ID
&msg, // Message object
1, // Target System
1, // Target Component (0 for all components)
195, // Command ID 195
0, // Confirmation
0, // Param 1: ROI Mode
0, // Param 2: Empty
0, // Param 3: Empty
0, // Param 4: Empty
lat, // Param 5: Latitude
lon, // Param 6: Longitude
alt // Param 7: Altitude
);
// Copy the message to the buffer and send it via Serial
uint16_t len = mavlink_msg_to_send_buffer(buf, &msg);
Serial2.write(buf, len);
}
Updated with future predictions and azimuth elevation tracking of any norad ID satelite.
void fetchUpcomingPasses() {
Serial.println("\n--- SCANNING FOR NEXT PASS ---");
String url = "https://api.n2yo.com/rest/v1/satellite/radiopasses/" +
NORAD_ID + "/" + String(MY_LATITUDE, 2) + "/" + String(MY_LONGITUDE, 2) + "/" +
String(MY_ALTITUDE, 2) + "/" + String(SEARCH_DAYS) + "/" +
String(ELEVATION_1000_MILES) + "/&apiKey=" + API_KEY;
HTTPClient http;
http.begin(url);
if (http.GET() > 0) {
DynamicJsonDocument doc(8192);
deserializeJson(doc, http.getString());
if (doc["info"]["passescount"].as<int>() > 0) {
nextPassStartUnix = doc["passes"][0]["startUTC"];
int duration = doc["passes"][0]["duration"];
nextPassEndUnix = nextPassStartUnix + duration;
isDataLoaded = true;
struct tm ts = *localtime((time_t*)&nextPassStartUnix);
char buf[80];
strftime(buf, sizeof(buf), "%H:%M:%S", &ts);
Serial.printf("Next window starts at: %s for %d seconds.\n", buf, duration);
}
}
http.end();
}
void fetchLivePosition() {
// Fetch 1 second of real-time movement data
String url = "https://api.n2yo.com/rest/v1/satellite/positions/" +
NORAD_ID + "/" + String(MY_LATITUDE, 2) + "/" + String(MY_LONGITUDE, 2) + "/" +
String(MY_ALTITUDE, 2) + "/1/&apiKey=" + API_KEY;
HTTPClient http;
http.begin(url);
if (http.GET() > 0) {
DynamicJsonDocument doc(2048);
deserializeJson(doc, http.getString());
float az = doc["positions"][0]["azimuth"];
float el = doc["positions"][0]["elevation"];
String dir = doc["positions"][0]["azimuthDirection"];
Serial.printf("[TRACKING] Elevation: %.2f° | Direction: %.2f° (%s)\n", el, az, dir.c_str());
}
http.end();
}
void set_gimbal_orientation(float elevation, float azimuth) {
mavlink_message_t msg;
uint8_t buf[MAVLINK_MAX_PACKET_LEN];
mavlink_msg_command_long_pack(
1,
191,
&msg,
1,
1, // Target component (0 for broadcast)
205,
0, // Confirmation
elevation, // Param 1: Pitch
0.0, // Param 2: Roll
azimuth, // Param 3: Yaw
0, 0, 0, // Param 4, 5, 6 (unused)
MAV_MOUNT_MODE_MAVLINK_TARGETING // Param 7: Mode 2
);
uint16_t len = mavlink_msg_to_send_buffer(buf, &msg);
Serial2.write(buf, len);
}
What a fun catch up on your system @geofrancis Thanks for sharing the nitty gritty details…
It’s been outside for around 2 weeks and it’s been working ok, with it waking up every hour with all the sensors, sending readings then going to sleep it can just generate more power than its using. driving it just isnt viable with such a low amount of power.
Luckily temperatures have been mild so the battery heater hasn’t been needed, its well insulated now so its temperature is much more stable than outdoors.
Unfortunately I went to drive it today and its not starting, I’m not getting heartbeats from the hoverboard manager, This is the RP2040 that switches the hoverboard relays and converts the hoverboard telemetry to mavlink. It’s probably just a bad connection somewhere, but I have been looking to replace the hoverboard motor controllers for a while, they are cheap, but they are very inefficient, hot, unreliable and need a lot of supporting hardware to manage them like a bank of relays just to turn them all off and on.
Tthe best low-cost controllers I have found so far are the esp32 SimpleFOC boards. I wrote some code for them back in may to work like the hoverboard controller using mavlink Servo input but I never tested past making a wheel turn on a bench using open loop velocity mode, This was just sensorless mode as I couldn’t get the hall sensors to work properly as the motor would just jitter whenever they were connected, so I just sat the controllers to the side and worked on other problems.
I wasn’t in a great hurry to use them as I had managed to burn 2 controllers at that point, since you are controlling everything, if something switches wrong it will just short a motor and fry instantly, and due to the hall sensors glitching that would happen almost instantly and smoke the board. After the second one I realised that testing with a current limited power supply was essential to stop it self-destructing, The libraries were also very out dated and needed old versions of the esp-if to make them work. after getting nowhere with the hall sensors I just sat them to the side. The problem was sensorless mode used a lot of power and would slip so it just wasn’t practical to use on a rover that needed to be efficient. And there was almost no documentation on how to use hall sensors with simplefoc or documentation on the controllers on even simple things like what pins to use so it took a lot of time and effort to just get a wheel to turn.
With the recent glitches and overheating of the hoverboard controllers it made me go back and have a look at them to see if i could fix the hall sensor issue as its something that had been annoying me as a little unsolved mystery. So with a hoverboard wheel clamped to a desk and the controller hooked up to the PC and a power supply I started testing…
First thing was to just verify that the hall sensors were connected and changing properly when the wheel turned. so i made this very simple program to check if they were doing what they were supposed to be doing and everything looked fine.
void setup() {
Serial.begin(115200);
pinMode(18, INPUT_PULLUP);
pinMode(15, INPUT_PULLUP);
pinMode(19, INPUT_PULLUP);
}
void loop() {
int A = digitalRead(18);
int B = digitalRead(15);
int C = digitalRead(19);
Serial.println("");
Serial.println(A);
Serial.println(B);
Serial.println(C);
delay(100);
}
I next checked the SimpleFOC libraries and I was presently surprised to see they had finally been updated to work with the latest versions of esp32 so i dont need to mess around with old library versions.
So the next step was to check that the halls were detected with SimpleFOC and that the motor rotation is reported correctly This also checked as all the sensor were detected and rotation was being tracked accurately.
#include <SimpleFOC.h>
HallSensor sensor = HallSensor(18, 15, 19, 15);
void doA(){sensor.handleA();}
void doB(){sensor.handleB();}
void doC(){sensor.handleC();}
void setup() {
Serial.begin(115200);
sensor.pullup = Pullup::USE_INTERN;
sensor.init();
sensor.enableInterrupts(doA, doB, doC);
Serial.println("Sensor ready");
delay(1000);
}
void loop() {
sensor.update();
Serial.print(sensor.getAngle());
Serial.print("\t");
Serial.println(sensor.getVelocity());
delay(100);
}
when that worked the next step was to try the closed loop control examples but as soon as it starts up it just starts juddering.
#include <SimpleFOC.h>
BLDCMotor motor = BLDCMotor(15);
BLDCDriver3PWM driver = BLDCDriver3PWM(32, 33, 25, 15);
HallSensor sensor = HallSensor(18, 15, 19, 15);
void doA(){sensor.handleA();}
void doB(){sensor.handleB();}
void doC(){sensor.handleC();}
float target_voltage = 0;
void setup() {
Serial.begin(115200);
SimpleFOCDebug::enable(&Serial);
sensor.pullup = Pullup::USE_INTERN;
sensor.init();
sensor.enableInterrupts(doA, doB, doC);
motor.linkSensor(&sensor);
driver.voltage_power_supply = 24;
driver.init();
motor.linkDriver(&driver);
motor.voltage_sensor_align = 1;
motor.foc_modulation = FOCModulationType::SpaceVectorPWM;
motor.controller = MotionControlType::torque;
motor.useMonitoring(Serial);
motor.init();
motor.initFOC();
Serial.println(F("Motor ready."));
delay(1000);
}
void loop() {
motor.loopFOC();
motor.move(target_voltage);
}
after a lot of head scratching and adding counters and timers to the code so I could analyse the pulses coming back I realised that it was only on specific pins the signal was detected, I also verified this on another board, different software versions and control types also done it. But it only happened when the motor was active. So I wondered if the pull up on an esp32 isn’t enough for a hall sensor when all the noise from the motor starts. so some googling shows that an esp32 pullup is around 10k, so I attached 1k resistors to all the hall signal wires pulling them up to 3.3v and re tested and it now WORKS!
In hindsight, I should have added pull ups at the beginning but it’s not something I have seen before when reading the hall sensors using a micro controller but I have never ran the signals though a very noisy motor controller at the same time.
The latest version of the code is uploaded its still got a lot of work to do to it but its moving.
The motor control code is now working on Core 0 since it’s not using Wi-Fi. That will let it run at faster than 1khz and free up core 1 for mavlink and any extra sensors that i add..
I’m sending some telemetry back but there isnt a lot of data from the controllers. I have been thinking about adding some some temperature sensors since they also lack any kind of protections. A fuse is probably also a good idea.
They are already very efficient using only around a watt when idle but I’m going to set it up to sleep the controllers when idle to save a little more. the idea is to keep one always on and sleep the front and rear when the wheels arent turning. it can be put to a light sleep that it can wake from in a fraction of a second.
I will probably be bringing it indoors for a rebuild in the next week or so.
Aft
It turns out those controllers do have current monitoring, I managed to find the manufacturers github where they have shared demo code for the specific controller. It gives you the current of each individual phase as well as the battery current. There are a few different control types so im going to add the ability to change motor control strategies on the fly.
Voltage mode, this controls the voltage to the motor. where the more you give it, the faster the motor turns and the more torque you can get from it. The motors will all turn at roughly the same speed with the same voltage but it will allow slip, its efficient but with less precise control at low speeds compared to RPM mode. the voltage is controlled directly from the throttle with no pid so for basic plug and play this is probably the best all around mode to get started with.
RPM mode, is what I currently use, it makes handling very predictable as it turns each wheel an exact amount at an exact speed, this is most noticeable on skid steer rovers that require you to break traction before it turns where it tends to jerk when it starts to turn. this is what I use with the hoverboard controllers but the downside is that it can use a lot of power since you can end up with wheels fighting each other if one falls behind and it tries to push the others to catch up. It requires a PID loop to control so it will need some tuning to get it handing precisely.
Torque mode, this uses the throttle to control the motor current, its probably the hardest one to get driving properly as it’s the least direct. The motors will all push with the same force, but there is no control of the speed so if one breaks traction it will just wheel spin up to maximum RPM. This would be a good mode for faster vehicles like Akkerman where you want smooth acceleration and breaking and dont need to break traction or if you wanted to pull on something with a specific force.
There is also angle mode where you can set it to move to an exact position I thought about using it as a big servo but its only accurate to 4 degrees with hall sensors.
To control all 3 controllers from a single serial port I got creative with the serial wiring. basically i dont have enough serial ports to give each esp32 ESC its own port. so to get them all sharing a port I have split the TX pin from the flight controller to the RX of serial 1 on all 3 escs, this lets each controller receiver mavlink servo packets directly. but that isn’t ideal for sending data back as you could get collisions without some creative timing. so I have just routed the tx of serial 1 on ESC3 to the rx of serial 2 of ESC2, that then forwards it to the tx of serial 1 on ESC2 to the rx of serial 2 of ESC1, where its forwarded to the flight controller. I dont like to daisy-chain devices like this since one if one fails you loose the rest down the chain, that’s why I have done a hybrid where servo packet are split so they will always make it, but telemetry is daisy-chained as it’s not critical.
With the never ending quest for efficiency I have been looking at the pi boards I’m currently using, at the moment I’m using 2, one for my cameras and one just for telemetry, it’s not ideal. They only use a few watts but with 2 of them that adds up to more than I would like. I have been looking to find a low power alternative like esp32 but I haven’t found many viable solutions. The idea I looked into a while ago was to try and use a ESP32 as a VPN client to connect the rover to my local network over the internet. There are a few esp32 VPN repositories out there but most are unsupported and extremely out of date and required almost all the CPU flash and ram on the micro controller making it of limited use.
Since it was about a year ago since I had a look again to see what was out there and came across Husarnet, it’s an IOT VPN service its peer to peer so you don’t need a server but most importantly they have an up to date esp32 library. So combined with some code that converts TCP to serial and back again I now have a way of controlling ardupilot across the internet with just an esp32. It’s still really heavy for a micro controller and I have struggled to get the library running with the rest of my code without it running out of ram or memory. so i will probably just have it as a separate esp32 device connected to the flight controller rather than doing it on the main esp32 controller. its not ideal but it’s still less power than a pi. Husarnet is free for up to 5 devices at a time.
By replacing the motor controllers and the pi boards I think I can cut its driving power consumption by over half.
It also works for ESP32-Camera boards for integrated video and telemetry but its got a lot of latency on the video.