Benchmark Test of the Twin Engine Mapping Planes

[Naterater]

This blog reflects my posts on RCGroups https://www.rcgroups.com/forums/showthread.php?3009817-Benchmark-Test-of-the-Twin-Engine-Electric-Mapping-Planes

I’m very interested in large area mapping/surveying at an affordable cost. To accomplish this goal, I’m beginning by researching the aircraft that can carry the most payload in the most efficient manner (subject to some constraints). There’s a ton of information online about new aircraft releases, and it’s great info. My goal is to test/verify it myself, and publish those results in a clean and accurate manner so that you can save yourself some time testing and tuning. I’d like to eliminate as many variables as possible and test the airframes on a level playing field. Ultimately I would like to sell the best setup as a ready-to-fly system to customers at a much lower price than others are charging (look at the eBee for example).

The planes I want to test are twin-engine foam planes under $500. In my opinion, they are the easiest and safest to hand launch for the amount of payload they carry. I’ve read, but not verified, that they are more yaw stable than flying wings. There have been quite a few released recently, and I’d like to benchmark their performance. The ones I want to test are as follows:

• MyTwinDream (MTD) - Benchmark
• Skywalker EVE-2000
• Believer
• X-UAV Clouds
• Skywalker Titan
• MFD Nimbus (maybe)

The independent variables I’d like to test on each aircraft are as follows:

• Payload/AUW
• Motor/prop combos
• Airspeeds
• CG locations

The dependent variables that I want to discover:

• Takeoff ability
• Cruise efficiency
• Stall and maximum level speed
• Glide speeds (l/d ratios)
• Stall characteristics
• Roll rates
• Load factors before accelerated stalls

I’ll also publish/maybe model the internal fuselage dimensions so you can see if your particular camera fits. This will also be combined with other advice/observations from other threads into a detailed but concise, unbiased, and accurate report. I’d also post the pixhawk parameter files for you to use for your maiden flights. I’m not planning on a build log because that’s already covered in detail on other threads.

I have experience building and testing the Skywalker EVE-2000, and it will be a benchmark for me. I had consistent 2-hour autonomous flights at 16 m/s on a 16Ah 6s li-po battery. I’m sure the performance could be better with many upgrades, but I was happy to not spend more than $600 on the setup.

Please let me know if you have any useful insight into this process, or if you have data or suggestions that I should be aware of.

[Naterater]

I’ve developed an equipment list that will be the same on each plane.

• Pixhawk 2.1
• Here M8N GPS module
• mRo Next-Gen MS5525 Airspeed Sensor
• Multistar 6s 12,000 (preferred) or 10,000 or 16,000 mAh battery pack
• Skywalker 2816-12 500kv motors
• Hobbywing Flyfun V5 40A ESC’s wiht active Freewheeling
• Mauch 100A Current sensor and BEC
• APC genuine 13x8E and EP props (to start - will change for tuning)
• Hobbywing RPM sensor for Motor RPM
• 16-gram HK digital coreless servo 1.4 kg*cm - .05 sec / 60deg
• YEP 20A HV SBEC (using 6v output for servos)
• 3DR 500mw SiK radio 900mhz
• FrSky X8R 16ch receiver

[Naterater]

Pre-Assembly Airframe Properties detailing weights and primary wing surface area (not tails).

[lucamax]

Skywalker EVE-2000, and it will be a benchmark for me. I had consistent 2-hour autonomous flights at 16 m/s on a 16Ah 6s li-po battery

With what payload ?

For mapping , I would guess that minimum cruise speed would be an important parameter since the mapping precision depends on plane speed during shoots , the slower the better , if you want accuracy.

[Naterater]

At the time, I used a Parrot Sequoia (1.2mp 4-band camera). It was very lightweight at maybe 100 grams.

I can see how minimum cruise speed would be a factor in the choice, however I do not think this will be an issue at the cruise speeds that we are expecting. These planes will go into the market that needs a very large area coverage area and not necessarily the highest accuracy.

For me, speed does not directly affect accuracy. Hotshoe feedback systems are able to attain a precise acquisition time, and by interpolating GPS positions, a very accurate position can be calculated. The speed does affect motion blur depending on altitude, however at high altitudes this is rarely a problem. A benchmark that I used for high detail image acquisition is included below.

Sensor: Sony APS-C 24mp (from Sony A6000)
Lens: 20mm prime E-mount
Flight Altitude: 400ft (122m)
GSD: 2.4cm
Cruise Speed: 17m/s
Shutter Speed: 1/800s (1.25ms)
Ground Blur/Smear: 2.1cm (<1 GSD)

Of course a lower blur would always be better, but the infomation that I found tells me that keeping the blur less than GSD is usually acceptable. If thousands of acres are sampled at that small GSD, it’s going to take a lot of time processing anyway. Wider lenses or sensors with less resolution would improve the smear. There are a lot of variables at play, but I think high accuracy is still achievable at the speeds we will be expecting.

[lucamax]

About precision with Pixhawk and hot shoe triggering , at best you should consider 200ms of precision in logs.

If you evaluate precisely stall speed I guess that it is possible to evaluate minimum cruise speed also , lets say , minimum speed 15-20% above stall speed and minimum cruise speed 10-20% above minimum speed depending on wind conditions.

About payload, mapping cameras usually goes from 400 to 1100 grams , so 900 grams , the weight of a Sony A7R with a small lens would be a fair choice.

I would suggest to make flights with payload and without to evaluated how the plane performances are affected by weight . I.e. choosing to fly with the same payload but a smaller battery.

[triise]

A tip; You´ll also have to consider the physical size of your payload.
We have previously used the MTD for this purpose, with a EOS-M and a TAU2 as payload. The EOS-M had to be mounted 90 deg off the flight direction due to the limited with of the plane. This of course reduces efficiency a lot, as you´ll have to fly more legs to get the same side lap.

Also, we got about 1:30 flight time with 11 Ah battery (4S).

We have also used sky walker X8´s in the past, and your thoughts about yaw-control are absolutely correct. That’s why we don’t use them anymore.

[kikislater]

Nice discussion here but why considering only twin engine ? Like xuav talon is very efficient in 4s, 6s is not needed.
Also you mention that you compare them to ebee but there is no comparison first between planes and wings, and between heavy weight like there is in your chart and lightweight ebee. Lightweight should be considered due to country regulation.

What could be more interesting IMO in a mapping platform is :

• landing like an ebee in a short space :
  - Reverse thrust (present in arduplane and working very well in my case),
  - deep stall at the end of reverse thrust (still need documentation),
• in flight ground detection (120-200meters) like ebee with lidar or other sensor (Not available with arduplane platform).

[Naterater]

About precision with Pixhawk and hot shoe triggering , at best you should consider 200ms of precision in logs.

I do not believe that this is true anymore. The code has improved, and PPK processing no longer ties the trigger with the actual shutter release. In addition, proper post-processing will interpolate camera position by precision timestamp between GPS log events (at 200ms or better). I believe the mission planner geotagging utility assigns a geotag for photos to the nearest GPS log event. That’s not what I will be doing.

I would suggest to make flights with payload and without to evaluated how the plane performances are affected by weight . I.e. choosing to fly with the same payload but a smaller battery.

That’s exactly what I’m doing! Great suggestion! I’ve noticed an increase in stall speed and a lot more induced drag at low airspeeds. The heavier payloads need to be flown faster to stay efficient.

Also you mention that you compare them to ebee but there is no comparison first between planes and wings, and between heavy weight like there is in your chart and lightweight ebee. Lightweight should be considered due to country regulation.

The only comparison to the eBee that I will make is the price and area coverage. Those are the main two factors in my target market. There are hundreds of ebee-sized planes that I don’t care to compete against. The landing argument is entirely differnt when you only have to land once (whereas an eBee would be landing 5-10x for battery swaps). It’s a different philosophy, but an eBee cannot carry a DSLR or heavy sensor at all. The talon can, but that’s more than what it was designed for. Right now there is a gap in the market for heavy-lift, high endurance ready-to-fly UAS for mapping. That’s what this project is geared towards. If you don’t need the capacity, then enjoy the incredibly long flight times. At least you’ll have the flexibility to.

About 6s - sure it is not needed here for these planes, but I plan to integrate VTOL. VTOL will require significant thrust and power, and thaat’s more easily accomplished on 6s. The tradeoff in efficiency between 4s and 6s is small.

Also, we got about 1:30 flight time with 11 Ah battery (4S).

Your endurance seems about right in line with my initial test results on the MTD (will be published very soon). I was able to cruise for a little over ~2 hours on a 10 Ah battery (6s), and I’ve been testing in the cold where it’s not so good for battery capacity.

[Hugues]

Great project.
In my opinion( and experience), there is no such thing as one best frame. There are lots of best frames for specific uses, contexts, payloads and climates. You won’t be able to compare frames universally and objectively since they all require different motors, props and ESC configurations to be flown in their optimal config (for example you won’t be able to mount 13" props on the XUAV Clouds. Max prop size is 9", unless you don’t care about breaking them at each landing. Cf my videos on building and flying the XUAC Cloud frame).

In short, I would start first by defining a use case, a context and a payload objective. Then only would I research the best frame for it.
Just a thought.

[Naterater]

@Hugues, thanks for the input. I think my use case is fairly well defined - long range/high payload mapping that can still be hand launched. You’re absolutely correct that certain airframes have limitations, especially in size. One size does not fit all. Maybe there’s a different way to think about this test - I’m finding the airframes with the worst efficiency. Do not use them unless you absolutely need to because of your payload characteristics.

The use case, to be more specific, is to map as much area with the Sony A6000 as possible. Broken down, that means I’m primarily looking to find the airframe with the longest range when carrying about 500 grams of payload. Since mapping can happen in the wind, airspeeeds and turning capability will play a factor in the decision as well.

Max prop size is 9", unless you don’t care about breaking them at each landing.

I see what you mean, but because the goal is primarily to map the most area, small prop size is a major limitation. With ESC’s that have braking capability, I’ve seen about a 95% of my landings don’t damage props that extend below the bottom of the aircraft. In addition, props could be treated like a consumable. They aren’t expensive compared to the additional value they add to the system by extending the range. Even if they broke on every flight, I would gladly pay $15 per flight to double my scan area. On initial results, that’s 1000 acres. Props are easy to change and having to change them doesn’t affect my objective enough for me to cap the max size. I also will be testing up to 14" props on the Clouds (the max size that prevents fuselage contact).

On a final note, prop size is a major variable in VTOL. These twins may use vectored thrust in the future if everything goes according to plan.

[Hugues]

About the motor/prop choice for the XUAV Clouds platform, I did a while back some bench tests that lead me to go for 9" props on 6S battery as best efficient combo : https://youtu.be/YAFJJnHktCo

[Naterater]

@Hugues, Thank you for the insights and video. I like how you documented your results. I strongly feel that bench tests of props used for aircraft do not provide accurate data for fixed-wing aircraft. I have spent quite a bit of time analyzing the computer modeled performance data of APC props on https://www.apcprop.com/technical-information/performance-data/.

My findings were similar to yours - lower pitched props give higher thrust per watt on the ground; HOWEVER, when the advance ratio (oncoming airspeed) is not still, higher pitched props have a significant efficiency boost. I’ll illustrate an example: 12x6 vs 12x12

0 airspeed (on the ground)
12x6 @ 7000 RPM = 171 watts and 3.2 lbf of thrust
12x12 @ 6000 RPM = 179 watts and 2.5 lbf of thrust - the higher pitched prop loses

36mph airspeed (in the air)
12x6 @ 6000 RPM = 75 watts and .785 lbf of thrust
12x12 @ 4000 RPM = 73 watts and .817 lbf of thrust - the higher pitched prop wins
9x4.5 @ 9000RPM = 83 watts and .816 lbf of thrust

To further back the data, you will be able to see the results posted here soon. They’re already up on the RC Groups page in my first post. 9" props may work well for takeoff and be a better all-around prop for your purpose. Because I am mainly focusing on range, the 9" and smaller props do not fit my purpose as well.

Unfortunately most motors are tested for multirotor applications, and manufacturers are publishing efficiency in grams per watt. Efficiency in airplane props with forward movement can truly be expressed in a unitless number related to the amount of work or power transferred to the air as thrust. A big reason that I use APC props is because of the published performance data - even if it’s computer-generated and theoretical.

[Hugues]

very interesting and quite logical (a higher pitch with faster moving air is physically analogue to a smaller pitch with slower moving air). You’re right my test objectives was to define the best combo (with constraints such as a max prop size) to hand launch without the need to use a catapult or any other launching device.

So your objective is endurance comparison between different airframes, motors, props, etc. Endurance optimum requires thus solving a multi variable non linear equation where main variables are:
-prop size
-prop pitch (and prop airfoil)
-RPM level that is minimally required to maintain your plane 's wished speed
-motor model and characteristics (Kv, weight, winding resistance, …) that runs at its optimum efficiency at that RPM (and with all other variables taken into account)
-Airframe aerodynamic characteristics (wing size, wing airfoil profile, CS factor, etc)
-All up weight
-Battery chemistry and behaviour when temperature increases during use
-I probably forget many more variables…like external real world variables : wind, temperature, air density, turbulences, …

[Naterater]

@Hugues

You’re right there are a LOT of variables. I’m trying to keep as many of them consistent (although that is nearly impossible) while changing the most important independent variables - All up weight, cruise airspeed, and prop choice. I’m also transferring the entire electrical and power control system from airframe to airframe to eliminate the equipment variables. After I find the best airframe for the job, then I’ll tackle the rest of those other ones. I also agree with you about hand launching being one of the constraints.

I don’t think a great, long, multivariable nonlinear equation is the best practical method to communicate the results to the audience here, so I’ll do my best to simplify the results while keeping them relevant. Another variable that may be significant is the ESC to add to your list. haha.

[Naterater]

On to my most recent testing - the TITAN:

BAD news - I’ve had three crashes - mostly user error.
#1: Attempted a launch with a significant crosswind probably about 7m/s (13mph). The wind basically pushed it into the ground about 3m away from the launch point. Broke the payload bay mounts on the fuselage, the props and my motor mounts. Root cause: user error of not launching into headwind

#2: One of the V-Tail sections fell off about 60 seconds after takeoff. I had done two or three gentle turns, and the air was incredible smooth. When I started the next turn, the tail blew off and the aircraft pitched and spun uncontrollably. It entered a semi-stable descent spiral and crashed belly-first. This cracked the tail off of the frame and broke, all else appeared fine. Root cause - VERY poor tail “latch” design. I may go into this into more detail about this in a later post, but I should’ve corrected this with some tape over the connection. It’s a push-snap-in design, and it’s too easy to remove.

#3: Wing delamination on takeoff causing the worst crash. I hand launched into the wind, and the aircraft almost hit the ground before continuing its automatic climb. After grabbing my transmitter, I looked at the wings in a slight U-shape but continuing to climb. The switch from auto mode to FBWA (flying level) caused enough of a jerk to completely separate the left wing. The result was a spectacular warlike bomber spiral dive into the ground causing catastrophic damage. Root cause - BSI Foam Cure glue AND a poor wing structure design. The carbon-fiber spars put all of the load onto the lower panel of the wing. Goop is the glue to use on this foam, and I will never go back. Even the tube glue that comes with the kit is better than the Foam Cure. I’m sure that crash #2 damaged the wing, and I didn’t notice the problem on pre-flight inspection.

I was very hopeful about the Titan, but the unreliable/poor design is really weighing on my decision to continue testing it. No other airframe in the test has the same design issues (except the EVE because it uses the same wing). The risks can be mitigated, but I’m disappointed to say the least. On a positive note, the flight controller and most of the internals are safe.

AIRWORTHINESS SAFETY NOTICE on SKYWALKER TITAN: secure your pop-in tail, and make sure your wing panel gluing is adequate.

The Pixhawk 2.1 has had no issues in these structural failures. Every other component has been doing its job.

[Coby]

Really like this approach. Very interested in your prop data as well. I hope to be teaching a course at a local community college this fall on basic design/build. I want to include a lot of flight testing and empirical data gathering in the syllabus. The focus is on real world use cases to sustain a business, so we need real world data to make decisions. If I get to that point we will focus on producing performance charts similar to below.

Max range should be very important for mapping and ag.

http://web.mit.edu/16.unified/www/FALL/thermodynamics/notes/node99.html

Maybe a better cruise speed…

http://www.eaa1000.av.org/technicl/perfspds/perfspds.htm

That cruise is based on fuel flow which all about reducing costs. For sUAS it will be about minimizing time spent in field to reduce labor costs. However, as pointed out above, best cruise could end up being set by how fast we can trigger the camera and get usable data. And all those curves shift with max gross weight and density altitude. Then throw in things like mean time between failure and time between overhaul, and perhaps shopping for cheap import parts is not such a value if they end up causing too much down time or have to be replaced often. Lots of operational and maintenance considerations.

[Hugues]

I feel your pain and frustration. What you experienced with the Titan, you will also experience on other foam airframes, like the XUAV clouds or the XUAV Talon (or with any other foam based frame which are very weak by design on all of their joints and hinges).
Here two videos I made about both above mentioned airframes explaining how to reinforce them, and not count on the factory system only:
XUAV Talon : https://youtu.be/HNHfCQU7ek4
XUAV Coulds : https://youtu.be/3CkiUq-qIeU

[Naterater]

After a few weeks of fretting from the previous crashes, I finally flew the Titan and gathered some data.

Bottom Line: The Titan is also a very efficient platform with an enormous payload bay, and massive prop clearance. Compared to the MTD as the benchmark, the Titan is about 10% less efficient. The volume within the fuselage is by far the largest, and it has some great features including a removable payload bay.

Test Setup:
Skywalker Titan plane with all options outlined in earlier post. Skywalker 510kv motors.
NOTE: V-Tails were semi-permanently taped on to the tail base to prevent separation as outlined in earlier post.
METAR: 152255Z AUTO 02008KT 10SM CLR 11/M03 A2992
Test Altitude: ~1300 feet MSL
Air Density: ~1.18kg/m^3
Density Altitude: ~1200 feet)
Air condition: Slightly turbulent for RC aircraft
Stall Speed: 13-14 m/s at 2300g payload, 14-15m/s at 3400g payload
Stall Characteristics: Slow stable descent with full up elevator. Stall does not really “break.” Was difficult to detect
Average Cruise Elevator: Neutral
Collection Method: ~2 minute straight track back and forth. Dataflash analysis of mAh used at start and end of periods with average voltage to determine power.

Test Notes:
Weights in the following charts are expressed as payload weight - Payload included Multistar Li-Po batteries and steel weights.
Payload is the same as expressed in the MTD results. Payload does not include the weight of motors, props, wiring, servos, autopilot, radios, etc. Empty weight was published in a previous post. All numbers are total numbers (not to be confused with per motor numbers). The numbers are also voltage-independent so that battery state of charge does not affect the data. The data was collected in one evening, and did not include every prop combination. Only the 13x6.5, 13x8, and 16x10 props were tested. The 16x10 props pulled too much current at 0 airspeed, so the throttle was limited. Hand launching into the wind was very easy. When released at about 30 degrees fully loaded, the plane would stay at about eye-level for 10m before gaining enough airspeed to climb at 30 degrees. Without a heavy payload, there is no drop at all. No running or significant pushing was necessary. I did break a few tips off of some of my props during landing only when the ESC brake was not completely engaged. Landings on hard soil were not a problem, and no props were broken. When the plane is light, the Titan glides a long ways before touchdown. Operations near trees are not advised without flaps. The height and energy control were managed by ardupilot plane 3.8.4 TECS, and the average error was less then 1m rms during slow cruise. I tested faster cruise speeds as some people are interested in higher speeds, but I ran into some pitch oscillations. The wings of the titan do flex up a lot more than most of the other airframes, and this isn’t helped by the spar that is about 2.5" too short. If you’re not careful, you could place the spar completely into one wing, and it would be about 5" short on one side. This was probably a contributing factor to my 2nd crash. The build quality of the fuselage is honestly disappointing. There were some very large gaps and imperfect fits all around. This didn’t really affect the performance though.

I’ll be testing the Titan further with a few more props and li-ion packs. I also want to put more payload into the plane. but I am afraid of breaking the spars. They already flex enough in steep turns. Attached is the first round of efficiency data.

[Naterater]

I tested the EVE as well, and the results are not surprising. Full report to come, but here are some charts to give you a comparison. I did not guess the correct CG of the EVE, and it was a bit nose-heavy when fully loaded. That it was why it carried about 250g more payload attached to the tail boom (for balance purposes). Because the EVE uses the same wing as the Titan, I only tested 2 different props that were efficient on the Titan and the MTD - the 12x12 and the 13x8. Be advised, the launch of the overweight EVE on a 35% charged battery and the 12x12 props was one of the worst successful takeoffs that I ever had. With no headwind and a 30-degree target pitch, the plane climbed only 10m in the first 6 seconds of flying. After pitching down, the airspeed got a little higher, allowing the plane to climb faster.

I’m 60% complete with this project’s initial data acquisition, and the next test subject will be the Believer.

As a reference, 100 watthour is about equivalent to 4,400mAh at 6s. or 6,600mAh at 4s. To convert an average lipo battery, watthr = 3.8 x #cells x mAh rating / 1000.