Network Connectivity with a Cell Phone as a Mavlink Bridge

Hello everyone,

I’m seeking advice from developers for the following idea:

  1. Basic concept: I plan to mount a cell phone on a gimbal on my quadcopter, and set up an AWS OpenVPN server as an intermediary server to enable the cell phone and laptop to be in the same network via LTE. This will allow the cell phone to receive UDP messages from mission planner and forward them to the USB, as well as receive USB messages and forward them to the UDP network for mission planner.
    in essence, the cell phone will function similarly to an “ESP8622 WIFI telemetry”

  2. Purpose: This setup is intended for maritime use of multicopters, where internet connectivity is limited. By utilizing the cell phone’s modem capabilities, I hope to improve the connectivity and functionality of the quadcopter in the sea.
    I have already made mission planner plugins that display the ship’s position and data with an AIS antenna and transceiver. my dream is connecting all the maritime and drone datas in the LTE and mission planner environment.

  3. What I have done :
    I’m very beginner in network and udp things, so I maybe have some mistakes or misconception here.
    however, I already setup my aws server and network.
    I have also made FTDI - USB C connection between Pixhawk cube and Galaxy android phone.
    the only remaining task is to create android code for communicating. I have written the code without errors, but it does nothing. when I push “connect” button in mission planner, no heartbeat packet is received.
    However, the usb data is being written(as indicated by the blinking of FTDI module TX led) and I have also confirmed that UDP data is being send and received. (I checked network packet through the “wire shark” program)

  4. mission planner environment :
    connect option : udpcl, ip 10.8.0.10, port 1194

5.Android Code :

package com.example.cellphonemissioncomputerbegin6;

import android.app.PendingIntent;
import android.content.Context;
import android.content.Intent;
import android.hardware.usb.UsbDevice;
import android.hardware.usb.UsbDeviceConnection;
import android.hardware.usb.UsbEndpoint;
import android.hardware.usb.UsbInterface;
import android.hardware.usb.UsbManager;
import android.os.Bundle;
import android.util.Log;
import android.widget.TextView;

import androidx.appcompat.app.AppCompatActivity;

import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.InetAddress;
import java.util.Arrays;

import static android.content.ContentValues.TAG;

public class MainActivity extends AppCompatActivity {
    private static final String ACTION_USB_PERMISSION = "com.example.cellphonemissioncomputerbegin2.ACTION_USB_PERMISSION";
    private static final String LAPTOP_IP = "10.8.0.6";
    private static final int LAPTOP_PORT = 1194;
    private static final int TIMEOUT = 1000; //TIMEOUT FOR USB

    private UsbManager mUsbManager = null;
    private UsbDevice mUsbDevice = null;
    private UsbInterface mUsbInterface = null;
    private UsbEndpoint mUsbEndpointIn = null; 
    private UsbEndpoint mUsbEndpointOut = null;
    private UsbDeviceConnection mUsbConnection = null;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        final TextView telemetryTextView = findViewById(R.id.textView);

        new Thread(() -> {
            DatagramSocket socket = null;
            while(true)
            {
                try{
                    socket = new DatagramSocket(1194);
                    break;
                }
                catch(Exception e)
                {
                    Log.e(TAG,"Failed to create socket object");
                }
            }

            while(true)
            {
                try {
                    mUsbManager = (UsbManager) getSystemService(Context.USB_SERVICE);
                    mUsbDevice = mUsbManager.getDeviceList().values().iterator().next(); 
                }catch(Exception e)
                {
                    Log.e(TAG,"Failed to create USB object");
                    continue;
                }

                if (!mUsbManager.hasPermission(mUsbDevice)) {
                    PendingIntent pi = PendingIntent.getBroadcast(this, 0, new Intent(ACTION_USB_PERMISSION), PendingIntent.FLAG_IMMUTABLE);
                    mUsbManager.requestPermission(mUsbDevice, pi);
                    Log.e(TAG, "USB Permission Requesting..");
                } else {
                    Log.d(TAG,"USB permission already granted"); 
                }

                while(!mUsbManager.hasPermission(mUsbDevice)) {
                    Log.e(TAG, "USB Permission has not been granted");
                }

                try{
                    mUsbConnection = mUsbManager.openDevice(mUsbDevice);
                    mUsbInterface = mUsbDevice.getInterface(0);
                    mUsbEndpointIn = mUsbInterface.getEndpoint(0);
                }
                catch(Exception e) 
                {
                    if(mUsbConnection != null)
                    {
                        mUsbConnection.close();
                        if(mUsbInterface != null)
                        {
                            mUsbConnection.releaseInterface(mUsbInterface);
                        }
                    }
                    Log.e(TAG,"Failed to open USB connection");
                    continue;
                }

                //try-with-resources statement for automatic release of object
                try{//Bind the socket to port 1194

                    InetAddress serverAddress = InetAddress.getByName(LAPTOP_IP);
                    byte[] buffer_8 = new byte[8];
                    byte[] buffer_usb_from_pixhawk = new byte[mUsbEndpointIn.getMaxPacketSize()];


                    int numBytesRead = 0;
                    // get data from USB
                    numBytesRead = mUsbConnection.bulkTransfer(mUsbEndpointIn, buffer_usb_from_pixhawk, buffer_usb_from_pixhawk.length, TIMEOUT);
                    // if there is data on USB
                    if (numBytesRead > 0) {
                        byte[] mUsbData = Arrays.copyOf(buffer_usb_from_pixhawk, numBytesRead);
                        // Send the USB data over the network
                        DatagramPacket packet = new DatagramPacket(mUsbData, mUsbData.length, serverAddress, LAPTOP_PORT);
                        socket.send(packet);
                        Log.d(TAG, "byte data sent");
                        System.out.println("usb data : " + mUsbData);
                    }

                    if (mUsbConnection != null) {
                        mUsbConnection.close();
                        if (mUsbInterface != null) {
                            mUsbConnection.releaseInterface(mUsbInterface);
                        }
                    }

                    try{
                        mUsbConnection = mUsbManager.openDevice(mUsbDevice);
                        mUsbInterface = mUsbDevice.getInterface(0);
                        mUsbEndpointOut = mUsbInterface.getEndpoint(1);
                    }
                    catch(Exception e)
                    {
                        Log.e(TAG, "Exception in USB writing object get");
                    }
                    byte[] buffer_usb_to_pixhawk = new byte[mUsbEndpointOut.getMaxPacketSize()];

                    DatagramPacket packet = new DatagramPacket(buffer_usb_to_pixhawk, buffer_usb_to_pixhawk.length);
                    //DatagramPacket packet = new DatagramPacket(buffer_2, buffer_2.length,serverAddress, LAPTOP_PORT);
                    socket.receive(packet); 
                    String packetContent = new String(packet.getData());
                    Log.d(TAG,"packet message has been received successfully");
                    Log.d(TAG, packetContent);

                    int numBytesWritten = mUsbConnection.bulkTransfer(mUsbEndpointOut, packet.getData(), packet.getLength(), 1000); //timeout : ms
                    if (numBytesWritten < 0) {
                        Log.e(TAG, "Error writing UDP data to USB: " + numBytesWritten);
                    } else {
                        Log.d(TAG, "UDP data written to USB: " + numBytesWritten);
                    }
                    if (mUsbConnection != null) {
                        mUsbConnection.close();
                        if (mUsbInterface != null) {
                            mUsbConnection.releaseInterface(mUsbInterface);
                        }
                    }

                } catch (Exception e) {
                    if (mUsbConnection != null) {
                        mUsbConnection.close();
                        if (mUsbInterface != null) {
                            mUsbConnection.releaseInterface(mUsbInterface);
                        }
                    }
                    Log.e(TAG, "Error while executing main try statement at line " + e.getStackTrace()[0].getLineNumber() + ": " + e);
                }
            }//end of while(true)
        }).start(); //end of new Thread()

    }

    @Override
    protected void onDestroy() {
        super.onDestroy();
    }
}

andruav already does what your looking to do
https://cloud.ardupilot.org/

3 Likes

I appreciate the information you provided, but I am still interested in creating my own code to improve the functionality and expandability of my application, as well as to gain a deeper understanding of ArduPilot.

2 Likes

Based on andruav’s icon, I have the impression that it is currently a basic application in development, and it appears that my assumption is correct.

Regarding the app, there seems to be an issue with video connectivity. To address this problem, I have raised the issue on GitHub. :

to get video to start you usually have to press the video button twice, once to start the camera on the vehicle phone and a second time to start the stream to the gcs.

I am doing something similar.Except not using a cell phone but a dongle with a SBC. I don’t find understand why are you reinventing the wheel with a inefficient tool and OS that was designed for such operation.

Thank you. I once read the information on the Ardupilot Wiki documentation. Afterward, I attempted to press the “FPV” button twice while turning on the device’s cell phone “FPV” screen.

I am unsure if it is prohibited to press the “FPV” button on the UAV’s cell phone first.

Regardless, I tried to press the video button twice on the GCS app(or web client) while the FPV screen was turned off on the UAV’s cell phone. My intention was to pop up the FPV screen on the UAV’s cell phone first and then press the video button again. Unfortunately, with this method, I was unable to get the video.

I believe that SBCs are inferior to phones as they are too large, not suitable for commercial use, and lack good cameras. They also have high weight(including camera and routers) and consume a lot of electricity.

Regarding the effectiveness of wheels, I don’t agree that they are inefficient, particularly in the maritime environment. Have you tried using your dongle at sea? I believe that a cell phone modem may be better than your dongle.

I believe that Android is an expandable platform like Pixhawk, as it is open-source and offers excellent expandability features. For instance, Android can be used to enable Pixhawk RF telemetry, which is not available with other cell phone operating systems.

Additionally, Android is not a specially designed tool, as you previously mentioned. Instead, it is very versatile and can even be used to create robots.

have you tried this OS before?

If you think so. I don’t like my system running a hundred other things while i run my script. I prefer a clean and lean os. Having a mobile phone you are pretty much fixed with the sets of devices like camera. You must be new to the SBC world. There are tons of board which are more powerful than phone not to mention the flexibility to change what devices i want to go. Sometime devices has size limitation on a phone but not on SBC. I won’t go into details what my setup is but in a industry commercial point of view no phone is anywhere near what I can do with SBC.

For your set-up, I would take a look at solutions like https://uavmatrix.com/
there is more to it than just a connection, especially if you have limited connectivity

I don’t know if you have already tried Android OS because you haven’t answered. Android doesn’t run as many things as Linux does. Linux also runs many services, including keyboard or daemon services, just like Android. Do those services disturb your program on SBC? Definitely not. The same applies to Android services.

In Korea, you can request two cell phones and fast LTE service from your mobile carrier. I don’t know about other countries, but I’ve heard that some countries have low LTE coverage due to their land being too wide compared to the population. Anyway, I requested fast LTE service for a second cell phone, and I don’t use my existing phone. I’m using a special phone, so it’s very clean and basic. You can even get root permission and make it even more basic. So, I don’t think it has too many things to run, especially compared to Linux.

You mentioned that SBC has a wider choice. However, can you achieve a 12MP wide lens triple camera, under 200g, with zero current consumption for the main battery and less simple power lines, with your SBC? The Jetson Xavier alone weighs 280g, and adding cameras and trustworthy modems will increase the weight. Even an 8MP camera on the Jetson weighs 17.5g. And how much will it cost if you select all the good things on SBC? and I believe mobile phone will be developed more and more to high specifications.

I understand that many fields require higher camera specifications than cell phone cameras, and many devices can add value with SBCs. But in my field, I don’t need that. I’m planned to use this on sea delivery system monitoring. If I need different option of camera, I can buy a thermal-camera Android phone, as there is one available.

I don’t have to explain these things. Whether to choose something is my option. It’s okay if you recommend the system and discuss, but saying something is inferior or superior is different from making a recommendation. You have to be open to any selections. There are always pros and cons for each system, but you shouldn’t force others to use one system. I respect your using of SBC, but without video processing, SBC is useless. Purpose is important for anyone. Everyone is different in purpose.

And for your reference, I’ve already tried the Jetson board and Raspberry Pi from several years before. You assumed that I didn’t know about SBC (because you said “you must be new to the SBC world”), but several years ago, I was already familiar with it. I connected it to Pixhawk with rover firmware and copter firmware. It was my hobby. I already know the SBC system, and you don’t have to say “SBC,” which is hard to understand for beginners. I think “companion computer” is a more familiar expression for new users.

Thank you for your recommendation. I will definitely look into it.

There is nothing Inferior or superior here. Just fit for purpose. One designed for IOT and one for phone. We use LTE and 5G as well. Phone generally are designed to fit alot into a small box. I can have a much more powerful industrial dongle for my SBC with more sensitive external antenna. You can try that with a phone? Don’t think so.

I wasn’t aware that there is a powerful industrial dongle available for SBC, as I had assumed that SBC would simply use a general router or simple dongle. If such a dongle does exist, it would be extremely helpful. Thank you for broadening my perspective on this topic.

Could you recommend which LTE dongle or antenna you are currently using? or you can recommend industrial stage antenna or dongle.

Also, I forgot to mention that the cell phone comes with a high-quality waterproof case for free.

Designing the PCB and case for SBC could be challenging and result in high-end costs for end-users, particularly during the commercialization phase.

Therefore, I think utilizing existing industrial devices is a better approach for the initial stage of commercialization. Once the service is established and stabilized, then we can consider SBC.

However, this is just my opinion, and there are many ways to accomplish the commercialization of the service.

We need to balance “efficiency” and “completeness.”
It is difficult to achieve completeness in the early stages of commercialization.

Currently, industrial drones are too expensive, so I believe that using a cell phone is an affordable and efficient way to achieve our goals.

Industrial Drone are expensive for a reason. The components are not cheap to start with. . You can just go aliexpress there are a number and easier to purchase. Need some knowledge on RF to get the most suitable antenna for your needs. Good luck

and there is recycling second-hand market for cell-phones. if I can buy with cheap cell phone for test, it would be cost-effective. in development stage. than SBC.

could you recommend any specific antenna you are using? aliexpress one is not believable, as I think.

There are many manufacturers of Android-based devices. What are your thoughts on Herelink? It uses an Android-based operating system and has yielded good results.

Using Android as the operating system for a Ground Control Station (GCS) transmitter can significantly reduce costs compared to using a standalone OS and firmware. how much money you need for making such well-made GCS transmitter with other OS than android?

Android devices can be customized extensively if needed, when you have budget comparable to that of using a Single-Board Computer (SBC) commercialization.

because you said there is industry dongle that better than cell phone, I’m looking for specific antenna or dongle names and specifications that offer better performance than a typical cell phone,please tell me the exact device name, rather than vague recommendations, if you are using devices better than cell phone. and please tell me with ground. it it better on the sea? did you checked the distance boundary of the module, when you communicate at sea?
if it is not better at sea, why do you need sensitive LTE antenna? its speed is better than phone? why do you need that? I think speed and connectivity security is already enough in cell phone. I want to hear why do you think cell phone’s connectivity is not enough quality now.

what I actually need is to modify my code to bypass the Mavlink data to the phone. I don’t want to be limited to the methods you suggest, as it appears you haven’t worked with Android before and may not be familiar with the necessary code to communicate with Android.

I use a Samsung A21s, it was a fast dual SIM phone and if you check eBay you can get them really cheap if they have screen damage, I bought mine with a shattered screen, the LCD was ok but it was just hard to see. I only used the screen to install TeamViewer then everything else was done from my pc. The screen isn’t that important when its getting embedded.

1 Like