USB interfering with UART2

Im attempting to write completely custom firmware to use on a APM 2.6. I want to use UART2 to connect to a XBee and still use the USB. Whenever i use send data to the XBee the APM disconnects from ym computer and when it reconnects a few seconds later the XBee stops working so its clear that the USB and UART2 are interfering with each other. My question is why is this happening and dose anyone know how to stop it? Im using the connections at the bottom of the board (not the GPS plug) to get to UART2 so as i understand it i don’t need to change the jumpers but i tried that as well just in case and no luck in either configuration.