I use the same setup, but for me it usually works fine. The only issue I have is that sometimes (rarely) I have to reboot to establish a telemtry link at first. But after that the link is rock solid.
I have reflashed the TXmods ESP module with Mav2PT, which gets updated much more often and seems to work best. You also get support from the developer in the discussion thread (unlike from RFDesign): Mavlink to FrSky S.Port Passthru Converter for LRS or PX4 Pro