These can be the simple rules to follow and make life easy:
Motor 1 is left, motor 2 is right.
Forward distance is positive, and backwards distance is negative.
Easy assignment: lower number FC signals to motor 1, higher to 2.
WENCx_PINx signals are inputs (with interrupt capability on FC).
Output controls: for reverse capability you normally have DIRx/PWMx output signals, but also you can use a RC type pulse (ESC: >1500μs forward, <1500μs backwards) for saving FC pins. DIRx correspond to RELAYx and PWMx or RCx to SERVOx.
Wheel distances WENCx_POSx to vehicle center should be defined.
Think about pivoting for your case: the FC can go crazy.
For adjustment and checking I prefer to move the vehicle a fixed distance (say 10m), which can be done manually without slipping, and check distances of each wheel on MP (down to cm precision). You should see this:
See that left wheel (red) distance is negative, and right wheel distance (green) is positive (opposite for CW pivoting). For correct results, wheel distances to center must be correct (with correct signs).
I got a problem about encoder datas ı cant see WHEEL_DISTANCE parameter in mavlink inspector , ı am using 2 encoders (US Digital E2-500-375-IE-D-G-B encoder) ı checked wiring couple times and my parameters are like this EN1_A = Aux4 , EN1_B = Aux3 , EN2_A = Aux6 EN2_B = Aux5 all SERVOx_FUNCTION of them are GPIO and after all this setup Mavlink inspector doesnt show WHEEL_DISTANCE parameter. And by the way I checked encoders with osiloscop they are working.