I am using apm2.6 quadcopter and having 3dr power module. My question is can i use 5200mah 40c 3s lipo battery
I think you can. I am a newbie. I am using 2200 and 4500 mah
I am also kinda new but I do not think that Capacity is as important as voltage and amps.
and voltage and amps depend on what power module you are using.
I am running 32,000mah so capacity should be no issue iirc 
I am also running 12s ( admittedly with a few bugs I am still trying to figure out) but that needed a power module with the bec disconnected and a separate 12s rated bec used instead.
this is on a pixhawk however but I think it would be the same for the APM.
This is correct, capacity is irrelevant as far as the APM is concerned. Voltage, however is down to the Power Module.
Be careful and make sure the Power Module can take the voltage.
Amps also, make sure the Power Module can carry the amps.
Power also need considered. Power is Voltage x Amps make sure the Power Module, and the wiring can handle the power.
3DR Power Module:-
The Power Module’s maximum input voltage is 18V. This is the maximum allowed by the on-board regulator. (4 cell LiPo max).
and maximum current of 90Amps, When used with an APM the full 90Amp current sensing range can be used, with the PX4/Pixhawks up to 60Amps can be measured.
Hobby King, HKPilot Mega 10s Power Module :-
MAX 45V i.e. 10s
MAX 90A
There are others out there also, I have seen 12s @150A