There’s so many things that could have caused that instability…
When you say T-Motor I immediately think of ESCs right out at the motors with extremely long signal and power wires, and cringe 
I would try to move ESCs inboard more and have longer motor wires than power wires. On the power wiring you’ll need extra capacitors right at the ESCs, probably at the distribution point too, and twisted wires.
For signal wiring you may need CAN nodes right near the ESCs so the PWM signal only travels a couple of inches, instead of several feet. Or I’d probably be using ESCs that do CAN comms natively.
If I was building something with 40inch props I’d be trying to simulate it in Realflight first. A month and few dollars spent doing that would be less than the cost of props. Even if it’s just to find out some slight variation in default parameters that would help with initial test flights.
Have a read of this: