Trim changes when flight mode changes

When in manual, I have been setting the proper trim for the aircraft I am using, however when switching into a new mode (like stabilize), the trim on the plane changes and is no longer what it should be. Is there any reason for this or any way that I can alter the trim to be dependent on the flight mode chosen?