Problem

Changing the limits of a main axes will update the secondary axes' limits (at draw time), but changing the limits of a secondary axes has silently no effect (they get again overwritten by the (converted) main limits at draw time, in SecondaryAxes._set_lims). This should at least warn or raise.
Alternatively, secondary axes could set {x,y}lim_changed callbacks in both directions to fully keep limits in sync.

Proposed solution

Warn or raise on ineffective limits updates, or use callbacks to fully keep limits in sync.