Asymmetric routing and the return path that betrays you

Asymmetric routing is not inherently broken. Packets take one path out and a different path back, and for plain routed traffic that is fine. It becomes a fault the moment something stateful sits in one of those paths.

Why the firewall drops it

A stateful firewall records outbound sessions so it can permit the replies. If the reply arrives at a different firewall, that device has no record of the session. It sees a TCP packet with ACK set and no matching state, and drops it. Correctly. It is doing exactly what it was built to do.

The symptom is maddening: the connection half-works, some sessions establish and others do not, and behaviour changes depending on which source address is used.

Finding it

Check the forward path and the return path separately, from both ends. Do not assume symmetry.

traceroute -n destination        # from here
# then, from the far end:
traceroute -n source

If the hop lists are not mirror images, you have asymmetry. Then ask whether any device in either path keeps state.

Common causes

Two uplinks with equal-cost defaults and no policy. A route learned from two protocols with different administrative distances at each end. A summarised route in one direction and a specific one in the other. Someone adding a static route to fix a different problem.

Fixing it properly

The clean fix is to make the paths symmetric: policy routing based on source, or adjusting metrics so both directions prefer the same link. The expedient fix is to disable stateful inspection for that traffic, and it is a bad habit, because you are turning off a security control to paper over a routing problem.

Before either, be sure the asymmetry is unintended. Some designs are deliberately asymmetric for good reasons, and in those the answer is to keep state out of the path entirely.