Skip to main content

Advanced Networking: Secondary IP with a Different Gateway (Policy Based Routing)

When you add a secondary IP address that belongs to a different subnet than your primary IP, it usually comes with its own Gateway.

Linux, by default, supports only one Default Gateway. This leads to Asymmetric Routing issues: traffic enters via the secondary IP but tries to leave via the primary gateway, causing connections to fail.

To solve this on RHEL-based systems (AlmaLinux, Rocky, CentOS), we use Policy Based Routing via NetworkManager.

Scope:
This guide is for systems using NetworkManager  
For Ubuntu/Debian (Netplan), refer to the standard networking documentation.

Step 1: Taming Cloud-init (Crucial)

Before configuring anything, we must ensure Cloud-init does not overwrite our manual NetworkManager changes upon the next reboot.

  1. Log in as root via SSH.
  2. Run the following command to create a lock file:
echo "network: {config: disabled}" > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

This tells Cloud-init to stop managing networking, while keeping other functions (like SSH key injection) active.

Step 2: Gather Information

First, find the name of your active connection.

nmcli connection show

(Common names:  System eth0, eth0, ens3, or ens18).

Have the following details ready:

  • Connection Name: (e.g., "System eth0")
  • Secondary IP: (e.g., 192.168.102.10)
  • Secondary Gateway: (e.g., 192.168.102.1)
Subnet Mask Warning:
Even if you ordered a single IP, usually you must configure it with the subnet mask of the gateway (often /24).
Do not use /32, or the gateway will be unreachable.

Step 3: Configure Routing Rules

We will execute three commands to configure the interface. Replace the values in brackets with your data.

1. Add the Secondary IP:

# Syntax: nmcli con mod "NAME" +ipv4.addresses IP/MASK
nmcli con mod "System eth0" +ipv4.addresses 192.168.102.10/24

2. Create a Routing Policy Rule:
This tells Linux: "If traffic originates from 84.22.105.9, look at Routing Table 200".

# Syntax: ... "priority 100 from IP table 200"
nmcli con mod "System eth0" +ipv4.routing-rules "priority 100 from 192.168.102.10 table 200"

3. Add the Route to Table 200:
This tells Linux: "In Table 200, the default gateway is 84.22.105.1".

# Syntax: ... "0.0.0.0/0 GATEWAY table=200"
nmcli con mod "System eth0" +ipv4.routes "0.0.0.0/0 192.168.102.1 table=200"

Step 4: Apply Changes

Reload the connection to activate the new configuration. Warning: If you made a typo, you might lose connectivity.

nmcli con up "System eth0"

Step 5: Verification

Check if the separate routing table works correctly.

1. Check the table:

ip route show table 200

Output should show: default via 192.168.102.1 dev eth0...

2. Test outgoing traffic:
Force a ping specifically from your new IP address.

ping -I 192.168.102.10 -c 2 8.8.8.8

If the ping succeeds, your Policy Based Routing is active and reboot-proof.