NE-07: NAT Troubleshooting Scenario

Edge Router NAT Failure -- Post-Firmware Upgrade Recovery

8 Steps 35-45 min Scenario Lab

Incident Brief

Three critical tickets came in after last night's firmware upgrade on the Meridian Corp edge router (RTR-EDGE-01). The upgrade scrambled the NAT configuration and the network is in bad shape.

TICK-4401 -- Internal users (10.10.0.0/16) cannot reach the internet
TICK-4402 -- DMZ web server (172.16.50.10) unreachable from outside
TICK-4403 -- Site-to-site VPN tunnels to branch offices are failing

Your job: inspect the router output at each step, diagnose the NAT issue, and select the correct fix. Steps unlock sequentially -- each fix builds on the last.

TOPOLOGY -- Meridian Corp Edge Router (RTR-EDGE-01) =================================================================== INTERNET | [Gi0/0 -- 203.0.113.1/24] <-- WAN (NAT Outside) | RTR-EDGE-01 (IOS 15.9) | | | [Gi0/1] [Gi0/2] [Gi0/3] 10.10.0.1 172.16.50.1 10.20.0.1 /16 LAN /24 DMZ /24 VPN | | | Internal DMZ Web Srv VPN Tunnel Users 172.16.50.10 Endpoint =================================================================== NAT Pool "MERIDIAN": 203.0.113.10 - 203.0.113.11 (2 IPs) Engineering VLAN: 10.10.20.0/24 (85 users) Accounting VLAN: 10.10.10.0/24 (30 users)

Investigation Progress 0 / 8 steps

Empty Translation Table

First look: why is nothing being translated?

Evidence -- show ip nat translations

RTR-EDGE-01# show ip nat translations % No NAT translations currently active RTR-EDGE-01# show ip nat statistics Total active translations: 0 (0 static, 0 dynamic; 0 extended) Outside interfaces: Gi0/0 Inside interfaces: Hits: 0 Misses: 0 Expired translations: 0 Dynamic mappings: -- Inside Source access-list 10 pool MERIDIAN refcount 0

Q: The translation table is empty. What is the most likely root cause?

Look at the statistics: inside interfaces shows no interface designated. Also, with only 2 pool IPs and no overload keyword, PAT is not enabled -- the pool would exhaust instantly with any real traffic. But the deeper issue here is the missing PAT overload rule that was wiped by the firmware upgrade.

Missing Interface Designations

NAT needs to know which side is inside and which is outside

Evidence -- show running-config | section interface

RTR-EDGE-01# show running-config | section interface interface GigabitEthernet0/0 description WAN-UPLINK ip address 203.0.113.1 255.255.255.0 ip nat outside no shutdown ! interface GigabitEthernet0/1 description LAN-INTERNAL ip address 10.10.0.1 255.255.0.0 ! NOTE: no ip nat inside/outside designation no shutdown ! interface GigabitEthernet0/2 description DMZ-SEGMENT ip address 172.16.50.1 255.255.255.0 no shutdown ! interface GigabitEthernet0/3 description VPN-ENDPOINT ip address 10.20.0.1 255.255.255.0 ip nat inside no shutdown

Q: Internal users on Gi0/1 still can't reach the internet. What's missing?

NAT requires every participating interface to be designated as either ip nat inside or ip nat outside. Gi0/1 connects to the LAN (internal users) so it must be designated ip nat inside. Without this, the router does not know to translate packets arriving on that interface.

Incorrect Static NAT Entry

The DMZ web server's static NAT rule has the wrong address

Evidence -- show running-config | include ip nat

RTR-EDGE-01# show running-config | include ip nat ip nat inside source static 172.16.50.20 203.0.113.50 ip nat inside source list 10 pool MERIDIAN overload ip nat pool MERIDIAN 203.0.113.10 203.0.113.11 netmask 255.255.255.0 RTR-EDGE-01# show ip nat translations Pro Inside global Inside local Outside local Outside global --- 203.0.113.50 172.16.50.20 --- --- ! DMZ web server actual IP: 172.16.50.10

Q: Outside users still can't reach the DMZ web server. What needs to change?

The static NAT maps 172.16.50.20 to the public IP, but the actual DMZ web server lives at 172.16.50.10. The firmware upgrade corrupted the inside local address. The fix: remove the incorrect static entry and recreate it with the correct inside local IP 172.16.50.10.

NAT Pool Exhaustion

Engineering VLAN (85 users) vs a 2-IP pool

Evidence -- show ip nat pool & ACL

RTR-EDGE-01# show ip nat pool name MERIDIAN pool MERIDIAN: netmask 255.255.255.0 start 203.0.113.10 end 203.0.113.11 type generic, total addresses 2, allocated 2, misses 147 RTR-EDGE-01# show access-list 10 Standard IP access list 10 10 permit 10.10.0.0, wildcard bits 0.0.255.255 (328 matches) RTR-EDGE-01# show ip nat statistics | include Misses Hits: 412 Misses: 147

Q: Engineering users (10.10.20.0/24, 85 hosts) report intermittent internet access. Accounting (30 users) works fine. Why are Engineering users failing?

The pool has only 2 addresses and 147 misses indicate frequent exhaustion. Even though the config now has "overload," the pool is too small. Without PAT working correctly (which was fixed in Step 1), the 2 IPs map 1:1 to the first 2 hosts that request. The fix: expand the pool range (e.g., 203.0.113.10 - 203.0.113.30) or confirm the overload keyword is applied so all hosts share the pool via PAT.

VPN Traffic Being NATted

Site-to-site tunnels fail because NAT changes the source

Evidence -- VPN config & NAT ACL

RTR-EDGE-01# show crypto isakmp sa IPv4 Crypto ISAKMP SA dst src state conn-id status 203.0.113.200 203.0.113.1 MM_NO_STATE 1001 ACTIVE (no response) RTR-EDGE-01# show running-config | section crypto map crypto map BRANCH-VPN 10 ipsec-isakmp set peer 203.0.113.200 set transform-set AES256-SHA match address 150 RTR-EDGE-01# show access-list 150 Extended IP access list 150 10 permit ip 10.20.0.0 0.0.0.255 192.168.100.0 0.0.0.255 RTR-EDGE-01# show access-list 10 Standard IP access list 10 10 permit 10.10.0.0, wildcard bits 0.0.255.255 20 permit 10.20.0.0, wildcard bits 0.0.0.255

Q: VPN tunnels to the branch office (192.168.100.0/24) won't establish. The crypto peer shows MM_NO_STATE. What's the problem?

ACL 10 line 20 permits 10.20.0.0/24 -- the VPN subnet. This means VPN traffic gets NATted to the pool address before it hits the crypto map. The remote peer expects source 10.20.0.x but receives 203.0.113.x, so the IPsec SA never matches. Fix: add a deny 10.20.0.0 0.0.0.255 line BEFORE the permit in ACL 10 (NAT exemption), or use a route-map to exclude VPN-bound traffic.

DMZ Interface Missing NAT Designation

The DMZ segment has no NAT role assigned

Evidence -- interface Gi0/2 config & NAT stats

RTR-EDGE-01# show running-config interface Gi0/2 interface GigabitEthernet0/2 description DMZ-SEGMENT ip address 172.16.50.1 255.255.255.0 ! No ip nat inside or outside configured no shutdown RTR-EDGE-01# show ip nat statistics Total active translations: 14 (1 static, 13 dynamic; 13 extended) Outside interfaces: GigabitEthernet0/0 Inside interfaces: GigabitEthernet0/1, GigabitEthernet0/3 ! Gi0/2 not listed in either direction

Q: The static NAT entry for the web server exists (fixed in Step 3), but external users still can't reach 172.16.50.10. What's missing on Gi0/2?

The static NAT maps an inside local address (172.16.50.10) to an inside global address (203.0.113.50). For this to work, the router must see 172.16.50.10 arriving on an ip nat inside interface. Even though we call it the "DMZ," from NAT's perspective the web server is an inside host being published outward. Gi0/2 needs ip nat inside.

Missing HTTPS Port Forward

HTTP works but HTTPS traffic is dropped

Evidence -- port forwarding rules

RTR-EDGE-01# show running-config | include ip nat inside source static ip nat inside source static tcp 172.16.50.10 80 203.0.113.50 80 ! No entry for port 443 RTR-EDGE-01# show ip nat translations Pro Inside global Inside local Outside local Outside global tcp 203.0.113.50:80 172.16.50.10:80 --- --- --- 203.0.113.50 172.16.50.10 --- --- RTR-EDGE-01# show ip nat translations | count tcp.*443 Number of lines which match regexp = 0

Q: Customers report they can access http://meridian.com but https://meridian.com times out. Why can't HTTPS work?

The config shows a static port forward for TCP 80 only. HTTPS uses TCP 443 and requires its own static NAT entry: ip nat inside source static tcp 172.16.50.10 443 203.0.113.50 443. The firmware upgrade likely dropped this rule. HTTP and HTTPS are separate port mappings in NAT; one does not imply the other.

Aggressive NAT Timeout

Connections drop mid-session due to premature translation expiry

Evidence -- NAT timer config

RTR-EDGE-01# show ip nat translations verbose Pro Inside global Inside local Outside local Outside global tcp 203.0.113.10:44821 10.10.10.5:44821 151.101.1.69:443 151.101.1.69:443 create 00:00:03, use 00:00:01, timeout: 10, timing-out, Map-Id(In): 1, flags: extended, use_count: 0 tcp 203.0.113.10:50112 10.10.10.22:50112 172.217.14.99:80 172.217.14.99:80 create 00:00:08, use 00:00:02, timeout: 10, timing-out, Map-Id(In): 1, flags: extended, use_count: 0 RTR-EDGE-01# show running-config | include nat.*timeout ip nat translation timeout 10 ip nat translation tcp-timeout 10 ip nat translation udp-timeout 10

Q: Users report that web pages load partially then stall, file downloads abort after a few seconds, and SSH sessions freeze. All three NAT timeouts are set to 10 seconds. What's needed?

A 10-second NAT timeout means the router tears down the translation entry after only 10 seconds of inactivity. Normal TCP sessions (web browsing, downloads, SSH) pause longer than that routinely. Cisco defaults are: TCP timeout 86400 (24h), UDP timeout 300 (5m), general timeout 86400. Setting TCP to at least 3600 (1h) and UDP to 300 restores normal operation.

Incident Brief

All NAT Issues Resolved