ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
Advanced Networking
OS / ENVIRONMENT
n/a
SUMMARY
When two static NAT IPs exist on a network, every time a firewall rule is added or removed, duplicate CONNMARK rules are added to mangle/PREROUTING. Over time and on a busy network, this causes a significant number of duplicate rules and an increase in processing time.
I believe this is caused by the rules being defined with -I PREROUTING instead of -A PREROUTING, as the code appears to be trying to match them to the output of iptables-save. This may have been introduced in 40d7746.
I've fixed this locally by changing in configure.py:
self.fw.append(["mangle", "",
"-I PREROUTING -s %s/32 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff" %
rule["internal_ip"]])
to
self.fw.append(["mangle", "front",
"-A PREROUTING -s %s/32 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff" %
rule["internal_ip"]])
However, there are more instances of adding rules with -I in the file related to VPC and VPNs, which I don't have a way of testing at the moment.
STEPS TO REPRODUCE
- Create advanced network
- Deploy two VMs
- Enable static NAT for both IPs
- Create one Firewall rule
- Log onto VR and run
iptables -t mangle -nL PREROUTING --line-numbers
- Add a rule and repeat
For a simpler repeat repro, log onto the VR, then:
- Find a firewall rules file in /var/cache/cloud/processed/firewall_rules,json.XXXXX.gz and gunzip it
iptables -t mangle -F PREROUTING --line-numbers/opt/cloud/bin/configure /var/cache/cloud/processed/firewall_rules,json.XXXXXiptables -t mangle -F PREROUTING --line-numbers
EXPECTED RESULTS
No increase in the number of rules in PREROUTING
# iptables -t mangle -nL PREROUTING --line-numbers
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 MARK all -- 10.1.1.167 0.0.0.0/0 state NEW MARK set 0x2
2 CONNMARK all -- 10.1.1.167 0.0.0.0/0 state NEW CONNMARK save
3 MARK all -- 10.1.1.75 0.0.0.0/0 state NEW MARK set 0x2
4 CONNMARK all -- 10.1.1.75 0.0.0.0/0 state NEW CONNMARK save
5 FIREWALL_192.168.23.137 all -- 0.0.0.0/0 192.168.23.137
6 VPN_192.168.23.137 all -- 0.0.0.0/0 192.168.23.137
7 FIREWALL_192.168.23.136 all -- 0.0.0.0/0 192.168.23.136
8 VPN_192.168.23.136 all -- 0.0.0.0/0 192.168.23.136
9 FIREWALL_192.168.23.135 all -- 0.0.0.0/0 192.168.23.135
10 VPN_192.168.23.135 all -- 0.0.0.0/0 192.168.23.135
11 CONNMARK all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED CONNMARK restore
12 CONNMARK all -- 0.0.0.0/0 0.0.0.0/0 state NEW CONNMARK set 0x2
ACTUAL RESULTS
PREROUTING rules were increased by N, where N is the number of static NAT rules.
Error messages when running configure.py
# /opt/cloud/bin/configure.py /var/cache/cloud/processed/firewall_rules.json.2c3c52b5-9888-4fe4-a9d7-8b559fe98622
iptables v1.6.2: CONNMARK target: No operation specified
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.6.2: CONNMARK target: No operation specified
Try `iptables -h' or 'iptables --help' for more information.
Duplicate firewall rules added (lines 1 & 2 here)
# iptables -t mangle -nL PREROUTING --line-numbers
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 CONNMARK all -- 10.1.1.167 0.0.0.0/0 state NEW CONNMARK save
2 CONNMARK all -- 10.1.1.75 0.0.0.0/0 state NEW CONNMARK save
3 MARK all -- 10.1.1.167 0.0.0.0/0 state NEW MARK set 0x2
4 CONNMARK all -- 10.1.1.167 0.0.0.0/0 state NEW CONNMARK save
5 MARK all -- 10.1.1.75 0.0.0.0/0 state NEW MARK set 0x2
6 CONNMARK all -- 10.1.1.75 0.0.0.0/0 state NEW CONNMARK save
7 FIREWALL_192.168.23.137 all -- 0.0.0.0/0 192.168.23.137
8 VPN_192.168.23.137 all -- 0.0.0.0/0 192.168.23.137
9 FIREWALL_192.168.23.136 all -- 0.0.0.0/0 192.168.23.136
10 VPN_192.168.23.136 all -- 0.0.0.0/0 192.168.23.136
11 FIREWALL_192.168.23.135 all -- 0.0.0.0/0 192.168.23.135
12 VPN_192.168.23.135 all -- 0.0.0.0/0 192.168.23.135
13 CONNMARK all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED CONNMARK restore
14 CONNMARK all -- 0.0.0.0/0 0.0.0.0/0 state NEW CONNMARK set 0x2
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
Advanced Networking
OS / ENVIRONMENT
n/a
SUMMARY
When two static NAT IPs exist on a network, every time a firewall rule is added or removed, duplicate CONNMARK rules are added to mangle/PREROUTING. Over time and on a busy network, this causes a significant number of duplicate rules and an increase in processing time.
I believe this is caused by the rules being defined with
-I PREROUTINGinstead of-A PREROUTING, as the code appears to be trying to match them to the output of iptables-save. This may have been introduced in 40d7746.I've fixed this locally by changing in
configure.py:to
However, there are more instances of adding rules with
-Iin the file related to VPC and VPNs, which I don't have a way of testing at the moment.STEPS TO REPRODUCE
iptables -t mangle -nL PREROUTING --line-numbersFor a simpler repeat repro, log onto the VR, then:
iptables -t mangle -F PREROUTING --line-numbers/opt/cloud/bin/configure /var/cache/cloud/processed/firewall_rules,json.XXXXXiptables -t mangle -F PREROUTING --line-numbersEXPECTED RESULTS
No increase in the number of rules in PREROUTING
ACTUAL RESULTS
PREROUTING rules were increased by N, where N is the number of static NAT rules.
Error messages when running configure.py
Duplicate firewall rules added (lines 1 & 2 here)