RDO Havana Neutron Namespaces Troubleshooting for OVS&VLAN(GRE) Config

April 14, 2014

The  OpenStack Networking components are deployed on the Controller, Compute, and Network nodes in the following configuration:

In case of Two Node Development Cluster :-

Controller node: hosts the Neutron server service, which provides the networking API and communicates with and tracks the agents.

DHCP agent: spawns and controls dnsmasq processes to provide leases to instances. This agent also spawns neutron-ns-metadata-proxy processes as part of the metadata system.

Metadata agent: Provides a metadata proxy to the nova-api-metadata service. The neutron-ns-metadata-proxy direct traffic that they receive in their namespaces to the proxy.

OVS plugin agent: Controls OVS network bridges and routes between them via patch, tunnel, or tap without requiring an external OpenFlow controller.

L3 agent: performs L3 forwarding and NAT.

In case of Three Node or more ( several Compute Nodes) :-

Separate box hosts Neutron Server and all services mentioned above

Compute node: has an OVS plugin agent and openstack-nova-compute service.

Namespaces (View  Identifying and Troubleshooting Neutron Namespaces )

For each network you create, the Network node (or Controller node, if combined) will have a unique network namespace (netns) created by the DHCP and Metadata agents. The netns hosts an interface and IP addresses for dnsmasq and the neutron-ns-metadata-proxy. You can view the namespaces with the `ip netns list`  command, and can interact with the namespaces with the `ip netns exec namespace command`   command.

Every l2-agent/private network has an associated dhcp namespace and

Every l3-agent/router has an associated router namespace.

Network namespace starts with dhcp- followed by the ID of the network.

Router namespace starts with qrouter- followed by the ID of the router.

Source admin credentials and get network list

[root@dfw02 ~(keystone_admin)]$ neutron net-list

+————————————–+——+—————————————————–+

| id                                   | name | subnets                                             |

+————————————–+——+—————————————————–+

| 1eea88bb-4952-4aa4-9148-18b61c22d5b7 | int  | fa930cea-3d51-4cbe-a305-579f12aa53c0 10.0.0.0/24    |

| 426bb226-0ab9-440d-ba14-05634a17fb2b | int1 | 9e0d457b-c4c4-45cf-84e2-4ac7550f3b06 40.0.0.0/24    |

| 780ce2f3-2e6e-4881-bbac-857813f9a8e0 | ext  | f30e5a16-a055-4388-a6ea-91ee142efc3d 192.168.1.0/24 |

+————————————–+——+—————————————————–+

Using command `ip netns list` run following commands to get tenants

qdhcp-* names

[root@dfw02 ~(keystone_admin)]$ ip netns list | grep 1eea88bb-4952-4aa4-9148-18b61c22d5b7

qdhcp-1eea88bb-4952-4aa4-9148-18b61c22d5b7

[root@dfw02 ~(keystone_admin)]$ ip netns list | grep 426bb226-0ab9-440d-ba14-05634a17fb2b

qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b

Check tenants Namespace via getting IP and ping this IP inside namespaces

[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b ifconfig

lo: flags=73  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10
loop  txqueuelen 0  (Local Loopback)
RX packets 35  bytes 4416 (4.3 KiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 35  bytes 4416 (4.3 KiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ns-343b0090-24: flags=4163  mtu 1500
inet 40.0.0.3  netmask 255.255.255.0  broadcast 40.0.0.255

inet6 fe80::f816:3eff:fe01:8b55  prefixlen 64  scopeid 0x20
ether fa:16:3e:01:8b:55  txqueuelen 1000  (Ethernet)
RX packets 3251  bytes 386284 (377.2 KiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 1774  bytes 344082 (336.0 KiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b ping  -c 3 40.0.0.3
PING 40.0.0.3 (40.0.0.3) 56(84) bytes of data.
64 bytes from 40.0.0.3: icmp_seq=1 ttl=64 time=0.041 ms
64 bytes from 40.0.0.3: icmp_seq=2 ttl=64 time=0.035 ms
64 bytes from 40.0.0.3: icmp_seq=3 ttl=64 time=0.034 ms

— 40.0.0.3 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.034/0.036/0.041/0.007 ms

Now verify that we have a copy of dnsmasq process to support every tenants namespace

[root@dfw02 ~(keystone_admin)]$ ps -aux | grep dhcp

neutron   2320  0.3  0.3 263908 30696 ?        Ss   08:18   2:14 /usr/bin/python /usr/bin/neutron-dhcp-agent –config-file /usr/share/neutron/neutron-dist.conf –config-file /etc/neutron/neutron.conf –config-file /etc/neutron/dhcp_agent.ini –log-file /var/log/neutron/dhcp-agent.log

nobody    3529  0.0  0.0  15532   832 ?        S    08:20   0:00 dnsmasq –no-hosts –no-resolv –strict-order –bind-interfaces –interface=ns-40dd712c-e4 –except-interface=lo –pid-file=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/pid –dhcp-hostsfile=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/host –dhcp-optsfile=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/opts –leasefile-ro –dhcp-range=set:tag0,10.0.0.0,static,120s –dhcp-lease-max=256 –conf-file=/etc/neutron/dnsmasq.conf –domain=openstacklocal

nobody    3530  0.0  0.0  15532   944 ?        S    08:20   0:00 dnsmasq –no-hosts –no-resolv –strict-order –bind-interfaces –interface=ns-343b0090-24 –except-interface=lo –pid-file=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/pid –dhcp-hostsfile=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/host –dhcp-optsfile=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/opts –leasefile-ro –dhcp-range=set:tag0,40.0.0.0,static,120s –dhcp-lease-max=256 –conf-file=/etc/neutron/dnsmasq.conf –domain=openstacklocal

List interfaces inside dhcp namespace

[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever

2: ns-343b0090-24: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:01:8b:55 brd ff:ff:ff:ff:ff:ff
inet 40.0.0.3/24 brd 40.0.0.255 scope global ns-343b0090-24
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe01:8b55/64 scope link
valid_lft forever preferred_lft forever

(A)( From the instance to a router)

Check routing inside dhcp namespace

[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b  ip r

default via 40.0.0.1 dev ns-343b0090-24

40.0.0.0/24 dev ns-343b0090-24  proto kernel  scope link  src 40.0.0.3

Check routing inside the router namespace

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 ip r

default via 192.168.1.1 dev qg-9c090153-08

40.0.0.0/24 dev qr-e031db6b-d0  proto kernel  scope link  src 40.0.0.1

192.168.1.0/24 dev qg-9c090153-08  proto kernel  scope link  src 192.168.1.114

Get routers list  via similar grep and network-id to obtain Routers Namespaces

[root@dfw02 ~(keystone_admin)]$ neutron router-list

+————————————–+———+—————————————————————————–+

| id                                   | name    | external_gateway_info                                                       |

+————————————–+———+—————————————————————————–+

| 86b3008c-297f-4301-9bdc-766b839785f1 | router2 | {“network_id”: “780ce2f3-2e6e-4881-bbac-857813f9a8e0″, “enable_snat”: true} |

| bf360d81-79fb-4636-8241-0a843f228fc8 | router1 | {“network_id”: “780ce2f3-2e6e-4881-bbac-857813f9a8e0″, “enable_snat”: true} |

+————————————–+———+—————————————————————————–+

Now get qrouter-* namespaces via `ip netns list` command :-

[root@dfw02 ~(keystone_admin)]$ ip netns list | grep 86b3008c-297f-4301-9bdc-766b839785f1
qrouter-86b3008c-297f-4301-9bdc-766b839785f1

[root@dfw02 ~(keystone_admin)]$ ip netns list | grep  bf360d81-79fb-4636-8241-0a843f228fc8
qrouter-bf360d81-79fb-4636-8241-0a843f228fc8

Now verify L3 forwarding  & NAT via command  `iptables -L -t nat` inside router namespace and check  routing   port 80 for 169.254.169.254 to the RDO Havana Controller’s ( in my configuration running Neutron Server Service along with all agents) host at metadata port 8700

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 iptables -L -t nat

Chain PREROUTING (policy ACCEPT)

target     prot opt source               destination

neutron-l3-agent-PREROUTING  all  —  anywhere             anywhere

Chain INPUT (policy ACCEPT)

target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination

neutron-l3-agent-OUTPUT  all  —  anywhere             anywhere

Chain POSTROUTING (policy ACCEPT)

target     prot opt source               destination

neutron-l3-agent-POSTROUTING  all  —  anywhere             anywhere

neutron-postrouting-bottom  all  —  anywhere             anywhere

Chain neutron-l3-agent-OUTPUT (1 references)

target     prot opt source               destination

DNAT       all  —  anywhere             dfw02.localdomain    to:40.0.0.2

DNAT       all  —  anywhere             dfw02.localdomain    to:40.0.0.6

DNAT       all  —  anywhere             dfw02.localdomain    to:40.0.0.5

Chain neutron-l3-agent-POSTROUTING (1 references)

target     prot opt source               destination

ACCEPT     all  —  anywhere             anywhere             ! ctstate DNAT

Chain neutron-l3-agent-PREROUTING (1 references)

target     prot opt source               destination

REDIRECT   tcp  —  anywhere             169.254.169.254      tcp dpt:http redir ports 8700

DNAT       all  —  anywhere             dfw02.localdomain    to:40.0.0.2

DNAT       all  —  anywhere             dfw02.localdomain    to:40.0.0.6

DNAT       all  —  anywhere             dfw02.localdomain    to:40.0.0.5

Chain neutron-l3-agent-float-snat (1 references)

target     prot opt source               destination

SNAT       all  —  40.0.0.2             anywhere             to:192.168.1.107

SNAT       all  —  40.0.0.6             anywhere             to:192.168.1.104

SNAT       all  —  40.0.0.5             anywhere             to:192.168.1.110

Chain neutron-l3-agent-snat (1 references)

target     prot opt source               destination

neutron-l3-agent-float-snat  all  —  anywhere             anywhere

SNAT       all  —  40.0.0.0/24          anywhere             to:192.168.1.114

Chain neutron-postrouting-bottom (1 references)

target     prot opt source               destination

neutron-l3-agent-snat  all  —  anywhere             anywhere

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-bf360d81-79fb-4636-8241-0a843f228fc8  iptables -L -t nat

Chain PREROUTING (policy ACCEPT)

target     prot opt source               destination

neutron-l3-agent-PREROUTING  all  —  anywhere             anywhere

Chain INPUT (policy ACCEPT)

target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination

neutron-l3-agent-OUTPUT  all  —  anywhere             anywhere

Chain POSTROUTING (policy ACCEPT)

target     prot opt source               destination

neutron-l3-agent-POSTROUTING  all  —  anywhere             anywhere

neutron-postrouting-bottom  all  —  anywhere             anywhere

Chain neutron-l3-agent-OUTPUT (1 references)

target     prot opt source               destination

DNAT       all  —  anywhere             dfw02.localdomain    to:10.0.0.2

Chain neutron-l3-agent-POSTROUTING (1 references)

target     prot opt source               destination

ACCEPT     all  —  anywhere             anywhere             ! ctstate DNAT

Chain neutron-l3-agent-PREROUTING (1 references)

target     prot opt source               destination

REDIRECT   tcp  —  anywhere             169.254.169.254      tcp dpt:http redir ports 8700

DNAT       all  —  anywhere             dfw02.localdomain    to:10.0.0.2

Chain neutron-l3-agent-float-snat (1 references)

target     prot opt source               destination

SNAT       all  —  10.0.0.2             anywhere             to:192.168.1.103

Chain neutron-l3-agent-snat (1 references)

target     prot opt source               destination

neutron-l3-agent-float-snat  all  —  anywhere             anywhere

SNAT       all  —  10.0.0.0/24          anywhere             to:192.168.1.100

Chain neutron-postrouting-bottom (1 references)

target     prot opt source               destination

neutron-l3-agent-snat  all  —  anywhere             anywhere

(B) ( through a NAT rule in the router namespace)

Check the NAT table

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 iptables -t nat -S

-P PREROUTING ACCEPT

-P INPUT ACCEPT

-P OUTPUT ACCEPT

-P POSTROUTING ACCEPT

-N neutron-l3-agent-OUTPUT

-N neutron-l3-agent-POSTROUTING

-N neutron-l3-agent-PREROUTING

-N neutron-l3-agent-float-snat

-N neutron-l3-agent-snat

-N neutron-postrouting-bottom

-A PREROUTING -j neutron-l3-agent-PREROUTING

-A OUTPUT -j neutron-l3-agent-OUTPUT

-A POSTROUTING -j neutron-l3-agent-POSTROUTING

-A POSTROUTING -j neutron-postrouting-bottom

-A neutron-l3-agent-OUTPUT -d 192.168.1.112/32 -j DNAT –to-destination 40.0.0.2

-A neutron-l3-agent-OUTPUT -d 192.168.1.113/32 -j DNAT –to-destination 40.0.0.4

-A neutron-l3-agent-OUTPUT -d 192.168.1.104/32 -j DNAT –to-destination 40.0.0.6

-A neutron-l3-agent-OUTPUT -d 192.168.1.110/32 -j DNAT –to-destination 40.0.0.5

-A neutron-l3-agent-POSTROUTING ! -i qg-9c090153-08 ! -o qg-9c090153-08 -m conntrack ! –ctstate DNAT -j ACCEPT

-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp –dport 80 -j REDIRECT –to-ports 8700

-A neutron-l3-agent-PREROUTING -d 192.168.1.112/32 -j DNAT –to-destination 40.0.0.2

-A neutron-l3-agent-PREROUTING -d 192.168.1.113/32 -j DNAT –to-destination 40.0.0.4

-A neutron-l3-agent-PREROUTING -d 192.168.1.104/32 -j DNAT –to-destination 40.0.0.6

-A neutron-l3-agent-PREROUTING -d 192.168.1.110/32 -j DNAT –to-destination 40.0.0.5

-A neutron-l3-agent-float-snat -s 40.0.0.2/32 -j SNAT –to-source 192.168.1.112

-A neutron-l3-agent-float-snat -s 40.0.0.4/32 -j SNAT –to-source 192.168.1.113

-A neutron-l3-agent-float-snat -s 40.0.0.6/32 -j SNAT –to-source 192.168.1.104

-A neutron-l3-agent-float-snat -s 40.0.0.5/32 -j SNAT –to-source 192.168.1.110

-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat

-A neutron-l3-agent-snat -s 40.0.0.0/24 -j SNAT –to-source 192.168.1.114

-A neutron-postrouting-bottom -j neutron-l3-agent-snat

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-bf360d81-79fb-4636-8241-0a843f228fc8 iptables -t nat -S

-P PREROUTING ACCEPT

-P INPUT ACCEPT

-P OUTPUT ACCEPT

-P POSTROUTING ACCEPT

-N neutron-l3-agent-OUTPUT

-N neutron-l3-agent-POSTROUTING

-N neutron-l3-agent-PREROUTING

-N neutron-l3-agent-float-snat

-N neutron-l3-agent-snat

-N neutron-postrouting-bottom

-A PREROUTING -j neutron-l3-agent-PREROUTING

-A OUTPUT -j neutron-l3-agent-OUTPUT

-A POSTROUTING -j neutron-l3-agent-POSTROUTING

-A POSTROUTING -j neutron-postrouting-bottom

-A neutron-l3-agent-OUTPUT -d 192.168.1.103/32 -j DNAT –to-destination 10.0.0.2

-A neutron-l3-agent-POSTROUTING ! -i qg-54e34740-87 ! -o qg-54e34740-87 -m conntrack ! –ctstate DNAT -j ACCEPT

-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp –dport 80 -j REDIRECT –to-ports 8700

-A neutron-l3-agent-PREROUTING -d 192.168.1.103/32 -j DNAT –to-destination 10.0.0.2

-A neutron-l3-agent-float-snat -s 10.0.0.2/32 -j SNAT –to-source 192.168.1.103

-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat

-A neutron-l3-agent-snat -s 10.0.0.0/24 -j SNAT –to-source 192.168.1.100

-A neutron-postrouting-bottom -j neutron-l3-agent-snat

Ping to verify network connections

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

64 bytes from 8.8.8.8: icmp_seq=1 ttl=47 time=42.6 ms

64 bytes from 8.8.8.8: icmp_seq=2 ttl=47 time=40.8 ms

64 bytes from 8.8.8.8: icmp_seq=3 ttl=47 time=41.6 ms

64 bytes from 8.8.8.8: icmp_seq=4 ttl=47 time=41.0 ms

Verifying  service listening at 8700 port  inside routers namespaces 

output seems like this :-

(C) (to an instance of the neutron-ns-metadata-proxy)

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1  netstat -lntp | grep 8700

tcp        0      0 0.0.0.0:8700            0.0.0.0:*               LISTEN      4946/python

Check process with pid 4946

[root@dfw02 ~(keystone_admin)]$ ps -ef | grep 4946

root      4946     1  0 08:58 ?        00:00:00 /usr/bin/python /bin/neutron-ns-metadata-proxy –pid_file=/var/lib/neutron/external/pids/86b3008c-297f-4301-9bdc-766b839785f1.pid –metadata_proxy_socket=/var/lib/neutron/metadata_proxy –router_id=86b3008c-297f-4301-9bdc-766b839785f1 –state_path=/var/lib/neutron –metadata_port=8700 –verbose –log-file=neutron-ns-metadata-proxy-86b3008c-297f-4301-9bdc-766b839785f1.log –log-dir=/var/log/neutron

root     10396 11489  0 16:33 pts/3    00:00:00 grep –color=auto 4946

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-bf360d81-79fb-4636-8241-0a843f228fc8  netstat -lntp | grep 8700

tcp        0      0 0.0.0.0:8700            0.0.0.0:*               LISTEN      4746/python

Check process with pid 4746

[root@dfw02 ~(keystone_admin)]$ ps -ef | grep 4746

root      4746     1  0 08:58 ?        00:00:00 /usr/bin/python /bin/neutron-ns-metadata-proxy –pid_file=/var/lib/neutron/external/pids/bf360d81-79fb-4636-8241-0a843f228fc8.pid –metadata_proxy_socket=/var/lib/neutron/metadata_proxy –router_id=bf360d81-79fb-4636-8241-0a843f228fc8 –state_path=/var/lib/neutron –metadata_port=8700 –verbose –log-file=neutron-ns-metadata-proxy-bf360d81-79fb-4636-8241-0a843f228fc8.log –log-dir=/var/log/neutron

Now run following commands inside routers namespaces to check status of neutron-metadata port :-

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1  netstat -na

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State

tcp        0      0 0.0.0.0:8700            0.0.0.0:*               LISTEN

Active UNIX domain sockets (servers and established)

Proto RefCnt Flags       Type       State         I-Node   Path

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-bf360d81-79fb-4636-8241-0a843f228fc8  netstat -na

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State

tcp        0      0 0.0.0.0:8700            0.0.0.0:*               LISTEN

Active UNIX domain sockets (servers and established)

Proto RefCnt Flags       Type       State         I-Node   Path

Outside routers namespace it would look like

(D) (to the actual Nova metadata service)

[root@dfw02 ~(keystone_admin)]$ netstat -lntp | grep 8700

tcp        0      0 0.0.0.0:8700            0.0.0.0:*               LISTEN      2746/python

Check process with pid  2746

[root@dfw02 ~(keystone_admin)]$ ps -ef | grep 2746

nova      2746     1  0 08:57 ?        00:02:31 /usr/bin/python /usr/bin/nova-api –logfile /var/log/nova/api.log

nova      2830  2746  0 08:57 ?        00:00:00 /usr/bin/python /usr/bin/nova-api –logfile /var/log/nova/api.log

nova      2851  2746  0 08:57 ?        00:00:10 /usr/bin/python /usr/bin/nova-api –logfile /var/log/nova/api.log

nova      2858  2746  0 08:57 ?        00:00:02 /usr/bin/python /usr/bin/nova-api –logfile /var/log/nova/api.log

root      9976 11489  0 16:31 pts/3    00:00:00 grep –color=auto 2746

So , we actually verified statement from Direct access to Nova metadata

in an environment running Neutron, a request from your instance must traverse a number of steps:

1. From the instance to a router, (A)

2. Through a NAT rule in the router namespace,  (B)

3. To an instance of the neutron-ns-metadata-proxy, (C)

4. To the actual Nova metadata service (D)

References

1. OpenStack Networking concepts


Follow

Get every new post delivered to your Inbox.