TripleO QuickStart HA Setup && Keeping undercloud persistent between cold reboots

June 25, 2016

This post follows up http://lxer.com/module/newswire/view/230814/index.html and might work as timer saver unless status undecloud.qcow2 per http://artifacts.ci.centos.org/artifacts/rdo/images/mitaka/delorean/stable/ requires fresh installation to be done from scratch
So, we intend to survive VIRTHOST cold reboot (downtime) and keep previous version of undercloud VM been able to bring it up avoiding build via quickstart.sh and restart procedure from logging into undercloud and immediately run overcloud deployment. Proceed as follows :-

1. System shutdown

Cleanly commit :-
[stack@undercloud~] $ openstack stack delete overcloud
2. Login into VIRTHOST as stack and gracefully shutdown undercloud
[stack@ServerCentOS72 ~]$ virsh shutdown undercloud

**************************************
Shutdown and bring up VIRTHOST
**************************************
Login as root to VIRTHOST :-
[boris@ServerCentOS72 ~]$ sudo su –
[sudo] password for boris:
Last login: Fri Jun 24 16:47:25 MSK 2016 on pts/0

********************************************************************************
This is core step , not to create /run/user/1001/libvirt by root
setting appropriate permissions, just only set correct permissions
on /run/user.  This will allow “stack” to issue `virsh list –all` and create
by himself /run/user/1001/libvirt. The rest works fine for myself
********************************************************************************

[root@ServerCentOS72 ~]# chown -R stack /run/user
[root@ServerCentOS72 ~]# chgrp -R stack /run/user
[root@ServerCentOS72 ~]# ls -ld  /run/user
drwxr-xr-x. 3 stack stack 60 Jun 24 20:01 /run/user

[root@ServerCentOS72 ~]# su – stack
Last login: Fri Jun 24 16:48:09 MSK 2016 on pts/0
[stack@ServerCentOS72 ~]$ virsh list –all

Id    Name                           State
—————————————————-
–     compute_0                   shut off
–     compute_1                   shut off
–     control_0                   shut off
–     control_1                   shut off
–     control_2                   shut off
–     undercloud                  shut off

**********************
Make sure :-
**********************
[stack@ServerCentOS72 ~]$ ls -ld /run/user/1001/libvirt
drwx——. 6 stack stack 160 Jun 24 21:38 /run/user/1001/libvirt

[stack@ServerCentOS72 ~]$ virsh start undercloud
Domain undercloud started

[stack@ServerCentOS72 ~]$ virsh list –all
Id    Name                           State
—————————————————————
2     undercloud                     running
–     compute_0                      shut off
–     compute_1                      shut off
–     control_0                      shut off
–     control_1                      shut off
–     control_2                      shut off

Wait about 5 min and access the undercloud from workstation by:-

[boris@fedora22wks tripleo-quickstart]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added ‘192.168.1.75’ (ECDSA) to the list of known hosts.
Warning: Permanently added ‘undercloud’ (ECDSA) to the list of known hosts.
Last login: Fri Jun 24 15:34:40 2016 from gateway

[stack@undercloud ~]$ ls -l
total 1640244

-rw-rw-r–. 1 stack stack   13287936 Jun 24 13:10 cirros.img
-rw-rw-r–. 1 stack stack    3740163 Jun 24 13:10 cirros.initramfs
-rw-rw-r–. 1 stack stack    4979632 Jun 24 13:10 cirros.kernel
-rw-rw-r–. 1  1001  1001      21769 Jun 24 11:56 instackenv.json
-rw-r–r–. 1 root  root   385824684 Jun 24 03:28 ironic-python-agent.initramfs
-rwxr-xr-x. 1 root  root     5158704 Jun 24 03:28 ironic-python-agent.kernel
-rwxr-xr-x. 1 stack stack        487 Jun 24 12:17 network-environment.yaml
-rwxr-xr-x. 1 stack stack        792 Jun 24 12:17 overcloud-deploy-post.sh
-rwxr-xr-x. 1 stack stack       2284 Jun 24 12:17 overcloud-deploy.sh
-rw-rw-r–. 1 stack stack       4324 Jun 24 13:50 overcloud-env.json
-rw-r–r–. 1 root  root    36478203 Jun 24 03:28 overcloud-full.initrd
-rw-r–r–. 1 root  root  1224070144 Jun 24 03:29 overcloud-full.qcow2
-rwxr-xr-x. 1 root  root     5158704 Jun 24 03:29 overcloud-full.vmlinuz
-rw-rw-r–. 1 stack stack        389 Jun 24 14:28 overcloudrc
-rwxr-xr-x. 1 stack stack       3374 Jun 24 12:17 overcloud-validate.sh
-rwxr-xr-x. 1 stack stack        284 Jun 24 12:17 run-tempest.sh
-rw-r–r–. 1 stack stack        161 Jun 24 12:17 skipfile
-rw——-. 1 stack stack        287 Jun 24 12:16 stackrc
-rw-rw-r–. 1 stack stack        232 Jun 24 14:28 tempest-deployer-input.conf
drwxrwxr-x. 9 stack stack       4096 Jun 24 15:23 tripleo-ci
-rw-rw-r–. 1 stack stack       1123 Jun 24 14:28 tripleo-overcloud-passwords
-rw——-. 1 stack stack       6559 Jun 24 11:59 undercloud.conf
-rw-rw-r–. 1 stack stack     782405 Jun 24 12:16 undercloud_install.log
-rwxr-xr-x. 1 stack stack         83 Jun 24 12:00 undercloud-install.sh
-rw-rw-r–. 1 stack stack       1579 Jun 24 12:00 undercloud-passwords.conf
-rw-rw-r–. 1 stack stack       7699 Jun 24 12:17 undercloud_post_install.log
-rwxr-xr-x. 1 stack stack       2780 Jun 24 12:00 undercloud-post-install.sh

[stack@undercloud ~]$ ./overcloud-deploy.sh

Fourth redeployment based on same undercloud VM.  DHCP pool of ctlplane
is obviosly increasing  starting point



Libvirt's pool && volumes configuration been built by QuickStart



***************************************************************************
A bit different way to manage - login as stack and invoke virt-manager
via `virt-manager --connect qemu:///session` when /run/user already got
a correct permissions.
***************************************************************************

$ sudo su -
# chown -R stack /run/user
# chgrp -R stack /run/user
^D

[stack@ServerCentOS72 ~]$ virsh list --all
Id Name State
----------------------------------------------------
- compute_0 shut off
- compute_1 shut off
- control_0 shut off
- control_1 shut off
- control_2 shut off
- undercloud shut off

[stack@ServerCentOS72 ~]$ virt-manager --connect qemu:///session
[stack@ServerCentOS72 ~]$ virsh list --all
Id Name State
----------------------------------------------------
2 undercloud running
- compute_0 shut off
- compute_1 shut off
- control_0 shut off
- control_1 shut off
- control_2 shut off



From workstation connect to undercloud

[boris@fedora22wks tripleo-quickstart]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud

[stack@undercloud~] ./overcloud-deploy.sh
In several minutes you will see




[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 40754e8a-461e-4328-b0c4-6740c71e9a0d | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.27 |
| df272524-a0bd-4ed7-b95c-92ac779c0b96 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.26 |
| 22802ff4-c472-4500-94d7-415c429073ab | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.29 |
| e79a8967-5c81-4ce1-9037-4e07b298d779 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.25 |
| 27a7c6ac-a480-4945-b4d5-72e32b3c1886 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.28 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+


[stack@undercloud ~]$ ssh heat-admin@192.0.2.27
Last login: Sat Jun 25 09:35:35 2016 from 192.0.2.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
Last login: Sat Jun 25 09:54:06 UTC 2016 on pts/0

[root@overcloud-controller-0 ~]# .  keystonerc_admin
[root@overcloud-controller-0 ~(keystone_admin)]# pcs status
Cluster name: tripleo_cluster
Last updated: Sat Jun 25 10:04:32 2016        Last change: Sat Jun 25 09:21:21 2016 by root via cibadmin on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-2 (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
3 nodes and 127 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-172.16.2.5    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.3.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-1
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.0.2.24    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 ip-10.0.0.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-1
 ip-172.16.1.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-1 ]
     Slaves: [ overcloud-controller-0 overcloud-controller-2 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume    (systemd:openstack-cinder-volume):    Started overcloud-controller-0
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: delay-clone [delay]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Failed Actions:
* openstack-aodh-evaluator_monitor_60000 on overcloud-controller-1 'not running' (7): call=92, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 09:16:45 2016', queued=0ms, exec=0ms
* openstack-gnocchi-metricd_monitor_60000 on overcloud-controller-1 'not running' (7): call=355, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 10:00:10 2016', queued=0ms, exec=0ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-1 'not running' (7): call=313, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 09:20:51 2016', queued=0ms, exec=2101ms
* openstack-ceilometer-central_start_0 on overcloud-controller-1 'not running' (7): call=328, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 09:23:05 2016', queued=0ms, exec=2121ms
* openstack-aodh-evaluator_monitor_60000 on overcloud-controller-0 'not running' (7): call=97, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 09:16:43 2016', queued=0ms, exec=0ms
* openstack-gnocchi-metricd_monitor_60000 on overcloud-controller-0 'not running' (7): call=365, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 10:00:12 2016', queued=0ms, exec=0ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-0 'not running' (7): call=324, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 09:22:32 2016', queued=0ms, exec=2237ms
* openstack-ceilometer-central_start_0 on overcloud-controller-0 'not running' (7): call=342, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 09:23:32 2016', queued=0ms, exec=2200ms
* openstack-aodh-evaluator_monitor_60000 on overcloud-controller-2 'not running' (7): call=94, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 09:16:47 2016', queued=0ms, exec=0ms
* openstack-gnocchi-metricd_monitor_60000 on overcloud-controller-2 'not running' (7): call=353, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 10:00:08 2016', queued=0ms, exec=0ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-2 'not running' (7): call=318, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 09:22:39 2016', queued=0ms, exec=2113ms
* openstack-ceilometer-central_start_0 on overcloud-controller-2 'not running' (7): call=322, status=complete, exitreason='none',
    last-rc-change='Sat Jun 25 09:22:48 2016', queued=0ms, exec=2123ms



PCSD Status:
overcloud-controller-0: Online
overcloud-controller-1: Online
overcloud-controller-2: Online

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled

Advertisements

RDO Triple0 QuickStart HA Setup on Intel Core i7-4790 Desktop (work in progress)

June 18, 2016

This post follows up https://www.linux.com/blog/rdo-triple0-quickstart-ha-setup-intel-core-i7-4790-desktop
In meantime undercloud-install,undercloud-post-install (openstack undercloud install, openstack overcloud image upload ) are supposed to be performed during original run  `bash quickstart.sh –config /path-to/ha.yml $VIRTHOST` run. Neutron networks deployment on undercloud and HA Server’s configuration has been significantly rebuilt since 06/03/2016. I believe design bellow is close to proposed in https://remote-lab.net/rdo-manager-ha-openstack-deployment
However , attempt to reproduce http://docs.openstack.org/developer/tripleo-docs/installation/installation.html
results  hanging  on  `openstack undercloud install`, wheh it attempts to start openstack-nova-compute on undercloud. Nova-compute.log report failure to connect 127.0.0.1:5672. Verification via `netstat -antp | grep 5672` reports port 5672 bind only to 192.0.2.1 ( ctlplane IP address ).
See also https://www.redhat.com/archives/rdo-list/2016-March/msg00171.html
Quoting ( complaints are not mine)  :-
By the way, I’d love to see and help to have an complete installation guide for TripleO powered by RDO on the RDO site (the instack virt setup without quickstart . . . . 

*****************************
Start on workstation :-
*****************************

$ git clone https://github.com/openstack/tripleo-quickstart
$ cd tripleo-quickstart
$ sudo bash quickstart.sh –install-deps
$ sudo yum -y  install redhat-rpm-config
$ export VIRTHOST=192.168.1.75 #put your own IP here
$ ssh-keygen
$ ssh-copy-id root@$VIRTHOST
$ ssh root@$VIRTHOST uname -a # no root login prompt
######################
# Template code
######################
compute_memory: 6144
compute_vcpu:1
undercloud_memory: 8192

# Giving the undercloud additional CPUs can greatly improve heat’s
# performance (and result in a shorter deploy time).

undercloud_vcpu: 4

# Create three controller nodes and one compute node.

overcloud_nodes:
– name: control_0

flavor: control
– name: control_1
flavor: control
– name: control_2
flavor: control

– name: compute_0
flavor: compute
– name: compute_1
flavor: compute

# We don’t need introspection in a virtual environment (because we are
# creating all the “hardware” we really know the necessary
# information).
introspect: false
# Tell tripleo about our environment.

network_isolation: true
extra_args: >-
  –control-scale 3 –compute-scale 2 –neutron-network-type vxlan
  –neutron-tunnel-types vxlan
  -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
  –ntp-server pool.ntp.org
deploy_timeout: 75
tempest: false
pingtest: true

***********************************************
Then run under tripleo-quickstart
***********************************************

$ bash quickstart.sh –config ./config/general_config/ha.yml  $VIRTHOST

During this run the most important is to reach this point on VIRTHOST

[root@ServerCentOS72 ~]# cd /var/cache/tripleo-quickstart/images

[root@ServerCentOS72 images]# ls -l
total 2638232
-rw-rw-r–. 1 stack stack 2701548544 Jun 17 19:25 83e62624dd7bd637dada343bbf4fe8f1.qcow2
lrwxrwxrwx. 1 stack stack         75 Jun 17 19:25 latest-undercloud.qcow2 -> /var/cache/tripleo-quickstart/images/83e62624dd7bd637dada343bbf4fe8f1.qcow2

Saturday 18 June 2016  12:07:05 +0300 (0:00:00.124)       0:26:21.276

===============================================================================
 tripleo/undercloud : Install the undercloud -------------------------- 1155.95s
/home/boris/tripleo-quickstart/roles/tripleo/undercloud/tasks/install-undercloud.yml:1 
setup/undercloud : Get undercloud vm ip address ------------------------ 81.26s
/home/boris/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:173 
setup/undercloud : Resize undercloud image (call virt-resize) ---------- 76.39s
/home/boris/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:122 
tripleo/undercloud : Prepare the undercloud for deploy ----------------- 70.15s
/home/boris/tripleo-quickstart/roles/tripleo/undercloud/tasks/post-install.yml:27 
setup/undercloud : Upload undercloud volume to storage pool ------------ 53.20s
/home/boris/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:142 
setup/undercloud : Copy instackenv.json to appliance ------------------- 35.25s
/home/boris/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:53
setup/undercloud : Get qcow2 image from cache -------------------------- 32.77s
/home/boris/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/fetch_image.yml:144 
setup/undercloud : Inject undercloud ssh public key to appliance -------- 7.07s
/home/boris/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:72 
setup ------------------------------------------------------------------- 6.68s
None --------------------------------------------------------------------------
setup/undercloud : Perform selinux relabel on undercloud image ---------- 3.47s
/home/boris/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:94
environment/teardown : Check if libvirt is available -------------------- 1.99s
/home/boris/tripleo-quickstart/roles/environment/teardown/tasks/main.yml:8 ----
setup ------------------------------------------------------------------- 1.92s
/home/boris/.quickstart/playbooks/provision.yml:29 ----------------------------
setup ------------------------------------------------------------------- 1.90s
None --------------------------------------------------------------------------
setup ------------------------------------------------------------------- 1.81s
None --------------------------------------------------------------------------
parts/libvirt : Install packages for libvirt ---------------------------- 1.78s
/home/boris/tripleo-quickstart/roles/parts/libvirt/tasks/main.yml:5 -----------
setup/overcloud : Create overcloud vm storage --------------------------- 1.57s
/home/boris/tripleo-quickstart/roles/libvirt/setup/overcloud/tasks/main.yml:55 
setup/overcloud : Define overcloud vms ---------------------------------- 1.48s
/home/boris/tripleo-quickstart/roles/libvirt/setup/overcloud/tasks/main.yml:67 
provision/teardown : Remove non-root user account ----------------------- 1.41s
/home/boris/tripleo-quickstart/roles/provision/teardown/tasks/main.yml:47 -----
provision/teardown : Wait for processes to exit ------------------------- 1.41s
/home/boris/tripleo-quickstart/roles/provision/teardown/tasks/main.yml:27 -----
environment/teardown : Stop libvirt networks ---------------------------- 1.35s
/home/boris/tripleo-quickstart/roles/environment/teardown/tasks/main.yml:29 ---

+ set +x

##################################
Virtual Environment Setup Complete
##################################

Access the undercloud by:
ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
There are scripts in the home directory to continue the deploy:
1. overcloud-deploy.sh will deploy the overcloud

Detailed syntax of `openstack overcloud deploy –templates … `
captured by snapshot bellow, compare with https://remote-lab.net/rdo-manager-ha-openstack-deployment

$ openstack overcloud deploy –control-scale 3 –compute-scale 2  \
–libvirt-type qemu –ntp-server pool.ntp.org –templates ~/the-cloud/  \
-e ~/the-cloud/environments/puppet-pacemaker.yaml  \
-e ~/the-cloud/environments/network-isolation.yaml  \
-e ~/the-cloud/environments/net-single-nic-with-vlans.yaml  \
-e ~/the-cloud/environments/network-environment.yaml

Screenshot from 2016-06-19 14-29-39
  2.   overcloud-deploy-post.sh will do any post-deploy configuration
  3.   overcloud-validate.sh will run post-deploy validation

Alternatively, you can ignore these scripts and follow the upstream docs,
starting from the overcloud deploy section:

http://ow.ly/1Vc1301iBlb

Then run 3 mentoned above scripts

[stack@undercloud ~]$ . stackrc
[stack@undercloud ~]$ heat stack-list
+--------------------------------------+------------+-----------------+---------------------+--------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time |
+--------------------------------------+------------+-----------------+---------------------+--------------+
| 356243b1-a071-45c8-8083-85b9a12532c6 | overcloud  | CREATE_COMPLETE | 2016-06-18T09:09:40 | None         |
+--------------------------------------+------------+-----------------+---------------------+--------------+

[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| dbb233ab-9108-4a22-b0dd-44c6ef9a481a | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| 1a91083e-e1ba-43c3-8ad2-78500f6b3ecb | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| 0b3f6ec8-0a13-4f40-b9e3-4557f1b8c7a3 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| 97a8a546-72a0-4431-8065-c1f81103ee25 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| e87a79db-75f8-437f-8ed7-f29aacfe7339 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+

[stack@undercloud ~]$ neutron net-list
+--------------------------------------+--------------+----------------------------------------+
| id                                   | name         | subnets                                |
+--------------------------------------+--------------+----------------------------------------+
| cde382ae-a7fa-4ebb-bbdc-9e2af9c0df83 | external     | 42fac214-7177-4b4f-8778-105015ed30da   |
|                                      |              | 10.0.0.0/24                            |
| 5fc97bca-fa67-4ede-b4d3-8234c0ace5e5 | storage_mgmt | 719f9a19-2f1d-4eed-914a-430468086f10   |
|                                      |              | 172.16.3.0/24                          |
| 4236d358-b4cd-4fb9-a337-f8a421bb13cd | tenant       | d6f1e772-c0a1-4869-a9bc-b551faf5be8e   |
|                                      |              | 172.16.0.0/24                          |
| a4155b70-a4d8-41bf-bbe6-a5f4e248c5ad | ctlplane     | 199a8e99-d9c7-43f2-8ccd-6a59b8424362   |
|                                      |              | 192.0.2.0/24                           |
| fae53fb0-c5da-427f-b473-bfaa0ab21877 | internal_api | 5f2ff369-1000-4361-8131-b0ae69821b9f   |
|                                      |              | 172.16.2.0/24                          |
| 41862220-b9e6-4000-8341-9fbdb34b47f5 | storage      | d0cf1cac-f841-41dd-923d-47d164c07d0f   |
|                                      |              | 172.16.1.0/24                          |
+--------------------------------------+--------------+----------------------------------------+

[stack@undercloud ~]$ cat overcloudrc
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://10.0.0.4:5000/v2.0
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,10.0.0.4,192.0.2.6
export OS_PASSWORD=gdjYmYMdB6aWX8PjBUWdCHkem
export PYTHONWARNINGS=”ignore:Certificate has no, ignore:A true SSLContext object is not available”
export OS_TENANT_NAME=admin

[stack@undercloud ~]$ nova list
+————————————–+————————-+——–+————+————-+———————+

| ID                                   | Name                    | Status | Task State | Power State | Networks            |

+————————————–+————————-+——–+————+————-+———————+
| dbb233ab-9108-4a22-b0dd-44c6ef9a481a | overcloud-controller-0  | ACTIVE | –          | Running     | ctlplane=192.0.2.11 |
| 1a91083e-e1ba-43c3-8ad2-78500f6b3ecb | overcloud-controller-1  | ACTIVE | –          | Running     | ctlplane=192.0.2.7  |
| 0b3f6ec8-0a13-4f40-b9e3-4557f1b8c7a3 | overcloud-controller-2  | ACTIVE | –          | Running     | ctlplane=192.0.2.9  |
| 97a8a546-72a0-4431-8065-c1f81103ee25 | overcloud-novacompute-0 | ACTIVE | –          | Running     | ctlplane=192.0.2.10 |
| e87a79db-75f8-437f-8ed7-f29aacfe7339 | overcloud-novacompute-1 | ACTIVE | –          | Running     | ctlplane=192.0.2.8  |
+————————————–+————————-+——–+————+———

[stack@undercloud ~]$ ssh heat-admin@192.0.2.11

The authenticity of host ‘192.0.2.11 (192.0.2.11)’ can’t be established.
ECDSA key fingerprint is 74:99:da:b1:c8:ac:58:e6:65:c1:51:45:64:e4:e9:ed.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.0.2.11’ (ECDSA) to the list of known hosts.
Last login: Sat Jun 18 09:52:37 2016 from 192.0.2.1
[heat-admin@overcloud-controller-0 ~]$ sudo su –
[root@overcloud-controller-0 ~]# vi keystonerc_admin
[root@overcloud-controller-0 ~]# .  keystonerc_admin
[root@overcloud-controller-0 ~(keystone_admin)]# psc status
-bash: psc: command not found
[root@overcloud-controller-0 ~(keystone_admin)]# pcs  status
Cluster name: tripleo_cluster
Last updated: Sat Jun 18 10:01:58 2016        Last change: Sat Jun 18 09:49:22 2016 by root via cibadmin on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-1 (version 1.1.13-10.el7_2.2-44eb2dd) – partition with quorum

3 nodes and 127 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Full list of resources:
ip-192.0.2.6    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
ip-172.16.2.5    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-1
ip-172.16.3.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
Clone Set: haproxy-clone [haproxy]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Master/Slave Set: galera-master [galera]
Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: memcached-clone [memcached]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
ip-10.0.0.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
ip-172.16.2.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-1
ip-172.16.1.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
Clone Set: rabbitmq-clone [rabbitmq]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-core-clone [openstack-core]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Master/Slave Set: redis-master [redis]
Masters: [ overcloud-controller-1 ]
Slaves: [ overcloud-controller-0 overcloud-controller-2 ]
Clone Set: mongod-clone [mongod]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
openstack-cinder-volume    (systemd:openstack-cinder-volume):    Started overcloud-controller-0
Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-clone [openstack-heat-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-glance-api-clone [openstack-glance-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-api-clone [openstack-nova-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: delay-clone [delay]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-server-clone [neutron-server]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: httpd-clone [httpd]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Failed Actions:
* openstack-aodh-evaluator_monitor_60000 on overcloud-controller-1 'not running' (7): call=95, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:44:43 2016', queued=0ms, exec=0ms
* openstack-gnocchi-metricd_monitor_60000 on overcloud-controller-1 'not running' (7): call=331, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:56:44 2016', queued=0ms, exec=0ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-1 'not running' (7): call=335, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:50:53 2016', queued=0ms, exec=2099ms
* openstack-ceilometer-central_start_0 on overcloud-controller-1 'not running' (7): call=339, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:51:17 2016', queued=0ms, exec=2117ms
* openstack-aodh-evaluator_monitor_60000 on overcloud-controller-0 'not running' (7): call=96, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:44:40 2016', queued=0ms, exec=0ms
* openstack-gnocchi-metricd_monitor_60000 on overcloud-controller-0 'not running' (7): call=332, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:56:42 2016', queued=0ms, exec=0ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-0 'not running' (7): call=339, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:51:13 2016', queued=0ms, exec=2145ms
* openstack-ceilometer-central_start_0 on overcloud-controller-0 'not running' (7): call=341, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:51:28 2016', queued=0ms, exec=2147ms
* openstack-aodh-evaluator_start_0 on overcloud-controller-2 'not running' (7): call=368, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:53:18 2016', queued=0ms, exec=2107ms
* openstack-gnocchi-metricd_monitor_60000 on overcloud-controller-2 'not running' (7): call=321, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:56:46 2016', queued=0ms, exec=0ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-2 'not running' (7): call=326, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:51:06 2016', queued=0ms, exec=2185ms
* openstack-ceilometer-central_start_0 on overcloud-controller-2 'not running' (7): call=378, status=complete, exitreason='none',
last-rc-change='Sat Jun 18 09:54:14 2016', queued=1ms, exec=2116ms

PCSD Status:
overcloud-controller-0: Online
overcloud-controller-1: Online
overcloud-controller-2: Online

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled

[root@overcloud-controller-0 ~(keystone_admin)]# ovs-vsctl show
8fea5ee4-62cf-4767-96c8-d9867cab9972
Bridge br-tun
fail_mode: secure
Port br-tun
Interface br-tun
type: internal
Port “vxlan-ac100004”
Interface “vxlan-ac100004”
type: vxlan
ptions: {df_default=”true”, in_key=flow, local_ip=”172.16.0.6″, out_key=flow, remote_ip=”172.16.0.4″}
Port “vxlan-ac100005”
Interface “vxlan-ac100005”
type: vxlan
options: {df_default=”true”, in_key=flow, local_ip=”172.16.0.6″, out_key=flow, remote_ip=”172.16.0.5″}
Port “vxlan-ac100008”
Interface “vxlan-ac100008”
type: vxlan
options: {df_default=”true”, in_key=flow, local_ip=”172.16.0.6″, out_key=flow, remote_ip=”172.16.0.8″}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port “vxlan-ac100007”
Interface “vxlan-ac100007”
type: vxlan
options: {df_default=”true”, in_key=flow, local_ip=”172.16.0.6″, out_key=flow, remote_ip=”172.16.0.7″}
Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Bridge br-ex
Port br-ex
Interface br-ex
  type: internal
Port “vlan20”
tag: 20
Interface “vlan20”
type: internal
Port “eth0”
Interface “eth0”
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Port “vlan40”
tag: 40
Interface “vlan40”
type: internal
Port “vlan50”
tag: 50
Interface “vlan50”
type: internal
Port “vlan10”
tag: 10
Interface “vlan10”
type: internal
Port “vlan30”
tag: 30
Interface “vlan30”
type: internal
ovs_version: “2.5.0”

[root@overcloud-controller-0 ~(keystone_admin)]# ifconfig

br-ex: flags=4163  mtu 1500
inet 192.0.2.11  netmask 255.255.255.0  broadcast 192.0.2.255
inet6 fe80::250:dcff:fecf:b7d5  prefixlen 64  scopeid 0x20
ether 00:50:dc:cf:b7:d5  txqueuelen 0  (Ethernet)
RX packets 15254  bytes 29305270 (27.9 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 15111  bytes 2037368 (1.9 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163  mtu 1500
inet6 fe80::250:dcff:fecf:b7d5  prefixlen 64  scopeid 0x20
ether 00:50:dc:cf:b7:d5  txqueuelen 1000  (Ethernet)
RX packets 554865  bytes 314056269 (299.5 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 537763  bytes 196316938 (187.2 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10
loop  txqueuelen 0  (Local Loopback)
RX packets 128951  bytes 42842317 (40.8 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 128951  bytes 42842317 (40.8 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vlan10: flags=4163  mtu 1500
inet 10.0.0.6  netmask 255.255.255.0  broadcast 10.0.0.255
inet6 fe80::2cf7:9cff:fe98:df2e  prefixlen 64  scopeid 0x20
ether 2e:f7:9c:98:df:2e  txqueuelen 0  (Ethernet)
RX packets 1563  bytes 22172141 (21.1 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 935  bytes 339459 (331.5 KiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vlan20: flags=4163  mtu 1500
inet 172.16.2.9  netmask 255.255.255.0  broadcast 172.16.2.255
inet6 fe80::9c4a:96ff:fe42:f562  prefixlen 64  scopeid 0x20
ether 9e:4a:96:42:f5:62  txqueuelen 0  (Ethernet)
RX packets 515281  bytes 202417994 (193.0 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 498334  bytes 112312907 (107.1 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vlan30: flags=4163  mtu 1500
inet 172.16.1.5  netmask 255.255.255.0  broadcast 172.16.1.255
inet6 fe80::8cbe:80ff:fe80:7945  prefixlen 64  scopeid 0x20
ether 8e:be:80:80:79:45  txqueuelen 0  (Ethernet)
RX packets 20275  bytes 45196003 (43.1 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 20405  bytes 52618634 (50.1 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vlan40: flags=4163  mtu 1500
inet 172.16.3.6  netmask 255.255.255.0  broadcast 172.16.3.255
inet6 fe80::8c06:98ff:fe7a:5b7  prefixlen 64  scopeid 0x20
ether 8e:06:98:7a:05:b7  txqueuelen 0  (Ethernet)
RX packets 2299  bytes 12722091 (12.1 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 2557  bytes 26854977 (25.6 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vlan50: flags=4163  mtu 1500
inet 172.16.0.6  netmask 255.255.255.0  broadcast 172.16.0.255
inet6 fe80::6454:dff:fe41:90e9  prefixlen 64  scopeid 0x20
ether 66:54:0d:41:90:e9  txqueuelen 0  (Ethernet)
RX packets 107  bytes 9834 (9.6 KiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 121  bytes 12394 (12.1 KiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@overcloud-controller-0 ~(keystone_admin)]# route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0 vlan10
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 vlan10
169.254.169.254 192.0.2.1       255.255.255.255 UGH   0      0        0 br-ex
172.16.0.0      0.0.0.0         255.255.255.0   U     0      0        0 vlan50
172.16.1.0      0.0.0.0         255.255.255.0   U     0      0        0 vlan30
172.16.2.0      0.0.0.0         255.255.255.0   U     0      0        0 vlan20
172.16.3.0      0.0.0.0         255.255.255.0   U     0      0        0 vlan40
192.0.2.0       0.0.0.0         255.255.255.0   U     0      0        0 br-ex

[root@overcloud-controller-0 ~]# cat /etc/os-net-config/config.json | jq ‘.[]’
[
{
“addresses”: [
{
“ip_netmask”: “192.0.2.11/24”
}
],
“type”: “ovs_bridge”,
“use_dhcp”: false,
“routes”: [
{
“next_hop”: “192.0.2.1”,
“ip_netmask”: “169.254.169.254/32”
}
],
“members”: [
{
“primary”: true,
“name”: “nic1”,
“type”: “interface”
},
{
“vlan_id”: 10,
“addresses”: [
{
“ip_netmask”: “10.0.0.6/24”
}
],
“type”: “vlan”,
“routes”: [
{
“next_hop”: “10.0.0.1”,
“default”: true
}
]
},
{
“vlan_id”: 20,
“addresses”: [
{
“ip_netmask”: “172.16.2.9/24”
}
],
“type”: “vlan”
},
{
“vlan_id”: 30,
“addresses”: [
{
“ip_netmask”: “172.16.1.5/24”
}
],
“type”: “vlan”
},
{
“vlan_id”: 40,
“addresses”: [
{
“ip_netmask”: “172.16.3.6/24”
}
],
“type”: “vlan”
},
{
“vlan_id”: 50,
“addresses”: [
{
“ip_netmask”: “172.16.0.6/24”
}
],
“type”: “vlan”
}
],
“name”: “br-ex”,
“dns_servers”: [
“8.8.8.8”,
“8.8.4.4”
]
}
]

************************
On undercloud
************************

[stack@undercloud ~]$ sudo su -
Last login: Sat Jun 18 10:47:31 UTC 2016 on pts/1
[root@undercloud ~]# ovs-vsctl show
7fb4d9b7-4704-410f-845f-6f3f0a1b65cd
Bridge br-ctlplane
Port "vlan10"
tag: 10
Interface "vlan10"
type: internal

Port br-ctlplane
Interface br-ctlplane
type: internal
Port phy-br-ctlplane
Interface phy-br-ctlplane
type: patch
options: {peer=int-br-ctlplane}
Port "eth1"
Interface "eth1"
Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "tap41a7c72c-39"
tag: 1
Interface "tap41a7c72c-39"
type: internal
Port int-br-ctlplane
Interface int-br-ctlplane
type: patch
options: {peer=phy-br-ctlplane}
ovs_version: "2.5.0"

[root@undercloud ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.23.1 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 vlan10
192.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 br-ctlplane
192.168.23.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

[root@undercloud ~]# ifconfig
br-ctlplane: flags=4163 mtu 1500
inet 192.0.2.1 netmask 255.255.255.0 broadcast 192.0.2.255

inet6 fe80::2ad:c4ff:fe6f:778a prefixlen 64 scopeid 0x20
ether 00:ad:c4:6f:77:8a txqueuelen 0 (Ethernet)
RX packets 4743446 bytes 382457275 (364.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6573214 bytes 31299066406 (29.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4163 mtu 1500
inet 192.168.23.46 netmask 255.255.255.0 broadcast 192.168.23.255
inet6 fe80::2ad:c4ff:fe6f:7788 prefixlen 64 scopeid 0x20
ether 00:ad:c4:6f:77:88 txqueuelen 1000 (Ethernet)
RX packets 402911 bytes 1166354846 (1.0 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 286351 bytes 63608008 (60.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth1: flags=4163 mtu 1500
inet6 fe80::2ad:c4ff:fe6f:778a prefixlen 64 scopeid 0x20
ether 00:ad:c4:6f:77:8a txqueuelen 1000 (Ethernet)
RX packets 4793675 bytes 390579748 (372.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6627325 bytes 32167819071 (29.9 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 0 (Local Loopback)
RX packets 5342779 bytes 31375282714 (29.2 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5342779 bytes 31375282714 (29.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0: flags=4099 mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:b7:65:c0 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vlan10: flags=4163 mtu 1500
inet 10.0.0.1 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::c4d1:81ff:fec1:6006 prefixlen 64 scopeid 0x20
ether c6:d1:81:c1:60:06 txqueuelen 0 (Ethernet)
RX packets 49362 bytes 7857042 (7.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 52980 bytes 868430005 (828.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0