TripleO QuickStart KSM vs instack-virt-setup deploying RDO Newton HA Overcloud

October 12, 2016

=================
UPDATE 10/17/2016
=================
I initiated KSM&&KSMTUNED on CentOS 7.2 VIRTHOST running instack-virt-setup 2 days ago along with instack-virt-setup HA overcloud deployment. Stack’s virthost’s .bashrc
[stack@Server72Centos ~]$ cat .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# Uncomment the following line if you don’t like systemctl’s auto-paging feature:
# export SYSTEMD_PAGER=
export NODE_DIST=centos7
export NODE_CPU=2
export NODE_MEM=6500
export NODE_COUNT=4
export UNDERCLOUD_NODE_CPU=2
export UNDERCLOUD_NODE_MEM=8000
export NODE_DISK=45
export UNDERCLOUD_NODE_DISK=35
export FS_TYPE=ext4
# User specific aliases and functions
export LIBVIRT_DEFAULT_URI=”qemu:///system”

So far I don’t see any negative drawback from ksmd daemon up and running on VIRTHOST ( HA Overcloud been built via instack-virt-setup). I also have to notice that performance problems caused by swap memory utilization around 3.5 GB are eliminated. Overcloud KVM nodes demonstrate performance close to TripleO QuickStart (in meantime unavailable for RDO Newton -> trunk/newton)

screenshot-from-2016-10-17-19-17-11
============
END UPDATE 
=============

Posting bellow is supposed to demonstrate KSM implementation on QuickStart providing significant relief  on 32 GB VIRTHOST vs quite the same deploymet described in previous draft http://lxer.com/module/newswire/view/234740/index.html   Deployment procedure for TripleO QuickStart is bit more complicated in meantime then it was designed for Mitaka stable release. Instructions bellow provide a step by step guide usually not required by QuickStart environment on undecloud VM after you logged into undercloud

Git clone repo bellow :-
[jon@fedora24wks release]$   git clone https://github.com/openstack/tripleo-quickstart
[jon@fedora24wks release]$  cd tripleo* ; cd ./config/release

**********************************************
Now verify that newton.yml is here.
**********************************************
[jon@fedora24wks release]$ cat newton.yml
release: newton
undercloud_image_url: http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/delorean/undercloud.qcow2
overcloud_image_url: http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/delorean/overcloud-full.tar
ipa_image_url: http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/delorean/ironic-python-agent.tar

**************************************************************************************
UPDATE ./config/general_config/ha.yml  of memory allocation for HA controller
up to  6.5 GB ( as minimum to avoid crash in step5 of overcloud deployment )
**************************************************************************************

[john@fedora24wks tripleo-quickstart]$ cat ./config/general_config/ha.yml# Deploy an HA openstack environment.
#
# This will require (6144 * 4) == approx. 24GB for the overcloud
# nodes, plus another 8GB for the undercloud, for a total of around
# 32GB.
control_memory: 6500
compute_memory: 6144
undercloud_memory: 8192
# Giving the undercloud additional CPUs can greatly improve heat’s
# performance (and result in a shorter deploy time).
undercloud_vcpu: 4
# Create three controller nodes and one compute node.
overcloud_nodes:

– name: control_0
flavor: control
– name: control_1
flavor: control
– name: control_2
flavor: control

– name: compute_0
flavor: compute

# We don’t need introspection in a virtual environment (because we are
# creating all the “hardware” we really know the necessary
# information).
step_introspect: false

# Tell tripleo about our environment.
network_isolation: true
extra_args: >-
–control-scale 3 –compute-scale 1 –neutron-network-type vxlan
–neutron-tunnel-types vxlan
–ntp-server pool.ntp.org
test_ping: true
enable_pacemaker: true
tempest_config: false
run_tempest: false

****************************************************************************
Run quickstart.sh to create undercloud VM on VIRTHOST
****************************************************************************

[john@fedora24wks tripleo-quickstart]$ bash quickstart.sh -R newton –config ./config/general_config/ha.yml $VIRTHOST

Login to undercloud when done

[john@fedora24wks tripleo-quickstart]$  ssh -F /home/john/.quickstart/ssh.config.ansible undercloud

****************************************************************************************************
In meantime QuickStart requires manual overcloud deployment
Now you are logged into undecloud VM running on VIRTHOST as stack
Building overcloud images is skipped due to QuickStart CI. There is no harm in attempt of building them. It will take a second, they are already there.
****************************************************************************************************

# Upload per-built overcloud images
[stack@undercloud ~]$ source stackrc 
[stack@undercloud ~]$ openstack overcloud image upload
[stack@undercloud ~]$ openstack baremetal import instackenv.json
[stack@undercloud ~]$ openstack baremetal configure boot
[stack@undercloud ~]$ openstack baremetal introspection bulk start
[stack@undercloud ~]$ ironic node-list
stack@undercloud ~]$ neutron subnet-list
[stack@undercloud ~]$ neutron subnet-update 1b7d82e5-0bf1-4ba5-8008-4aa402598065 \
                         --dns-nameserver 192.168.122.1

**************************************
Create external interface vlan10
*************************************
[stack@undercloud ~]$  sudo vi /etc/sysconfig/network-scripts/ifcfg-vlan10
DEVICE=vlan10
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSIntPort
BOOTPROTO=static
IPADDR=10.0.0.1
NETMASK=255.255.255.0
OVS_BRIDGE=br-ctlplane
OVS_OPTIONS=”tag=10″

[stack@undercloud ~]$ sudo ifup vlan10
[stack@undercloud ~]$ sudo ovs-vsctl show

0d9f9351-93cd-4c83-8eb4-82e8b1ca6665
Manager “ptcp:6640:127.0.0.1”
is_connected: true
Bridge br-ctlplane
Controller “tcp:127.0.0.1:6633”
is_connected: true
fail_mode: secure
Port br-ctlplane
Interface br-ctlplane
type: internal
Port phy-br-ctlplane
Interface phy-br-ctlplane
type: patch
options: {peer=int-br-ctlplane}
Port “eth1”
Interface “eth1”
Port “vlan10”
tag: 10
Interface “vlan10”
type: internal
Bridge br-int
Controller “tcp:127.0.0.1:6633”
is_connected: true
fail_mode: secure
Port int-br-ctlplane
Interface int-br-ctlplane
type: patch
options: {peer=phy-br-ctlplane}
Port br-int
Interface br-int
type: internal
Port “tapb0b80495-42”
tag: 1
Interface “tapb0b80495-42”
type: internal
ovs_version: “2.5.0”

*********************************************
Create manually network_env.yaml
*********************************************

[stack@instack ~]$vi network_env.yaml

   {
    “parameter_defaults”: {
        “ControlPlaneDefaultRoute”: “192.0.2.1”,
        “ControlPlaneSubnetCidr”: “24”,
        “DnsServers”: [
            “192.168.122.1”
        ],
        “EC2MetadataIp”: “192.0.2.1”,
        “ExternalAllocationPools”: [
            {
                “end”: “10.0.0.250”,
                “start”: “10.0.0.4”
            }
        ],
        “ExternalNetCidr”: “10.0.0.1/24”,
        “NeutronExternalNetworkBridge”: “”
    }
   }

$ sudo iptables -A BOOTSTACK_MASQ -s 10.0.0.0/24 ! -d 10.0.0.0/24 -j MASQUERADE -t nat

[stack@undercloud ~]$ sudo touch -f \
     /usr/share/openstack-tripleo-heat-templates/puppet/post.yaml

[stack@undercloud ~]$ cat overcloud-deploy.sh
#!/bin/bash -x
source /home/stack/stackrc   
openstack overcloud deploy  \
 --control-scale 3 --compute-scale 1 \
 --libvirt-type qemu \
 --ntp-server pool.ntp.org  \
 --templates  /usr/share/openstack-tripleo-heat-templates \
 -e  /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \
 -e  /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
 -e  /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
 -e  $HOME/network_env.yaml

[stack@undercloud ~]$ ./overcloud-deploy.sh
+ source /home/stack/stackrc
++ export NOVA_VERSION=1.1
++ NOVA_VERSION=1.1
+++ sudo hiera admin_password
++ export OS_PASSWORD=a8cf847f82ff64b158afde70183b268fecd9f492
++ OS_PASSWORD=a8cf847f82ff64b158afde70183b268fecd9f492
++ export OS_AUTH_URL=http://192.0.2.1:5000/v2.0
++ OS_AUTH_URL=http://192.0.2.1:5000/v2.0
++ export OS_USERNAME=admin
++ OS_USERNAME=admin
++ export OS_TENANT_NAME=admin
++ OS_TENANT_NAME=admin
++ export COMPUTE_API_VERSION=1.1
++ COMPUTE_API_VERSION=1.1
++ export OS_BAREMETAL_API_VERSION=1.15
++ OS_BAREMETAL_API_VERSION=1.15
++ export OS_NO_CACHE=True
++ OS_NO_CACHE=True
++ export OS_CLOUDNAME=undercloud
++ OS_CLOUDNAME=undercloud
++ export OS_IMAGE_API_VERSION=1
++ OS_IMAGE_API_VERSION=1
+ openstack overcloud deploy --control-scale 3 --compute-scale 1 --libvirt-type qemu --ntp-server pool.ntp.org --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network_env.yaml
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: 5511b4a9-4d0c-4937-9450-e2d9e7e36ab3
Plan updated
Deploying templates in the directory /tmp/tripleoclient-LwH7ZR/tripleo-heat-templates
Started Mistral Workflow. Execution ID: 5e331cfa-4b4e-49dd-bc4c-89b50aa42740
2016-10-12 07:30:50Z [overcloud]: CREATE_IN_PROGRESS  Stack CREATE started
2016-10-12 07:30:51Z [overcloud.HorizonSecret]: CREATE_IN_PROGRESS  state changed
2016-10-12 07:30:52Z [overcloud.PcsdPassword]: CREATE_IN_PROGRESS  state changed
2016-10-12 07:30:52Z [overcloud.MysqlRootPassword]: CREATE_IN_PROGRESS  state changed
2016-10-12 07:30:52Z [overcloud.ServiceNetMap]: CREATE_IN_PROGRESS  state changed
2016-10-12 07:30:53Z [overcloud.HeatAuthEncryptionKey]: CREATE_IN_PROGRESS  state changed
2016-10-12 07:30:53Z [overcloud.RabbitCookie]: CREATE_IN_PROGRESS  state changed
2016-10-12 07:30:54Z [overcloud.Networks]: CREATE_IN_PROGRESS  state changed
2016-10-12 07:30:54Z [overcloud.PcsdPassword]: CREATE_COMPLETE  state changed
2016-10-12 07:30:55Z [overcloud.ServiceNetMap]: CREATE_COMPLETE  state changed
2016-10-12 07:30:55Z [overcloud.HeatAuthEncryptionKey]: CREATE_COMPLETE  state changed
2016-10-12 07:30:55Z [overcloud.Networks]: CREATE_IN_PROGRESS  Stack CREATE started
2016-10-12 07:30:55Z [overcloud.MysqlRootPassword]: CREATE_COMPLETE  state changed
2016-10-12 07:30:55Z [overcloud.RabbitCookie]: CREATE_COMPLETE  state changed
2016-10-12 07:30:55Z [overcloud.Networks.StorageMgmtNetwork]: CREATE_IN_PROGRESS  state changed
2016-10-12 07:30:56Z [overcloud.HorizonSecret]: CREATE_COMPLETE  state changed
2016-10-12 07:30:56Z [overcloud.DefaultPasswords]: CREATE_IN_PROGRESS  state changed
2016-10-12 07:30:56Z [overcloud.Networks.TenantNetwork]: CREATE_IN_PROGRESS  state changed

. . . . . .

2016-10-12 08:18:50Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-10-12 08:18:51Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_COMPLETE  state changed
2016-10-12 08:18:51Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:51Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:51Z [overcloud.AllNodesDeploySteps.BlockStoragePostConfig]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:51Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:51Z [overcloud.AllNodesDeploySteps.CephStoragePostConfig]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:53Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_COMPLETE  state changed
2016-10-12 08:18:53Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_COMPLETE  state changed
2016-10-12 08:18:53Z [overcloud.ControllerAllNodesDeployment.0]: SIGNAL_COMPLETE  Unknown
2016-10-12 08:18:54Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_COMPLETE  state changed
2016-10-12 08:18:54Z [overcloud.AllNodesDeploySteps.BlockStoragePostConfig]: CREATE_COMPLETE  state changed
2016-10-12 08:18:54Z [overcloud.AllNodesDeploySteps.CephStoragePostConfig]: CREATE_COMPLETE  state changed
2016-10-12 08:18:54Z [overcloud.AllNodesDeploySteps.BlockStorageExtraConfigPost]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:55Z [overcloud.AllNodesDeploySteps.ComputeExtraConfigPost]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:55Z [overcloud.AllNodesDeploySteps.CephStorageExtraConfigPost]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:56Z [overcloud.AllNodesDeploySteps.ControllerExtraConfigPost]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:56Z [overcloud.Controller.0.ControllerDeployment]: SIGNAL_COMPLETE  Unknown
2016-10-12 08:18:56Z [overcloud.AllNodesDeploySteps.ObjectStorageExtraConfigPost]: CREATE_IN_PROGRESS  state changed
2016-10-12 08:18:58Z [overcloud.AllNodesDeploySteps.ComputeExtraConfigPost]: CREATE_COMPLETE  state changed
2016-10-12 08:18:58Z [overcloud.AllNodesDeploySteps.BlockStorageExtraConfigPost]: CREATE_COMPLETE  state changed
2016-10-12 08:18:59Z [overcloud.AllNodesDeploySteps.CephStorageExtraConfigPost]: CREATE_COMPLETE  state changed
2016-10-12 08:18:59Z [overcloud.Controller.0.NetworkDeployment]: SIGNAL_COMPLETE  Unknown
2016-10-12 08:18:59Z [overcloud.AllNodesDeploySteps.ControllerExtraConfigPost]: CREATE_COMPLETE  state changed
2016-10-12 08:18:59Z [overcloud.AllNodesDeploySteps.ObjectStorageExtraConfigPost]: CREATE_COMPLETE  state changed
2016-10-12 08:19:00Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-10-12 08:19:01Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  state changed
2016-10-12 08:19:01Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

 Stack overcloud CREATE_COMPLETE 

Overcloud Endpoint: http://10.0.0.10:5000/v2.0
Overcloud Deployed
[stack@undercloud ~]$ . stackrc
[stack@undercloud ~]$ nova list
--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| e6951ba8-a467-4c54-a853-b1fa5f1f3d20 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.6  |
| 1a4c436f-0aab-4fb3-bb86-34fbf38bec4a | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
| 5bc7e75d-2a99-4e73-b440-a37b6164c0b6 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 6e379541-37de-4f3b-8667-fbe5284de10b | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
screenshot-from-2016-10-12-20-06-13
[stack@undercloud ~]$ cat overcloudrc
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://10.0.0.10:5000/v2.0
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,10.0.0.10,192.0.2.10
export OS_PASSWORD=8WvqWXUv4z3a4gdk2EdRzYuZU
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"
export OS_TENANT_NAME=admin
 [stack@undercloud ~]$ ssh heat-admin@192.0.2.6 
The authenticity of host '192.0.2.6 (192.0.2.6)' can't be established.
ECDSA key fingerprint is d1:71:51:eb:72:d2:50:fb:c6:30:13:49:0d:4b:c8:b1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.6' (ECDSA) to the list of known hosts.
[heat-admin@overcloud-controller-0 ~]$ sudo su -
[root@overcloud-controller-0 ~]# vi overcloudrc
[root@overcloud-controller-0 ~]# .  overcloudrc
[root@overcloud-controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Wed Oct 12 08:21:33 2016  Last change: Wed Oct 12 08:09:48 2016 by root via cibadmin on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 19 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-10.0.0.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
 ip-192.0.2.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1
 ip-172.16.2.5 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-172.16.1.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
 ip-172.16.2.13 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-172.16.3.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-0 ]
     Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0

PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled


[root@overcloud-controller-0 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32
Galera cluster node is synced.

Final “top” snapshot on VIRTHOST ( for QuickStart ) after same deployment as

was done in RDO Newton Overcloud HA deployment via instack-virt-setup on CentOS 7.2 VIRTHOST 

Compare numbers under SHR header in reports provided down here.

Cloud F24 VM is running on overcloud-novacompute-0

   Quite the same configuration been done by instack-virt-setup

Swap area utilization at least 2.5 GB ( up to 3.5 GB ) during cloud VM F24 runtime.
***************************************************************************************
System information provided via dashboard ( remote sshuttle connection )
***************************************************************************************

 

  Network Configuration


TripleO deployment of ‘master’ branch via instack-virt-setup on VIRTHOST (2)

September 29, 2016

UPDATE 09/29/2016
$ sudo route add -net 192.0.2.0/24 gw 192.0.2.1 ( on instack VM )
no longer needed , moreover affects ssh connect to overcloud nodes
END UPDATE

Upstream gets close to Newton Release , bugs scheduled for RC2 went away.

Following bellow is a clean and smoothly running procedure of Overcloud deployment TripleO Master branch via instack-virt-setup on 32 GB VIRTHOST Network isolation in overcloud is pre-configured on instack (undercloud ) , step which is usually very hard to locate in official docs http://tripleo.org/index.html

and which is silently skipped in TripleO related blogs. Also mistral execution list gets verified up on overcloud deployment completion. Running `systemctl status keepalived -l` on overcloud controller we see sending gratuitous ARPs what confirms network isolation implementation.

Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]:
VRRP_Instance(51) Sending gratuitous ARPs on br-ex for 192.0.2.13
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]:
VRRP_Instance(56) Entering MASTER STATE
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]:
VRRP_Instance(56) setting protocol VIPs.
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]:
VRRP_Instance(56) Sending gratuitous ARPs on vlan20 for 172.16.2.7
Sep 27 14:55:03 overcloud-controller-0 Keepalived_healthcheckers[18505]:
Netlink reflector reports IP 172.16.2.7 added
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]:
VRRP_Instance(52) Entering MASTER STATE
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]:
VRRP_Instance(52) setting protocol VIPs.
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]:
VRRP_Instance(52) Sending gratuitous ARPs on br-ex for 10.0.0.4
Sep 27 14:55:03 overcloud-controller-0 Keepalived_healthcheckers[18505]:
Netlink reflector reports IP 10.0.0.4 added

*****************************************
Tune stack environment on VIRTHOST
*****************************************
# useradd stack
# echo “stack:stack” | chpasswd
# echo “stack ALL=(root) NOPASSWD:ALL” | sudo tee -a /etc/sudoers.d/stack
# chmod 0440 /etc/sudoers.d/stack
# su – stack

***************************
Tune stack ENV
**************************
export NODE_DIST=centos7
export NODE_CPU=2
export NODE_MEM=7550
export NODE_COUNT=3
export UNDERCLOUD_NODE_CPU=4
export UNDERCLOUD_NODE_MEM=9000
export FS_TYPE=ext4

****************************************************************
Re-login to stack (highlight long line and copy if needed)
****************************************************************

$ sudo yum -y install epel-release
$ sudo yum -y install yum-plugin-priorities
$ sudo curl -o /etc/yum.repos.d/delorean.repo \
http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/delorean.repo
$ sudo curl -o /etc/yum.repos.d/delorean-deps.repo \
http://trunk.rdoproject.org/centos7/delorean-deps.repo
$ sudo yum install -y instack-undercloud
$ instack-virt-setup

*********************
On instack VM
*********************
Create swap file per http://www.anstack.com/blog/2016/07/04/manually-installing-tripleo-recipe.html :-

#Add a 4GB swap file to the Undercloud
sudo dd if=/dev/zero of=/swapfile bs=1024 count=4194304
sudo mkswap /swapfile
#Turn ON the swap file
sudo chmod 600 /swapfile
sudo swapon /swapfile
#Enable it on start
sudo echo “/swapfile swap swap defaults 0 0″ >> /etc/fstab

***************************
Restart instack VM
***************************

Next :-
# su – stack

*************************************
Update .bashrc under ~stack/
*************************************

export USE_DELOREAN_TRUNK=1
export DELOREAN_TRUNK_REPO=”http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/”
export DELOREAN_REPO_FILE=”delorean.repo”
export FS_TYPE=ext4

[stack@instack ~]$ git clone https://github.com/openstack/tripleo-heat-templates
[stack@instack ~]$ git clone https://github.com/openstack-infra/tripleo-ci.git

[stack@instack ~]$ ./tripleo-ci/scripts/tripleo.sh –repo-setup
[stack@instack ~]$ ./tripleo-ci/scripts/tripleo.sh –undercloud
[stack@instack ~]$ source stackrc
[stack@instack ~]$ ./tripleo-ci/scripts/tripleo.sh –overcloud-images
[stack@instack ~]$ ./tripleo-ci/scripts/tripleo.sh –register-nodes
[stack@instack ~]$ ./tripleo-ci/scripts/tripleo.sh –introspect-nodes

Image file overcloud-full.qcow2 created…
Successfully built all requested images
You must source a stackrc file for the Undercloud.
Attempting to source /home/stack/stackrc
Done
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Image “overcloud-full-vmlinuz” was uploaded.
+————————————–+————————+————-+———+——–+
| ID | Name | Disk Format | Size | Status |
+————————————–+————————+————-+———+——–+
| 37dba3bf-5683-4a33-b6d5-9ed90e1f189d | overcloud-full-vmlinuz | aki | 5157296 | active |
+————————————–+————————+————-+———+——–+
Image “overcloud-full-initrd” was uploaded.
+————————————–+———————–+————-+———-+——–+
| ID | Name | Disk Format | Size | Status |
+————————————–+———————–+————-+———-+——–+
| 0bfd61f2-1c03-43ab-82e5-811c346dadd0 | overcloud-full-initrd | ari | 42124221 | active |
+————————————–+———————–+————-+———-+——–+
Image “overcloud-full” was uploaded.
+————————————–+—————-+————-+————+——–+
| ID | Name | Disk Format | Size | Status |
+————————————–+—————-+————-+————+——–+
| d2c41746-fb4c-4438-995b-22811df6f772 | overcloud-full | qcow2 | 1178590720 | active |
+————————————–+—————-+————-+————+——–+
Image “bm-deploy-kernel” was uploaded.
+————————————–+——————+————-+———+——–+
| ID | Name | Disk Format | Size | Status |
+————————————–+——————+————-+———+——–+
| f237b9a5-33a8-4f33-998a-571059f0522b | bm-deploy-kernel | aki | 5157296 | active |
+————————————–+——————+————-+———+——–+
Image “bm-deploy-ramdisk” was uploaded.
+————————————–+——————-+————-+———–+——–+
| ID | Name | Disk Format | Size | Status |
+————————————–+——————-+————-+———–+——–+
| 97c40ed7-296f-42b8-9d3c-3d40b36040eb | bm-deploy-ramdisk | ari | 318648193 | active |
+————————————–+——————-+————-+———–+——–+
~
#################
tripleo.sh — Overcloud images – DONE.
#################
#################
[stack@instack ~]$ ./tripleo-ci/scripts/tripleo.sh –register-nodes
#################
tripleo.sh — Register nodes
#################
You must source a stackrc file for the Undercloud.
Attempting to source /home/stack/stackrc
Done
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Started Mistral Workflow. Execution ID: 9a148b8b-fe55-43b1-b3e1-cb13fad49759
Successfully registered node UUID 1f031f3f-edb1-434c-8b6f-c60bffce9941
Successfully registered node UUID cca63d2d-6912-4878-9ea7-a90510fc09b2
Successfully registered node UUID 584bb979-b715-4c08-836f-2200c6d4d937
Started Mistral Workflow. Execution ID: 071e4b85-2b7c-420e-96bd-bbbe980f9db7
Successfully set all nodes to available.
+————————————–+——+—————+————-+——————–+————-+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+————————————–+——+—————+————-+——————–+————-+
| 1f031f3f-edb1-434c-8b6f-c60bffce9941 | None | None | power off | available | False |
| cca63d2d-6912-4878-9ea7-a90510fc09b2 | None | None | power off | available | False |
| 584bb979-b715-4c08-836f-2200c6d4d937 | None | None | power off | available | False |
+————————————–+——+—————+————-+——————–+————-+
#################
tripleo.sh — Register nodes – DONE.
#################
[stack@instack ~]$ ./tripleo-ci/scripts/tripleo.sh –introspect-nodes
#################
tripleo.sh — Introspect nodes
#################
You must source a stackrc file for the Undercloud.
Attempting to source /home/stack/stackrc
Done
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Setting nodes for introspection to manageable…
Starting introspection of manageable nodes
Started Mistral Workflow. Execution ID: e4e63d1a-3e6c-42d5-9575-e4166853cdd0
Waiting for introspection to finish…
Introspection for UUID 1f031f3f-edb1-434c-8b6f-c60bffce9941 finished successfully.
Introspection for UUID cca63d2d-6912-4878-9ea7-a90510fc09b2 finished successfully.
Introspection for UUID 584bb979-b715-4c08-836f-2200c6d4d937 finished successfully.
Introspection completed.
Setting manageable nodes to available…
Started Mistral Workflow. Execution ID: 1d119a65-a5a8-4b81-b5da-2fd3b15f26e1

#################
tripleo.sh — Introspect nodes – DONE.
#################

Now create external interface vlan10.

[stack@instack ~]$ sudo vi /etc/sysconfig/network-scripts/ifcfg-vlan10

DEVICE=vlan10
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSIntPort
BOOTPROTO=static
IPADDR=10.0.0.1
NETMASK=255.255.255.0
OVS_BRIDGE=br-ctlplane
OVS_OPTIONS=”tag=10″

[stack@instack ~]$ sudo ifup vlan10

[stack@instack ~]$ sudo ovs-vsctl show
43ccb3e7-74ed-4192-a87d-80b5a71a7e80
Manager “ptcp:6640:127.0.0.1”
is_connected: true
Bridge br-int
Controller “tcp:127.0.0.1:6633”
is_connected: true
fail_mode: secure
Port int-br-ctlplane
Interface int-br-ctlplane
type: patch
options: {peer=phy-br-ctlplane}
Port br-int
Interface br-int
type: internal
Port “tap0d0fb165-79”
tag: 1
Interface “tap0d0fb165-79”
type: internal
Bridge br-ctlplane
Controller “tcp:127.0.0.1:6633”
is_connected: true
fail_mode: secure
Port “eth1”
Interface “eth1”
Port br-ctlplane
Interface br-ctlplane
type: internal
Port “vlan10”
tag: 10
Interface “vlan10”
type: internal
Port phy-br-ctlplane
Interface phy-br-ctlplane
type: patch
options: {peer=int-br-ctlplane}
ovs_version: “2.5.0”
[stack@instack ~]$ ifconfig
br-ctlplane: flags=4163 mtu 1500
inet 192.0.2.1 netmask 255.255.255.0 broadcast 192.0.2.255
inet6 fe80::283:bbff:feda:c642 prefixlen 64 scopeid 0x20
ether 00:83:bb:da:c6:42 txqueuelen 0 (Ethernet)
RX packets 43022 bytes 2956223 (2.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 15525 bytes 972453334 (927.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4163 mtu 1500
inet 192.168.122.90 netmask 255.255.255.0 broadcast 192.168.122.255
inet6 fe80::5054:ff:fe28:530d prefixlen 64 scopeid 0x20
ether 52:54:00:28:53:0d txqueuelen 1000 (Ethernet)
RX packets 881966 bytes 1281751784 (1.1 GiB)
RX errors 0 dropped 3 overruns 0 frame 0
TX packets 539560 bytes 43216702 (41.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth1: flags=4163 mtu 1500
inet6 fe80::283:bbff:feda:c642 prefixlen 64 scopeid 0x20
ether 00:83:bb:da:c6:42 txqueuelen 1000 (Ethernet)
RX packets 43015 bytes 2955825 (2.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 15538 bytes 972454368 (927.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 0 (Local Loopback)
RX packets 867304 bytes 4826379602 (4.4 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 867304 bytes 4826379602 (4.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vlan10: flags=4163 mtu 1500
inet 10.0.0.1 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::dc07:f5ff:fe72:2c9 prefixlen 64 scopeid 0x20
ether de:07:f5:72:02:c9 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12 bytes 816 (816.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Instack IP is 192.168.122.90
[stack@instack ~]$ vi network_env.yaml
{
“parameter_defaults”: {
“ControlPlaneDefaultRoute”: “192.0.2.1”,
“ControlPlaneSubnetCidr”: “24”,
“DnsServers”: [
“192.168.122.90”
],
“EC2MetadataIp”: “192.0.2.1”,
“ExternalAllocationPools”: [
{
“end”: “10.0.0.250”,
“start”: “10.0.0.4”
}
],
“ExternalNetCidr”: “10.0.0.1/24”,
“NeutronExternalNetworkBridge”: “”
}
}

[stack@instack ~]$ sudo iptables -A BOOTSTACK_MASQ -s 10.0.0.0/24 ! -d 10.0.0.0/24 -j MASQUERADE -t nat
[stack@instack ~]$ vi overcloud-deploy.sh
#!/bin/bash -x
source /home/stack/stackrc
openstack overcloud deploy \
–libvirt-type qemu \
–ntp-server pool.ntp.org \
–templates /home/stack/tripleo-heat-templates \
-e /home/stack/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml \
-e /home/stack/tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
-e $HOME/network_env.yaml \
–control-scale 1 –compute-scale 2

[stack@instack ~]$ cat $HOME/network_env.yaml
[stack@instack ~]$ chmod a+x overcloud-deploy.sh
[stack@instack ~]$ touch -f /home/stack/tripleo-heat-templates/puppet/post.yaml

[stack@instack ~]$ neutron subnet-list
+————————————+——+————–+————————————+
| id | name | cidr | allocation_pools |
+————————————+——+————–+————————————+
| bc762c84-558a-4091-aeca- | | 192.0.2.0/24 | {“start”: “192.0.2.5”, “end”: |
| b0b1a428e5f1 | | | “192.0.2.24”} |
+————————————+——+————–+————————————+
[stack@instack ~]$ neutron subnet-update bc762c84-558a-4091-aeca-b0b1a428e5f1 –dns-nameserver 83.221.202.254
Updated subnet: bc762c84-558a-4091-aeca-b0b1a428e5f1
[stack@instack ~]$ chmod a+x overcloud-deploy.sh
[stack@instack ~]$ ./overcloud-deploy.sh
+ source /home/stack/stackrc
++ export NOVA_VERSION=1.1
++ NOVA_VERSION=1.1
+++ sudo hiera admin_password
++ export OS_PASSWORD=05440e148840c9bb860584d4e9c42fe573096409
++ OS_PASSWORD=05440e148840c9bb860584d4e9c42fe573096409
++ export OS_AUTH_URL=http://192.0.2.1:5000/v2.0
++ OS_AUTH_URL=http://192.0.2.1:5000/v2.0
++ export OS_USERNAME=admin
++ OS_USERNAME=admin
++ export OS_TENANT_NAME=admin
++ OS_TENANT_NAME=admin
++ export COMPUTE_API_VERSION=1.1
++ COMPUTE_API_VERSION=1.1
++ export OS_BAREMETAL_API_VERSION=1.15
++ OS_BAREMETAL_API_VERSION=1.15
++ export OS_NO_CACHE=True
++ OS_NO_CACHE=True
++ export OS_CLOUDNAME=undercloud
++ OS_CLOUDNAME=undercloud
++ export OS_IMAGE_API_VERSION=1
++ OS_IMAGE_API_VERSION=1
+ openstack overcloud deploy –libvirt-type qemu \
–ntp-server pool.ntp.org –templates \
/home/stack/tripleo-heat-templates \
-e /home/stack/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml\
-e /home/stack/tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
-e /home/stack/network_env.yaml \
–control-scale 1 –compute-scale 2

WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: 08b899f5-0444-4eb8-8719-a6eba9a81fa0
Plan updated
Deploying templates in the directory /home/stack/tripleo-heat-templates
Started Mistral Workflow. Execution ID: 1c291c68-aec7-49e8-836f-658b06763c92
2016-09-27 14:09:37Z [overcloud]: CREATE_IN_PROGRESS Stack CREATE started
2016-09-27 14:09:37Z [MysqlRootPassword]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:37Z [RabbitCookie]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:37Z [HorizonSecret]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:37Z [HeatAuthEncryptionKey]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:37Z [Networks]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:37Z [PcsdPassword]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [ServiceNetMap]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [overcloud-Networks-oqxpnzrtiaem]: CREATE_IN_PROGRESS Stack CREATE started
2016-09-27 14:09:38Z [InternalNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [ManagementNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [StorageMgmtNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [overcloud-Networks-oqxpnzrtiaem-InternalNetwork-l4jjn3botrgj]: CREATE_IN_PROGRESS Stack CREATE started
2016-09-27 14:09:38Z [InternalApiNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [RabbitCookie]: CREATE_COMPLETE state changed
2016-09-27 14:09:38Z [StorageNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [HorizonSecret]: CREATE_COMPLETE state changed
2016-09-27 14:09:38Z [PcsdPassword]: CREATE_COMPLETE state changed
2016-09-27 14:09:38Z [overcloud-Networks-oqxpnzrtiaem-StorageMgmtNetwork-jnuiofatb5tu]: CREATE_IN_PROGRESS Stack CREATE started
2016-09-27 14:09:38Z [HeatAuthEncryptionKey]: CREATE_COMPLETE state changed
2016-09-27 14:09:38Z [MysqlRootPassword]: CREATE_COMPLETE state changed
2016-09-27 14:09:38Z [StorageMgmtNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [TenantNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [DefaultPasswords]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [overcloud-Networks-oqxpnzrtiaem-StorageNetwork-x3pafjtlirmz]: CREATE_IN_PROGRESS Stack CREATE started
2016-09-27 14:09:38Z [StorageNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [ExternalNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [overcloud-Networks-oqxpnzrtiaem-TenantNetwork-2nndp6au2sfp]: CREATE_IN_PROGRESS Stack CREATE started
2016-09-27 14:09:38Z [TenantNetwork]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [NetworkExtraConfig]: CREATE_IN_PROGRESS state changed
2016-09-27 14:09:38Z [overcloud-Networks-oqxpnzrtiaem-ExternalNetwork-ht7dkpzmiskb]: CREATE_IN_PROGRESS Stack CREATE started
2016-09-27 14:09:38Z [ManagementNetwork]: CREATE_COMPLETE state changed

. . . . . .

2016-09-27 14:38:22Z [0]: CREATE_COMPLETE state changed
2016-09-27 14:38:22Z [overcloud-AllNodesDeploySteps-zs2vx53nvajt-ControllerDeployment_Step5-is6nnvfdpzxg]: CREATE_COMPLETE Stack CREATE completed successfully
2016-09-27 14:38:22Z [ControllerDeployment_Step5]: CREATE_COMPLETE state changed
2016-09-27 14:38:22Z [BlockStoragePostConfig]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:22Z [ObjectStoragePostConfig]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:22Z [ComputePostConfig]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:22Z [CephStoragePostConfig]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:22Z [ControllerPostConfig]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:23Z [BlockStoragePostConfig]: CREATE_COMPLETE state changed
2016-09-27 14:38:23Z [ObjectStoragePostConfig]: CREATE_COMPLETE state changed
2016-09-27 14:38:23Z [ComputePostConfig]: CREATE_COMPLETE state changed
2016-09-27 14:38:24Z [CephStoragePostConfig]: CREATE_COMPLETE state changed
2016-09-27 14:38:24Z [ControllerPostConfig]: CREATE_COMPLETE state changed
2016-09-27 14:38:24Z [CephStorageExtraConfigPost]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:24Z [ComputeExtraConfigPost]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:24Z [BlockStorageExtraConfigPost]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:24Z [ControllerExtraConfigPost]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:24Z [ObjectStorageExtraConfigPost]: CREATE_IN_PROGRESS state changed
2016-09-27 14:38:25Z [ComputeExtraConfigPost]: CREATE_COMPLETE state changed
2016-09-27 14:38:25Z [CephStorageExtraConfigPost]: CREATE_COMPLETE state changed
2016-09-27 14:38:25Z [ControllerExtraConfigPost]: CREATE_COMPLETE state changed
2016-09-27 14:38:25Z [ObjectStorageExtraConfigPost]: CREATE_COMPLETE state changed
2016-09-27 14:38:25Z [BlockStorageExtraConfigPost]: CREATE_COMPLETE state changed
2016-09-27 14:38:25Z [overcloud-AllNodesDeploySteps-zs2vx53nvajt]: CREATE_COMPLETE Stack CREATE completed successfully
2016-09-27 14:38:26Z [AllNodesDeploySteps]: CREATE_COMPLETE state changed
2016-09-27 14:38:26Z [overcloud]: CREATE_COMPLETE Stack CREATE completed successfully

Stack overcloud CREATE_COMPLETE

Overcloud Endpoint: http://10.0.0.4:5000/v2.0
Overcloud Deployed

***********************************************************
Checking for errors in mistral execution list
which is new in Newton release
***********************************************************

[stack@instack ~]$ mistral execution-list

+----------+-------------+---------------+-------------+-------------------+---------+------------+------------+---------------+
| ID       | Workflow ID | Workflow name | Description | Task Execution ID | State   | State info | Created at | Updated at    |
+----------+-------------+---------------+-------------+-------------------+---------+------------+------------+---------------+
| 5bce3202 | bde1cc99-ef | tripleo.plan_ |             |             | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -c2b3-47 | e5-40df-    | management.v1 |             |                   |         |            | 12:31:44   | 12:31:58      |
| 35-ad40- | bc5a-359141 | .create_defau |             |                   |         |            |            |               |
| 2b35e775 | e48a73      | lt_deployment |             |                   |         |            |            |               |
| 4647     |             | _plan         |             |                   |         |            |            |               |
| 9a148b8b | 75fb3808    | tripleo.barem |             |             | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -fe55-43 | -142c-4d98- | etal.v1.regis |             |                   |         |            | 13:40:34   | 13:40:49      |
| b1-b3e1- | a509-edfcee | ter_or_update |             |                   |         |            |            |               |
| cb13fad4 | 056fe5      |               |             |                   |         |            |            |               |
| 9759     |             |               |             |                   |         |            |            |               |
| 0a450c25 | bcc2d68d-cd | tripleo.barem | sub-        | 191d237c-1322     | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -bf68    | da-4919-998 | etal.v1.set_n | workflow    | -4b3c-867e-       |         |            | 13:40:42   | 13:40:45      |
| -487a-af | 4-00e03bbdf | ode_state     | execution   | ddbd7121e01f      |         |            |            |               |
| 98-7125e | db8         |               |             |                   |         |            |            |               |
| 809a737  |             |               |             |                   |         |            |            |               |
| 3e8eff39 | bcc2d68d-cd | tripleo.barem | sub-        | 191d237c-1322     | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -f732-49 | da-4919-998 | etal.v1.set_n | workflow    | -4b3c-867e-       |         |            | 13:40:42   | 13:40:45      |
| 76-b541- | 4-00e03bbdf | ode_state     | execution   | ddbd7121e01f      |         |            |            |               |
| b3446c6c | db8         |               |             |                   |         |            |            |               |
| 6fde     |             |               |             |                   |         |            |            |               |
| 55385132 | bcc2d68d-cd | tripleo.barem | sub-        | 191d237c-1322     | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -6e79    | da-4919-998 | etal.v1.set_n | workflow    | -4b3c-867e-       |         |            | 13:40:42   | 13:40:45      |
| -400b-   | 4-00e03bbdf | ode_state     | execution   | ddbd7121e01f      |         |            |            |               |
| 965d-1a2 | db8         |               |             |                   |         |            |            |               |
| f6ccada2 |             |               |             |                   |         |            |            |               |
| 4        |             |               |             |                   |         |            |            |               |
| 071e4b85 | 22cd7376    | tripleo.barem |             |             | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -2b7c-   | -d6cd-49db- | etal.v1.provi |             |                   |         |            | 13:40:49   | 13:40:56      |
| 420e-    | 801b-74ef4e | de            |             |                   |         |            |            |               |
| 96bd-bbb | 197f3f      |               |             |                   |         |            |            |               |
| e980f9db |             |               |             |                   |         |            |            |               |
| 7        |             |               |             |                   |         |            |            |               |
| 288e2ff2 | bcc2d68d-cd | tripleo.barem | sub-        | 52352bbc-8667-458 | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -2161    | da-4919-998 | etal.v1.set_n | workflow    | 5-8099-39f6738381 |         |            | 13:40:49   | 13:40:52      |
| -473f-84 | 4-00e03bbdf | ode_state     | execution   | 5a                |         |            |            |               |
| c9-382bd | db8         |               |             |                   |         |            |            |               |
| 7b6b0bd  |             |               |             |                   |         |            |            |               |
| 28e41b0a | bcc2d68d-cd | tripleo.barem | sub-        | 52352bbc-8667-458 | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -eb5f-4e | da-4919-998 | etal.v1.set_n | workflow    | 5-8099-39f6738381 |         |            | 13:40:49   | 13:40:52      |
| 70-b537- | 4-00e03bbdf | ode_state     | execution   | 5a                |         |            |            |               |
| e5da84e2 | db8         |               |             |                   |         |            |            |               |
| af7b     |             |               |             |                   |         |            |            |               |
| 2c30a482 | bcc2d68d-cd | tripleo.barem | sub-        | 52352bbc-8667-458 | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -a4fc-   | da-4919-998 | etal.v1.set_n | workflow    | 5-8099-39f6738381 |         |            | 13:40:49   | 13:40:52      |
| 42be-    | 4-00e03bbdf | ode_state     | execution   | 5a                |         |            |            |               |
| a66b-568 | db8         |               |             |                   |         |            |            |               |
| b5bb7fa5 |             |               |             |                   |         |            |            |               |
| 0        |             |               |             |                   |         |            |            |               |
| 2026ebb3 | a3b2b56e-   | tripleo.barem | sub-        | 68dfd5f3-ba31-432 | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -2cf9-43 | 0a12-4354   | etal.v1.intro | workflow    | 1-9b0e-           |         |            | 13:41:12   | 13:43:19      |
| 90-90fc- | -b7de-2cbaa | spect         | execution   | 1d35b5181c49      |         |            |            |               |
| 05688171 | c7b3406     |               |             |                   |         |            |            |               |
| 9836     |             |               |             |                   |         |            |            |               |
| e4e63d1a | 4130daad-38 | tripleo.barem |             |             | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -3e6c-42 | 0d-4406-b02 | etal.v1.intro |             |                   |         |            | 13:41:12   | 13:43:22      |
| d5-9575- | 1-9e8b4bb3b | spect_managea |             |                   |         |            |            |               |
| e4166853 | 0e1         | ble_nodes     |             |                   |         |            |            |               |
| cdd0     |             |               |             |                   |         |            |            |               |
| 1d119a65 | 29c51fab-   | tripleo.barem |             |             | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -a5a8-4b | f6b9-4cc1   | etal.v1.provi |             |                   |         |            | 13:43:21   | 13:43:33      |
| 81-b5da- | -94bc-14144 | de_manageable |             |                   |         |            |            |               |
| 2fd3b15f | bc8284f     | _nodes        |             |                   |         |            |            |               |
| 26e1     |             |               |             |                   |         |            |            |               |
| 9262a91e | 22cd7376    | tripleo.barem | sub-        | 8052fcbb-b511     | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -a7bf-   | -d6cd-49db- | etal.v1.provi | workflow    | -423a-bfdc-       |         |            | 13:43:22   | 13:43:30      |
| 439b-    | 801b-74ef4e | de            | execution   | 8555b8e169a8      |         |            |            |               |
| b9fc-e48 | 197f3f      |               |             |                   |         |            |            |               |
| ff49f5aa |             |               |             |                   |         |            |            |               |
| d        |             |               |             |                   |         |            |            |               |
| ab696f08 | bcc2d68d-cd | tripleo.barem | sub-        | da2c83ad-30bf-425 | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -76a8    | da-4919-998 | etal.v1.set_n | workflow    | 1-8859-20c8d9b895 |         |            | 13:43:22   | 13:43:25      |
| -478c-8c | 4-00e03bbdf | ode_state     | execution   | cb                |         |            |            |               |
| d2-7b5bb | db8         |               |             |                   |         |            |            |               |
| 3f1fba0  |             |               |             |                   |         |            |            |               |
| 77ede1a7 | bcc2d68d-cd | tripleo.barem | sub-        | da2c83ad-30bf-425 | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -4167-48 | da-4919-998 | etal.v1.set_n | workflow    | 1-8859-20c8d9b895 |         |            | 13:43:23   | 13:43:26      |
| 95-b786- | 4-00e03bbdf | ode_state     | execution   | cb                |         |            |            |               |
| 4ff65ce7 | db8         |               |             |                   |         |            |            |               |
| 1c6b     |             |               |             |                   |         |            |            |               |
| feec4ba0 | bcc2d68d-cd | tripleo.barem | sub-        | da2c83ad-30bf-425 | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -7331    | da-4919-998 | etal.v1.set_n | workflow    | 1-8859-20c8d9b895 |         |            | 13:43:23   | 13:43:26      |
| -4adb-8a | 4-00e03bbdf | ode_state     | execution   | cb                |         |            |            |               |
| 08-352c3 | db8         |               |             |                   |         |            |            |               |
| 151965c  |             |               |             |                   |         |            |            |               |
| 08b899f5 | 9210744f-   | tripleo.plan_ |             |             | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -0444-4e | 42f1-45de-  | management.v1 |             |                   |         |            | 14:09:19   | 14:09:26      |
| b8-8719- | 902a-a1e0a1 | .update_deplo |             |                   |         |            |            |               |
| a6eba9a8 | 4f91bb      | yment_plan    |             |                   |         |            |            |               |
| 1fa0     |             |               |             |                   |         |            |            |               |
| 1c291c68 | feef43e7-28 | tripleo.deplo |             |             | SUCCESS | None       | 2016-09-27 | 2016-09-27    |
| -aec7-49 | 65-4123-b0e | yment.v1.depl |             |                   |         |            | 14:09:26   | 14:09:40      |
| e8-836f- | 9-f4eaf6d5d | oy_plan       |             |                   |         |            |            |               |
| 658b0676 | 77d         |               |             |                   |         |            |            |               |
| 3c92     |             |               |             |                   |         |            |            |               |
+----------+-------------+---------------+-------------+-------------------+---------+------------+------------+---------------+

[stack@instack ~]$ sudo route add -net 192.0.2.0/24 gw 192.0.2.1 <=== No longer needed as of 09/29/2016
[stack@instack ~]$ sudo route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.122.1 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 vlan10
192.0.2.0 192.0.2.1 255.255.255.0 UG 0 0 0 br-ctlplane
192.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 br-ctlplane
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

[root@overcloud-controller-0 ~]# nova service-list
+—-+——————+————————————-+———-+———+——-+—————————-+—————–+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+—-+——————+————————————-+———-+———+——-+—————————-+—————–+
| 3 | nova-consoleauth | overcloud-controller-0.localdomain | internal | enabled | up | 2016-09-27T14:54:34.000000 | – |
| 4 | nova-scheduler | overcloud-controller-0.localdomain | internal | enabled | up | 2016-09-27T14:54:35.000000 | – |
| 5 | nova-conductor | overcloud-controller-0.localdomain | internal | enabled | up | 2016-09-27T14:54:27.000000 | – |
| 6 | nova-compute | overcloud-novacompute-0.localdomain | nova | enabled | up | 2016-09-27T14:54:26.000000 | – |
| 7 | nova-compute | overcloud-novacompute-1.localdomain | nova | enabled | up | 2016-09-27T14:54:27.000000 | – |
+—-+——————+————————————-+———-+———+——-+—————————-+—————–+

****************************************************************************************************

Verification status VIP 10.0.0.4 via keepalived status on overcloud-controller-0.localdomain

****************************************************************************************************
[root@overcloud-controller-0 ~]# systemctl status keepalived -l
● keepalived.service – LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2016-09-27 14:55:01 UTC; 2s ago
Process: 18503 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 850 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/keepalived.service
├─18504 /usr/sbin/keepalived -D
├─18505 /usr/sbin/keepalived -D
└─18506 /usr/sbin/keepalived -D

Sep 27 14:55:03 overcloud-controller-0 Keepalived_healthcheckers[18505]: Netlink reflector reports IP 192.0.2.13 added
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]: VRRP_Instance(51) Sending gratuitous ARPs on br-ex for 192.0.2.13
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]: VRRP_Instance(56) Entering MASTER STATE
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]: VRRP_Instance(56) setting protocol VIPs.
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]: VRRP_Instance(56) Sending gratuitous ARPs on vlan20 for 172.16.2.7
Sep 27 14:55:03 overcloud-controller-0 Keepalived_healthcheckers[18505]: Netlink reflector reports IP 172.16.2.7 added
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]: VRRP_Instance(52) Entering MASTER STATE
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]: VRRP_Instance(52) setting protocol VIPs.
Sep 27 14:55:03 overcloud-controller-0 Keepalived_vrrp[18506]: VRRP_Instance(52) Sending gratuitous ARPs on br-ex for 10.0.0.4
Sep 27 14:55:03 overcloud-controller-0 Keepalived_healthcheckers[18505]: Netlink reflector reports IP 10.0.0.4 added

screenshot-from-2016-09-27-18-09-14screenshot-from-2016-09-27-18-42-42screenshot-from-2016-09-27-18-53-14screenshot-from-2016-09-27-20-22-50

********************************
Compute Node Status
********************************

[heat-admin@overcloud-novacompute-0 ~]$ sudo su -
Last login: Tue Sep 27 15:30:34 UTC 2016 on pts/0
[root@overcloud-novacompute-0 ~]# virsh --connect qemu:///system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # version
Compiled against library: libvirt 1.2.17
Using library: libvirt 1.2.17
Using API: QEMU 1.2.17
Running hypervisor: QEMU 2.3.0

virsh # list --all
 Id    Name                           State
----------------------------------------------------
 3     instance-00000001              running 

screenshot-from-2016-09-27-19-27-51screenshot-from-2016-09-27-19-35-27


TripleO deployment of ‘master’ branch via instack-virt-setup

September 16, 2016

UPDATE 09/23/2016

Fix released for (1622683, 1622720 ) in :-
https://bugs.launchpad.net/tripleo/+bug/1622683 
****************************************************
Deploy completed OK the first time
****************************************************

2016-09-23 09:08:28Z [overcloud-AllNodesDeploySteps-yrsd7pkitjij]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-09-23 09:08:28Z [AllNodesDeploySteps]: CREATE_COMPLETE  state changed
2016-09-23 09:08:28Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully
Stack overcloud CREATE_COMPLETE

Overcloud Endpoint: http://10.0.0.6:5000/v2.0
Overcloud Deployed

[stack@instack ~]$ nova list

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| b3d97bcf-9318-48ef-91c7-09c8386a75aa | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| 148aa223-513d-44d5-b865-2cb2c3dcbc6f | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| e3ee61fb-c243-4454-949d-84c22e66b147 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+

[stack@instack ~]$ mistral environment-list

+-----------+-------------+---------+---------------------+---------------------+
| Name      | Description | Scope   | Created at          | Updated at          |
+-----------+-------------+---------+---------------------+---------------------+
| overcloud | None        | private | 2016-09-23 07:33:40 | 2016-09-23 08:41:29 |
+-----------+-------------+---------+---------------------+---------------------+

[stack@instack ~]$ swift list
ov-jjf6fn4qyjt-0-gfpul73m4fdl-Controller-dekw3w5stcqd
ov-pb3uu5djue-0-lmazr26t3z4u-NovaCompute-sqfaz5lstqov
ov-pb3uu5djue-1-7prlyxolsdhd-NovaCompute-ltmkwmq74iyq
overcloud

[stack@instack ~]$ openstack stack delete overcloud
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Are you sure you want to delete this stack(s) [y/N]? y
[stack@instack ~]$ openstack stack list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils

+---------------------+------------+--------------------+----------------------+--------------+
| ID                  | Stack Name | Stack Status       | Creation Time        | Updated Time |
+---------------------+------------+--------------------+----------------------+--------------+
| 6e3ae2b6-5ce1-45db- | overcloud  | DELETE_IN_PROGRESS | 2016-09-23T08:41:38Z | None         |
| bde5-06d2ce2e571b   |            |                    |                      |              |
+---------------------+------------+--------------------+----------------------+--------------+

[stack@instack ~]$ openstack stack list

WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils

***************************************************************************
Empty output – overcloud stack has been deleted
****************************************************************************

[stack@instack ~]$ mistral environment-list

+-----------+-------------+---------+---------------------+---------------------+
| Name      | Description | Scope   | Created at          | Updated at          |
+-----------+-------------+---------+---------------------+---------------------+
| overcloud | None        | private | 2016-09-23 07:33:40 | 2016-09-23 08:41:29 |
+-----------+-------------+---------+---------------------+---------------------+

[stack@instack ~]$ swift list

overcloud

******************************************************************************
Now attempt to redeploy the second time .  Success on 09/23/2016
******************************************************************************

[stack@instack ~]$ touch -f  /home/stack/tripleo-heat-templates/puppet/post.yaml
[stack@instack ~]$ ./overcloud-deploy.sh
+ source /home/stack/stackrc
++ export NOVA_VERSION=1.1

++ NOVA_VERSION=1.1
+++ sudo hiera admin_password
++ export OS_PASSWORD=68a350a2972f7ff9e88d0e9ea79056b3e0bb90ec
++ OS_PASSWORD=68a350a2972f7ff9e88d0e9ea79056b3e0bb90ec
++ export OS_AUTH_URL=http://192.0.2.1:5000/v2.0
++ OS_AUTH_URL=http://192.0.2.1:5000/v2.0
++ export OS_USERNAME=admin
++ OS_USERNAME=admin
++ export OS_TENANT_NAME=admin
++ OS_TENANT_NAME=admin
++ export COMPUTE_API_VERSION=1.1
++ COMPUTE_API_VERSION=1.1
++ export OS_BAREMETAL_API_VERSION=1.15
++ OS_BAREMETAL_API_VERSION=1.15
++ export OS_NO_CACHE=True
++ OS_NO_CACHE=True
++ export OS_CLOUDNAME=undercloud
++ OS_CLOUDNAME=undercloud
++ export OS_IMAGE_API_VERSION=1
++ OS_IMAGE_API_VERSION=1
+ openstack overcloud deploy –libvirt-type qemu –ntp-server pool.ntp.org –templates /home/stack/tripleo-heat-templates -e /home/stack/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /home/stack/tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network_env.yaml –control-scale 1 –compute-scale 2

WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: 4d744a89-a2e7-43a5-82af-26bab11e6342
Plan updated
Deploying templates in the directory /home/stack/tripleo-heat-templates
Object GET failed: http://192.0.2.1:8080/v1/AUTH_7ea6220c67c84c828f4249b95886259f/overcloud/overcloud-without-mergepy.yaml 404 Not Found  [first 60 chars of response]

Started Mistral Workflow. Execution ID: 807a7047-a1c3-4686-9be7-11d73e72dfb8
2016-09-23 09:15:34Z [overcloud]: CREATE_IN_PROGRESS  Stack CREATE started
2016-09-23 09:15:34Z [HorizonSecret]: CREATE_IN_PROGRESS  state changed
2016-09-23 09:15:34Z [RabbitCookie]: CREATE_IN_PROGRESS  state changed
2016-09-23 09:15:35Z [ServiceNetMap]: CREATE_IN_PROGRESS  state changed
2016-09-23 09:15:35Z [MysqlRootPassword]: CREATE_IN_PROGRESS  state changed
2016-09-23 09:15:35Z [HeatAuthEncryptionKey]: CREATE_IN_PROGRESS  state changed
2016-09-23 09:15:35Z [PcsdPassword]: CREATE_IN_PROGRESS  state changed
2016-09-23 09:15:35Z [Networks]: CREATE_IN_PROGRESS  state changed
2016-09-23 09:15:35Z [ServiceNetMap]: CREATE_COMPLETE  state changed
2016-09-23 09:15:35Z [RabbitCookie]: CREATE_COMPLETE  state changed
2016-09-23 09:15:35Z [HeatAuthEncryptionKey]: CREATE_COMPLETE  state changed
2016-09-23 09:15:35Z [PcsdPassword]: CREATE_COMPLETE  state changed
2016-09-23 09:15:35Z [HorizonSecret]: CREATE_COMPLETE  state changed

. . . . . .

2016-09-23 09:39:50Z [BlockStorageExtraConfigPost]: CREATE_COMPLETE  state changed
2016-09-23 09:39:51Z [CephStorageExtraConfigPost]: CREATE_COMPLETE  state changed
2016-09-23 09:39:51Z [ComputeExtraConfigPost]: CREATE_COMPLETE  state changed
2016-09-23 09:39:51Z [ObjectStorageExtraConfigPost]: CREATE_COMPLETE  state changed
2016-09-23 09:39:51Z [overcloud-AllNodesDeploySteps-5bfecsxdagiz]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-09-23 09:39:51Z [AllNodesDeploySteps]: CREATE_COMPLETE  state changed
2016-09-23 09:39:51Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

Stack overcloud CREATE_COMPLETE
Overcloud Endpoint: http://10.0.0.12:5000/v2.0
Overcloud Deployed

END UPDATE

UPDATE 09/21/2016

Work around for 1622720 which allows redeploy second time

During run time :-
[stack@instack ~]$ mistral environment-list
+———–+————-+———+———————+———————+
| Name | Description | Scope | Created at | Updated at |
+———–+————-+———+———————+———————+
| overcloud | None | private | 2016-09-21 12:35:43 | 2016-09-21 12:35:51 |
+———–+————-+———+———————+———————+

[stack@instack ~]$ swift list
ov-a2o6ekfrck5-0-zesuo2wtu2ed-Controller-ushkojdgxsim
ov-yfn5tgwipf-0-jebdxn5jfduz-NovaCompute-4hjdhzij3czv
ov-yfn5tgwipf-1-vypbavbviwxv-NovaCompute-luo274m3kmn2
overcloud

Here is snapshot which is bug evidence

Next step is workaround itself per https://bugs.launchpad.net/tripleo/+bug/1622720/comments/1  :-

[stack@instack ~]$ mistral environment-delete overcloud
Request to delete environment overcloud has been accepted.
[stack@instack ~]$ swift delete –all

$ touch -f /home/stack/tripleo-heat-templates/puppet/post.yaml
$ overcloud-deploy.sh

See following bugs at Launchpad :-

https://bugs.launchpad.net/tripleo/+bug/1622720
https://bugs.launchpad.net/tripleo/+bug/1622683
https://bugs.launchpad.net/tripleo/+bug/1622720/comments/2

END UPDATE

Due to Launchpad Bug  introspection hangs due to broken ipxe config
finally resolved on 09/01/2016  approach suggested in
TripleO manual deployment of ‘master’ branch by Carlo Camacho
has been retested.  As appears things in meantime have been changed. Following bellow is the way how mentioned above post worked for me right now on 32 GB VIRTHOST (i7 4790)

*****************************************
Tune stack environment on VIRTHOST
*****************************************

# useradd stack
# echo “stack:stack” | chpasswd
# echo “stack ALL=(root) NOPASSWD:ALL” | sudo tee -a /etc/sudoers.d/stack
#  chmod 0440 /etc/sudoers.d/stack
#  su – stack

***************************
Tune stack ENV
**************************

export NODE_DIST=centos7
export NODE_CPU=2
export NODE_MEM=7550
export NODE_COUNT=2
export UNDERCLOUD_NODE_CPU=2
export UNDERCLOUD_NODE_MEM=9000
export FS_TYPE=ext4

****************************************************************

Re-login to stack (highlight long line and copy if needed)

****************************************************************
$ sudo yum -y install epel-release sudo
$ yum -y install yum-plugin-priorities
$ sudo curl -o /etc/yum.repos.d/delorean.repo  http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/delorean.repo
$ sudo curl -o /etc/yum.repos.d/delorean-deps.repo  http://trunk.rdoproject.org/centos7/delorean-deps.repo
$ sudo yum install -y instack-undercloud
$ instack-virt-setup

*********************

On instack VM

*********************

Create swap file per http://www.anstack.com/blog/2016/07/04/manually-installing-tripleo-recipe.html  :-

#Add a 4GB swap file to the Undercloud
sudo dd if=/dev/zero of=/swapfile bs=1024 count=4194304
sudo mkswap /swapfile
#Turn ON the swap file
sudo chmod 600 /swapfile
sudo swapon /swapfile
#Enable it on start
sudo echo “/swapfile   swap   swap    defaults        0 0” &gt;&gt; /etc/fstab

***************************
Restart instack VM
***************************
Next
su – stack
sudo yum -y install yum-plugin-priorities

*************************************
Update .bashrc under ~stack/
*************************************

export USE_DELOREAN_TRUNK=1
export DELOREAN_TRUNK_REPO="http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/"
export DELOREAN_REPO_FILE="delorean.repo"
export FS_TYPE=ext4

************************************

  Re-login to stack

************************************

$ git clone https://github.com/openstack/tripleo-heat-templates
$ git clone https://github.com/openstack-infra/tripleo-ci.git

$ ./tripleo-ci/scripts/tripleo.sh –repo-setup
$ ./tripleo-ci/scripts/tripleo.sh –undercloud
$  source stackrc
$ ./tripleo-ci/scripts/tripleo.sh –overcloud-images
$ ./tripleo-ci/scripts/tripleo.sh –register-nodes
$ ./tripleo-ci/scripts/tripleo.sh –introspect-nodes

************************************************

  Passing step affected by mentioned bug

************************************************

  $ ./tripleo-ci/scripts/tripleo.sh –overcloud-deploy

Issue at start up of Overcloud deployment

###########################################################################################
tripleo.sh — Overcloud create started.
###########################################################################################
See Launchpad bugs 1622720 1622683 status . UPDATE 09/17/2016 is providing links. Back porting patch https://review.openstack.org/gitweb?p=openstack/tripleo-common.git;a=patch;h=203460176750aeda6c0a2d39ce349ad827053b11
via rebuilding openstack-tripleo-common-5.0.1-0.20160917031337.15c97e6.el7.centos.src.rpm && re installing new rpm doesn’t work for me.
###########################################################################################
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
WARNING: openstackclient.common.exceptions is deprecated and will be removed after Jun 2017. Please use osc_lib.exceptions
Creating Swift container to store the plan
Creating plan from template files in: /usr/share/openstack-tripleo-heat-templates/
Plan created
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
Object GET failed: http://192.0.2.1:8080/v1/AUTH_b4438648a72446eca04d2d216261c373/overcloud/overcloud-without-mergepy.yaml 404 Not Found  [first 60 chars of response]


Finally overcloud gets deployed

****************************************************************************************

On instack VM  verified  https://bugs.launchpad.net/tripleo/+bug/1604770  #9
****************************************************************************************

[stack@instack ~]$ sudo su –
Last login: Thu Sep 15 16:19:07 UTC 2016 from 192.168.122.1 on pts/1
[root@instack ~]# rpm -qa \*ipxe\*
ipxe-roms-qemu-20160127-1.git6366fa7a.el7.noarch
ipxe-bootimgs-20160127-1.git6366fa7a.el7.noarch

[stack@instack ~]$ openstack stack list

WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils

+————————+————+—————–+———————-+————–+
| ID                     | Stack Name | Stack Status    | Creation Time        | Updated Time |
+————————+————+—————–+———————-+————–+
| 7657df62-da09-4c0f-    | overcloud  | CREATE_COMPLETE | 2016-09-15T14:48:49Z | None         |
| bbdb-b9c95bdad537      |            |                 |                      |        |
+————————+————+—————–+———————-+————–+

[stack@instack ~]$ nova list

+————————————–+————————-+——–+————+————-+———————+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+————————————–+————————-+——–+————+————-+———————+
| 400e1499-5e02-4c92-a41b-814918f0edc3 | overcloud-controller-0  | ACTIVE | –          | Running     | ctlplane=192.0.2.15 |
| 58f3591f-c72f-4d97-9278-a33b3f631248 | overcloud-novacompute-0 | ACTIVE | –          | Running     | ctlplane=192.0.2.6  |
+————————————–+————————-+——–+————+————-+———————+

Managing and fixes required in overcloud

********************************************************************
Fix IP on Compute node &amp;&amp; Open 6080 on Controller
********************************************************************

On Compute :-

[vnc]
vncserver_proxyclient_address=192.0.2.6
vncserver_listen=0.0.0.0
keymap=en-us
enabled=True
novncproxy_base_url=http://192.0.2.15:6080/vnc_auto.html &lt;===

On Controller

Add line to /etc/sysconfig/iptables
-A INPUT -p tcp -m multiport –dports 6080 -m comment –comment “novncproxy” -m state –state NEW -j ACCEPT
Save /etc/sysconfig/iptables

#service iptables restart

[root@overcloud-controller-0 ~(keystone_admin)]# netstat -antp | grep 6080

tcp        0      0 192.0.2.15:6080         0.0.0.0:*               LISTEN      8397/python2       
tcp        1      0 192.0.2.8:56080         192.0.2.8:8080          CLOSE_WAIT  11606/gnocchi-metri
tcp        0      0 192.0.2.15:6080         192.0.2.1:47598         ESTABLISHED 28260/python2
tcp        0      0 192.0.2.15:6000         192.0.2.15:36080        TIME_WAIT   –

[root@overcloud-controller-0 ~(keystone_admin)]# ps -ef | grep 8397

nova      8397     1  0 15:06 ?                 00:00:05 /usr/bin/python2 /usr/bin/nova-novncproxy –web /usr/share/novnc/
nova      28260  8397  3 17:37 ?           00:00:56 /usr/bin/python2 /usr/bin/nova-novncproxy –web /usr/share/novnc/
root       31149 23941  0 18:06 pts/0    00:00:00 grep –color=auto 8397

**********************************
Create flavors as follows
**********************************

[root@overcloud-controller-0 ~]# nova flavor-create “m2.small” 2  1000 20 1

+—-+———-+———–+——+———–+——+——-+————-+———–+
| ID | Name     | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+—-+———-+———–+——+———–+——+——-+————-+———–+
| 2  | m2.small | 1000      | 20   | 0         |      | 1     | 1.0         | True      |
+—-+———-+———–+——+———–+——+——-+————-+———–+

[root@overcloud-controller-0 ~]# nova flavor-list

+————————————–+———————+———–+——+———–+——+——-+————-+———–+
| ID                                   | Name                | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+————————————–+———————+———–+——+———–+——+——-+————-+———–+
| 1                                    | 500MB Tiny Instance | 500       | 1    | 0         |      | 1     | 1.0         | True      |
| 2                                    | m2.small            | 1000      | 20   | 0         |      | 1     | 1.0         | True      |
+————————————–+———————+———–+——+———–+——+——-+————-+———–+

[root@overcloud-controller-0 ~]# nova flavor-list

+—-+———————+———–+——+———–+——+——-+————-+———–+
| ID | Name                | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+—-+———————+———–+——+———–+——+——-+————-+———–+
| 1  | 500MB Tiny Instance | 500       | 1    | 0         |      | 1     | 1.0         | True      |
| 2  | m2.small            | 1000      | 20   | 0         |      | 1     | 1.0         | True      |
+—-+———————+———–+——+———–+——+——-+————-+———–+

[root@overcloud-controller-0 ~]# glance image-list

+————————————–+—————+
| ID                                   | Name          |
+————————————–+—————+
| c9faf86d-4a06-401a-839c-c5bd48ff704a | CirrOS34Cloud |
| 4bf6f43d-8cba-43d7-9e34-347cff2d4769 | UbuntuCloud   |
| 81e031b0-11b7-440b-946f-b8f9e3a83c95 | VF24Cloud     |
+————————————–+—————+

[root@overcloud-controller-0 ~]# neutron net-list

+————————————–+————–+—————————————-+
| id                                   | name         | subnets                                |
+————————————–+————–+—————————————-+
| 2d0ccb5f-0cc8-4710-819d-7c148137aea2 | public       | 795e0fea-0550-44e8-abf3-afd316cd7843   |
|                                      |              | 192.0.2.0/24                           |
| e2a9edb9-8e01-4e99-83b2-6c6e705967fe | demo_network | 56b70753-e776-4ce8-9b28-650431b43a63   |
|                                      |              | 50.0.0.0/24                            |
+————————————–+————–+—————————————-+

[root@overcloud-controller-0 ~]# nova boot –flavor 2 –key-name oskey09152016 \
–image 81e031b0-11b7-440b-946f-b8f9e3a83c95 \
–nic net-id=e2a9edb9-8e01-4e99-83b2-6c6e705967fe  VF24Devs05

+————————————–+————————————————–+
| Property                             | Value                                            |
+————————————–+————————————————–+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          |                                                  |
| OS-EXT-SRV-ATTR:host                 | –                                                |
| OS-EXT-SRV-ATTR:hostname             | vf24devs05                                       |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | –                                                |
| OS-EXT-SRV-ATTR:instance_name        |                                                  |
| OS-EXT-SRV-ATTR:kernel_id            |                                                  |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                                  |
| OS-EXT-SRV-ATTR:reservation_id       | r-psorddod                                       |
| OS-EXT-SRV-ATTR:root_device_name     | –                                                |
| OS-EXT-SRV-ATTR:user_data            | –                                                |
| OS-EXT-STS:power_state               | 0                                                |
| OS-EXT-STS:task_state                | scheduling                                       |
| OS-EXT-STS:vm_state                  | building                                         |
| OS-SRV-USG:launched_at               | –                                                |
| OS-SRV-USG:terminated_at             | –                                                |
| accessIPv4                           |                                                  |
| accessIPv6                           |                                                  |
| adminPass                            | dsFB8vrfUmv4                                     |
| config_drive                         |                                                  |
| created                              | 2016-09-15T12:01:34Z                             |
| description                          | –                                                |
| flavor                               | m2.small (2)                                     |
| hostId                               |                                                  |
| host_status                          |                                                  |
| id                                   | 212e06de-e971-428b-9e94-79dc8d91b6db             |
| image                                | VF24Cloud (81e031b0-11b7-440b-946f-b8f9e3a83c95) |
| key_name                             | oskey09152016                                    |
| locked                               | False                                            |
| metadata                             | {}                                               |
| name                                 | VF24Devs05                                       |
| os-extended-volumes:volumes_attached | []                                               |
| progress                             | 0                                                |
| security_groups                      | default                                          |
| status                               | BUILD                                            |
| tags                                 | []                                               |
| tenant_id                            | a1c9c1c1a1134384b4a496d585981aff                 |
| updated                              | 2016-09-15T12:01:34Z                             |
| user_id                              | e2383104829c45e1a3d70e11cc87d399                 |
+————————————–+————————————————–+

[root@overcloud-controller-0 ~]# nova list

+————————————–+————-+——–+————+————-+————————————-+
| ID                                   | Name        | Status | Task State | Power State | Networks                            |
+————————————–+————-+——–+————+————-+————————————-+
| c7cea368-9602-421d-beb3-c0ed37379b57 | CirrOSDevs1 | ACTIVE | –          | Running     | demo_network=50.0.0.17, 192.0.2.104 |
| 212e06de-e971-428b-9e94-79dc8d91b6db | VF24Devs05  | BUILD  | spawning   | NOSTATE     | demo_network=50.0.0.15              |
+————————————–+————-+——–+————+————-+————————————-+

[root@overcloud-controller-0 ~]# nova list

+————————————–+————-+——–+————+————-+————————————-+
| ID                                   | Name        | Status | Task State | Power State | Networks                            |
+————————————–+————-+——–+————+————-+————————————-+
| c7cea368-9602-421d-beb3-c0ed37379b57 | CirrOSDevs1 | ACTIVE | –          | Running     | demo_network=50.0.0.17, 192.0.2.104 |
| 212e06de-e971-428b-9e94-79dc8d91b6db | VF24Devs05  | ACTIVE | –          | Running     | demo_network=50.0.0.15              |
+————————————–+————-+——–+————+————-+————————————-+

Another option activate vlan10 following
http://bderzhavets.blogspot.com/2016/07/stable-mitaka-ha-instack-virt-setup.html
and instead of `./tripleo-ci/scripts/tripleo.sh –overcloud-deploy`
run following deployment with network isolation activated :-

#!/bin/bash -x
source /home/stack/stackrc
openstack overcloud deploy \
–control-scale 1 –compute-scale 1 \
–libvirt-type qemu \
–ntp-server pool.ntp.org \
–templates /home/stack/tripleo-heat-templates \
-e /home/stack/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml \
-e /home/stack/tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
-e $HOME/network_env.yaml

Presence of overcloud-resource-registry-puppet.yaml might explain avoiding

overcloud deployment failure no matter that overcloud-without-mergepy.yaml was not found at usual location


Backport upstream commits to stable RDO Mitaka release && Deployments with Keystone API V3

May 23, 2016

Posting bellow is written  with intend to avoid waiting until “koji” build will appear in updates repo of stable RDO Mitaka release, what might take a couple of months or so. Actually, it doesn’t require knowledge how to write properly source RH’s rpm file. It just needs picking up raw content of git commits from upstream git repo converting them into patches and rebuild required src.rpm(s) with patch(es) needed. There is also not commonly known command `rpm -qf` which is very useful when you need to detect which rpm has installed particular file. Just to know which src.rpm should be downloaded for git commit referencing say “cinder.rb”

[root@ServerCentOS72 /]# find . -name cinder.rb -print
find: ‘./run/user/1000/gvfs’: Permission denied
./usr/share/openstack-puppet/modules/cinder/lib/puppet/provider/cinder.rb

[root@ServerCentOS72 /]# rpm -qf /usr/share/openstack-puppet/modules/cinder/lib/puppet/provider/cinder.rb
openstack-puppet-modules-8.0.4-2.el7.centos.noarch

*******************************
Thus download from
*******************************

1. https://cbs.centos.org/koji/buildinfo?buildID=10895
openstack-packstack-8.0.0-1.el7.src.rpm

2. https://cbs.centos.org/koji/buildinfo?buildID=10859
openstack-puppet-modules-8.0.4-1.el7.src.rpm

[boris@ServerCentOS72 Downloads]$ ls -l
total 3116
-rw-rw-r–. 1 boris boris  170107 May 21 21:26 openstack-packstack-8.0.0-1.el7.src.rpm
-rw-rw-r–. 1 boris boris 3015046 May 21 18:33 openstack-puppet-modules-8.0.4-1.el7.src.rpm

****************
Then run :-
****************

$ rpm -iv openstack-packstack-8.0.0-1.el7.src.rpm
$ rpm -iv  openstack-puppet-modules-8.0.4-1.el7.src.rpm
$ cd ../rpmbuild

In folder ~boris/rpmbuild/SOURCES
create to patch files :-

0001-Use-versionless-auth_url-for-cinder.patch
0001-Enable-keystone-v3-support-for-cinder_type.patch

********************************************************************
In second patch file insert “cinder” in path to *.rb files
********************************************************************

diff –git a/cinder/lib/puppet/provider/cinder_type/openstack.rb b/cinder/lib/puppet/provider/cinder_type/openstack.rb
index feaea49..9aa31c5 100644
— a/cinder/lib/puppet/provider/cinder_type/openstack.rb
+++ b/cinder/lib/puppet/provider/cinder_type/openstack.rb
@@ -32,6 +32,10 @@ class Puppet::Provider::Cinder &lt; Puppet::Provider::Openstack

. . . . .

diff –git a/cinder/lib/puppet/provider/cinder_type/openstack.rb b/cinder/lib/puppet/provider/cinder_type/openstack.rb
index feaea49..9aa31c5 100644
— a/cinder/lib/puppet/provider/cinder_type/openstack.rb
+++ b/cinder/lib/puppet/provider/cinder_type/openstack.rb
@@ -7,7 +7,7 @@ Puppet::Type.type(:cinder_type).provide(

. . . . . .

diff –git a/cinder/spec/unit/provider/cinder_spec.rb b/cinder/spec/unit/provider/cinder_spec.rb
index cfc8850..246ae58 100644
— a/cinder/spec/unit/provider/cinder_spec.rb
+++ b/cinder/spec/unit/provider/cinder_spec.rb
@@ -24,10 +24,12 @@ describe Puppet::Provider::Cinder do

Finally SOURES folder would  look like :-

**********************
Next step is :-
**********************

$ cd ../SPECS

and update *.spec files , so that they would understand that patches placed
into SOURCES folder have to be applied to corresponding *.tar.gz archives
before building phase itself.

*****************************************
First openstack-packstack.spec :-
*****************************************

Name:           openstack-packstack
Version:        8.0.0
Release:        2%{?milestone}%{?dist}   <== increase 1 to 2
Summary:        Openstack Install Utility
Group:          Applications/System
License:        ASL 2.0 and GPLv2
URL:            https://github.com/openstack/packstack
Source0:        http://tarballs.openstack.org/packstack/packstack-%{upstream_version}.tar.gz
Patch0:         0001-Use-versionless-auth_url-for-cinder.patch  <=== Add line

. . . . . .

%prep
%setup -n packstack-%{upstream_version}
%patch0 -p1  <==  Add line
:wq

*****************************************
Second openstack-puppet-modules.spec
*****************************************

Name:           openstack-puppet-modules
Epoch:          1
Version:        8.0.4
Release:        2%{?milestone}%{?dist}  <===  increase 1 to 2
Summary:        Puppet modules used to deploy OpenStack
License:        ASL 2.0 and GPLv2 and GPLv3
URL:         https://github.com/redhat-openstack
Source0:    https://github.com/redhat-openstack/%{name}/archive/%{upstream_version}.tar.gz
Patch0:    0001-Enable-keystone-v3-support-for-cinder_type.patch  <== Add line

. . . . .

%prep
%setup -q -n %{name}-%{?upstream_version}
%patch0 -p1  <== Add line
:wq

******************************************
Attempt rpmbuild for each spec file
******************************************

$ rpmbuild -bb openstack-packstack.spec
$ rpmbuild -bb openstack-puppet-modules.spec

If particular build is missing some packages it will report their’s names to screen
This packages could be usually installed via yum, otherwise you have a problem
with local build.
If each build output finishes with message like

Wrote: /home/boris/rpmbuild/RPMS/noarch/openstack-puppet-modules-8.0.4-2.el7.centos.noarch.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.wX6p3q
+ umask 022
+ cd /home/boris/rpmbuild/BUILD
+ cd openstack-puppet-modules-8.0.4
+ /usr/bin/rm -rf /home/boris/rpmbuild/BUILDROOT/openstack-puppet-modules-8.0.4-2.el7.centos.x86_64
+ exit 0

Everything is going fine. In particular case results will be written

to ../RPMS/noarch

Then

$ cd ../RPMS/noarch

and create installation script

[boris@ServerCentOS72 SPECS]$ cd ../RPMS/noarch

[boris@ServerCentOS72 noarch]$ ls -l
total 3428
-rwxrwxr-x. 1 boris boris     239 May 21 21:40 install
-rw-rw-r–. 1 boris boris  247312 May 21 21:34 openstack-packstack-8.0.0-2.el7.centos.noarch.rpm
-rw-rw-r–. 1 boris boris   17376 May 21 21:34 openstack-packstack-doc-8.0.0-2.el7.centos.noarch.rpm
-rw-rw-r–. 1 boris boris   16792 May 21 21:34 openstack-packstack-puppet-8.0.0-2.el7.centos.noarch.rpm
-rw-rw-r–. 1 boris boris 3212844 May 21 21:38 openstack-puppet-modules-8.0.4-2.el7.centos.noarch.rpm

[boris@ServerCentOS72 noarch]$ cat install

sudo yum install openstack-packstack-8.0.0-2.el7.centos.noarch.rpm \
openstack-packstack-doc-8.0.0-2.el7.centos.noarch.rpm \
openstack-packstack-puppet-8.0.0-2.el7.centos.noarch.rpm \
openstack-puppet-modules-8.0.4-2.el7.centos.noarch.rpm

****************************
Run install :-
****************************

[boris@ServerCentOS72 noarch]$ ./install
Due to increased  release (1=>2) old rpms should be replaced by just been built

[root@ServerCentOS72 ~]# rpm -qa  \*openstack-packstack\*
openstack-packstack-doc-8.0.0-2.el7.centos.noarch
openstack-packstack-puppet-8.0.0-2.el7.centos.noarch
openstack-packstack-8.0.0-2.el7.centos.noarch

[root@ServerCentOS72 ~]# rpm -qa \*openstack-puppet-modules\*
openstack-puppet-modules-8.0.4-2.el7.centos.noarch

****************************************************************
Since that point following entry in your answer-file :-
****************************************************************
# Identity service API version string. [‘v2.0’, ‘v3’]
CONFIG_KEYSTONE_API_VERSION=v3
won’t cause cinder puppet to crash packstack run, no matter of kind of your deployment

References
1. https://bugzilla.redhat.com/show_bug.cgi?id=1330289


Storage Node (LVMiSCSI) deployment for RDO Kilo on CentOS 7.2

January 4, 2016

RDO deployment bellow has been done via straightforward RDO Kilo packstack run demonstrates that Storage Node might work as traditional iSCSI Target Server and each Compute Node is actually iSCSI initiator client. This functionality is provided by tuning Cinder && Glance Services running on Storage Node.
Following bellow is set up for 3 node deployment test Controller/Network & Compute & Storage on RDO Kilo (CentOS 7.2), which was performed on Fedora 23 host with KVM/Libvirt Hypervisor (32 GB RAM, Intel Core i7-4790 Haswell CPU, ASUS Z97-P ) .Three VMs (4 GB RAM, 4 VCPUS) have been setup. Controller/Network VM two (external/management subnet,vteps’s subnet) VNICs, Compute Node VM two VNICS (management,vtep’s subnets), Storage Node VM one VNIC (management)

Setup :-

192.169.142.127 – Controller/Network Node
192.169.142.137 – Compute Node
192.169.142.157 – Storage Node (LVMiSCSI)

Deployment could be done via answer-file from https://www.linux.com/community/blogs/133-general-linux/864102-storage-node-lvmiscsi-deployment-for-rdo-liberty-on-centos-71

Notice that Glance,Cinder, Swift Services are not running on Controller. Connection to http://StorageNode-IP:8776/v1/xxxxxx/types will be satisfied as soon as dependencies introduced by https://review.openstack.org/192883 will be satisfied on Storage Node, otherwise it could be done only via second run of RDO Kilo installer, having this port ready to respond on Controller (cinder-api port) previously been set up as first storage node. Thanks to Javier Pena, who did the this troubleshooting in https://bugzilla.redhat.com/show_bug.cgi?id=1234038. Issue has been fixed in RDO Liberty release.

 

SantiagoController1

Storage Node

SantiagoStorage1

SantiagoStorage2

SantiagoStorage3

Compute Node

SantiagoCompute1

[root@ip-192-169-142-137 ~(keystone_admin)]# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 6.2.0.873-30
Target: iqn.2010-10.org.openstack:volume-3ab60233-5110-4915-9998-7cec7d3ac919 (non-flash)
Current Portal: 192.169.142.157:3260,1
Persistent Portal: 192.169.142.157:3260,1
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat:576fc73f30e9
Iface IPaddress: 192.169.142.137
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 120
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: hBbbvVmompAY6ikd8DJF
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 262144
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 2 State: running
scsi2 Channel 00 Id 0 Lun: 0
Attached scsi disk sda State: running
Target: iqn.2010-10.org.openstack:volume-2087aa9a-7984-4f4e-b00d-e461fcd02099 (non-flash)
Current Portal: 192.169.142.157:3260,1
Persistent Portal: 192.169.142.157:3260,1
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat:576fc73f30e9
Iface IPaddress: 192.169.142.137
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 2
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 120
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: TB8qiKbMdrWwoLBPdCTs
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 262144
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 3 State: running
scsi3 Channel 00 Id 0 Lun: 0
Attached scsi disk sdb State: running


Iftop -i eth0 running on Controller vs iftop -i eth0 running on Compute

September 29, 2015

Controller 192.169.142.127
Compute nodes 192.169.142.147,192.169.142.137

Screenshot from 2015-09-29 19-32-55

Screenshot from 2015-09-29 19-33-40

VM vf22devs02 is running on second compute node 192.169.142.137

Screenshot from 2015-09-29 21-44-45


CPU Pinning and NUMA Topology on RDO Kilo on Fedora Server 22

August 1, 2015
Posting bellow follows up http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/
on RDO Kilo installed on Fedora 22 . After upgrade to upstream version of openstack-puppet-modules-2015.1.9 procedure of RDO Kilo install on F22 significantly changed. Details follow bellow :-
*****************************************************************************************
RDO Kilo set up on Fedora ( openstack-puppet-modules-2015.1.9-4.fc23.noarch)
*****************************************************************************************
# dnf install -y https://rdoproject.org/repos/rdo-release.rpm
# dnf install -y openstack-packstack
Generate answer-file and make update :-
# packstack  –gen-answer-file answer-file-aio.txt
and set CONFIG_KEYSTONE_SERVICE_NAME=httpd
****************************************************************************
I also commented out second line in  /etc/httpd/conf.d/mod_dnssd.conf
****************************************************************************
You might be hit by bug  https://bugzilla.redhat.com/show_bug.cgi?id=1249482
As pre-install step apply patch https://review.openstack.org/#/c/209032/
to fix neutron_api.pp. Location of puppet templates
/usr/lib/python2.7/site-packages/packstack/puppet/templates.
You might be also hit by  https://bugzilla.redhat.com/show_bug.cgi?id=1234042
Workaround is in comments 6,11
****************
Then run :-
****************

# packstack  –answer-file=./answer-file-aio.txt

Final target is to reproduce mentioned article on i7 4790 Haswell CPU box, perform launching nova instance with CPU pinning.

[root@fedora22server ~(keystone_admin)]# uname -a
Linux fedora22server.localdomain 4.1.3-200.fc22.x86_64 #1 SMP Wed Jul 22 19:51:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@fedora22server ~(keystone_admin)]# rpm -qa \*qemu\*
qemu-system-x86-2.3.0-6.fc22.x86_64
qemu-img-2.3.0-6.fc22.x86_64
qemu-guest-agent-2.3.0-6.fc22.x86_64
qemu-kvm-2.3.0-6.fc22.x86_64
ipxe-roms-qemu-20150407-1.gitdc795b9f.fc22.noarch
qemu-common-2.3.0-6.fc22.x86_64
libvirt-daemon-driver-qemu-1.2.13.1-2.fc22.x86_64

[root@fedora22server ~(keystone_admin)]# numactl –hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 15991 MB
node 0 free: 4399 MB
node distances:
node 0
0: 10

[root@fedora22server ~(keystone_admin)]# virsh capabilities

<capabilities>
<host>
<uuid>00fd5d2c-dad7-dd11-ad7e-7824af431b53</uuid>
<cpu>
<arch>x86_64</arch>
<model>Haswell-noTSX</model>
<vendor>Intel</vendor>
<topology sockets=’1′ cores=’4′ threads=’2’/>
<feature name=’invtsc’/>
<feature name=’abm’/>
<feature name=’pdpe1gb’/>
<feature name=’rdrand’/>
<feature name=’f16c’/>
<feature name=’osxsave’/>
<feature name=’pdcm’/>
<feature name=’xtpr’/>
<feature name=’tm2’/>
<feature name=’est’/>
<feature name=’smx’/>
<feature name=’vmx’/>
<feature name=’ds_cpl’/>
<feature name=’monitor’/>
<feature name=’dtes64’/>
<feature name=’pbe’/>
<feature name=’tm’/>
<feature name=’ht’/>
<feature name=’ss’/>
<feature name=’acpi’/>
<feature name=’ds’/>
<feature name=’vme’/>
<pages unit=’KiB’ size=’4’/>
<pages unit=’KiB’ size=’2048’/>
</cpu>
<power_management>
<suspend_mem/>
<suspend_disk/>
<suspend_hybrid/>
</power_management>
<migration_features>
<live/>
<uri_transports>
<uri_transport>tcp</uri_transport>
<uri_transport>rdma</uri_transport>
</uri_transports>
</migration_features>
<topology>
<cells num=’1′>
<cell id=’0′>
<memory unit=’KiB’>16374824</memory>
<pages unit=’KiB’ size=’4′>4093706</pages>
<pages unit=’KiB’ size=’2048′>0</pages>
<distances>
<sibling id=’0′ value=’10’/>
</distances>
<cpus num=’8′>
<cpu id=’0′ socket_id=’0′ core_id=’0′ siblings=’0,4’/>
<cpu id=’1′ socket_id=’0′ core_id=’1′ siblings=’1,5’/>
<cpu id=’2′ socket_id=’0′ core_id=’2′ siblings=’2,6’/>
<cpu id=’3′ socket_id=’0′ core_id=’3′ siblings=’3,7’/>
<cpu id=’4′ socket_id=’0′ core_id=’0′ siblings=’0,4’/>
<cpu id=’5′ socket_id=’0′ core_id=’1′ siblings=’1,5’/>
<cpu id=’6′ socket_id=’0′ core_id=’2′ siblings=’2,6’/>
<cpu id=’7′ socket_id=’0′ core_id=’3′ siblings=’3,7’/>
</cpus>
</cell>
</cells>
</topology>

On each Compute node that pinning of virtual machines will be permitted on open the /etc/nova/nova.conf file and make the following modifications:

Set the vcpu_pin_set value to a list or range of logical CPU cores to reserve for virtual machine processes. OpenStack Compute will ensure guest virtual machine instances are pinned to these virtual CPU cores.
vcpu_pin_set=2,3,6,7

Set the reserved_host_memory_mb to reserve RAM for host processes. For the purposes of testing used the default of 512 MB:
reserved_host_memory_mb=512

# systemctl restart openstack-nova-compute.service

************************************
SCHEDULER CONFIGURATION
************************************

Update /etc/nova/nova.conf

scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,
ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,
NUMATopologyFilter,AggregateInstanceExtraSpecsFilter

# systemctl restart openstack-nova-scheduler.service

At this point if creating a guest you may see some changes to appear in the XML, pinning the guest vCPU(s) to the cores listed in vcpu_pin_set:

<vcpu placement=’static’ cpuset=’2-3,6-7′>1</vcpu>

Add to vmlinuz grub2 command line at the end
isolcpus=2,3,6,7

***************
REBOOT
***************
[root@fedora22server ~(keystone_admin)]# nova aggregate-create performance

+—-+————-+——————-+——-+———-+

| Id | Name | Availability Zone | Hosts | Metadata |

+—-+————-+——————-+——-+———-+

| 1 | performance | – | | |

+—-+————-+——————-+——-+———-+

[root@fedora22server ~(keystone_admin)]# nova aggregate-set-metadata 1 pinned=true
Metadata has been successfully updated for aggregate 1.
+—-+————-+——————-+——-+—————+

| Id | Name | Availability Zone | Hosts | Metadata |

+—-+————-+——————-+——-+—————+

| 1 | performance | – | | ‘pinned=true’ |

+—-+————-+——————-+——-+—————+

[root@fedora22server ~(keystone_admin)]# nova flavor-create m1.small.performance 6 4096 20 4
+—-+———————-+———–+——+———–+——+——-+————-+———–+

| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |

+—-+———————-+———–+——+———–+——+——-+————-+———–+

| 6 | m1.small.performance | 4096 | 20 | 0 | | 4 | 1.0 | True |

+—-+———————-+———–+——+———–+——+——-+————-+———–+

[root@fedora22server ~(keystone_admin)]# nova flavor-key 6 set hw:cpu_policy=dedicated
[root@fedora22server ~(keystone_admin)]# nova flavor-key 6 set aggregate_instance_extra_specs:pinned=true
[root@fedora22server ~(keystone_admin)]# hostname
fedora22server.localdomain

[root@fedora22server ~(keystone_admin)]# nova aggregate-add-host 1 fedora22server.localdomain
Host fedora22server.localdomain has been successfully added for aggregate 1
+—-+————-+——————-+——————————+—————+

| Id | Name | Availability Zone | Hosts | Metadata |

+—-+————-+——————-+——————————+—————+
| 1 | performance | – | ‘fedora22server.localdomain’ | ‘pinned=true’ |
+—-+————-+——————-+——————————+—————+

[root@fedora22server ~(keystone_admin)]# . keystonerc_demo
[root@fedora22server ~(keystone_demo)]# glance image-list
+————————————–+———————————+————-+——————+————-+——–+
| ID | Name | Disk Format | Container Format | Size | Status |
+————————————–+———————————+————-+——————+————-+——–+
| bf6f5272-ae26-49ae-b0f9-3c4fcba350f6 | CentOS71Image | qcow2 | bare | 1004994560 | active |
| 05ac955e-3503-4bcf-8413-6a1b3c98aefa | cirros | qcow2 | bare | 13200896 | active |
| 7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52 | VF22Image | qcow2 | bare | 228599296 | active |
| c695e7fa-a69f-4220-abd8-2269b75af827 | Windows Server 2012 R2 Std Eval | qcow2 | bare | 17182752768 | active |
+————————————–+———————————+————-+——————+————-+——–+

[root@fedora22server ~(keystone_demo)]#neutron net-list

+————————————–+———-+—————————————————–+
| id | name | subnets |
+————————————–+———-+—————————————————–+
| 0daa3a02-c598-4c46-b1ac-368da5542927 | public | 8303b2f3-2de2-44c2-bd5e-fc0966daec53 192.168.1.0/24 |
| c85a4215-1558-4a95-886d-a2f75500e052 | demo_net | 0cab6cbc-dd80-42c6-8512-74d7b2cbf730 50.0.0.0/24 |
+————————————–+———-+—————————————————–+

*************************************************************************
At this point attempt to launch F22 Cloud instance with created flavor
m1.small.performance
*************************************************************************

[root@fedora22server ~(keystone_demo)]# nova boot –image 7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52 –key-name oskeydev –flavor m1.small.performance –nic net-id=c85a4215-1558-4a95-886d-a2f75500e052 vf22-instance

+————————————–+————————————————–+
| Property | Value |
+————————————–+————————————————–+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | – |
| OS-SRV-USG:terminated_at | – |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | XsGr87ZLGX8P |
| config_drive | |
| created | 2015-07-31T08:03:49Z |
| flavor | m1.small.performance (6) |
| hostId | |
| id | 4b99f3cf-3126-48f3-9e00-94787f040e43 |
| image | VF22Image (7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52) |
| key_name | oskeydev |
| metadata | {} |
| name | vf22-instance |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 14f736e6952644b584b2006353ca51be |
| updated | 2015-07-31T08:03:50Z |
| user_id | 4ece2385b17a4490b6fc5a01ff53350c |
+————————————–+————————————————–+

[root@fedora22server ~(keystone_demo)]#nova list

+————————————–+—————+———+————+————-+———————————–+
| ID | Name | Status | Task State | Power State | Networks |
+————————————–+—————+———+————+————-+———————————–+
| 93906a61-ec0b-481d-b964-2bb99d095646 | CentOS71RLX | SHUTOFF | – | Shutdown | demo_net=50.0.0.21, 192.168.1.159 |
| ac7e9be5-d2dc-4ec0-b0a1-4096b552e578 | VF22Devpin | ACTIVE | – | Running | demo_net=50.0.0.22 |
| b93c9526-ded5-4b7a-ae3a-106b34317744 | VF22Devs | SHUTOFF | – | Shutdown | demo_net=50.0.0.19, 192.168.1.157 |
| bef20a1e-3faa-4726-a301-73ca49666fa6 | WinSrv2012 | SHUTOFF | – | Shutdown | demo_net=50.0.0.16 |
| 4b99f3cf-3126-48f3-9e00-94787f040e43 | vf22-instance | ACTIVE | – | Running | demo_net=50.0.0.23, 192.168.1.160 |
+————————————–+—————+———+————+————-+———————————–+

[root@fedora22server ~(keystone_demo)]#virsh list

Id Name State

—————————————————-
2 instance-0000000c running
3 instance-0000000d running

Please, see http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/
regarding detailed explanation of highlighted blocks, keeping in mind that pinning is done to logical CPU cores ( not physical due to 4 Core CPU with HT enabled ). Multiple cells are also absent, due limitations of i7 47XX Haswell CPU architecture

[root@fedora22server ~(keystone_demo)]#virsh dumpxml instance-0000000d > vf22-instance.xml
<domain type=’kvm’ id=’3′>
<name>instance-0000000d</name>
<uuid>4b99f3cf-3126-48f3-9e00-94787f040e43</uuid>
<metadata>
<nova:instance xmlns:nova=”http://openstack.org/xmlns/libvirt/nova/1.0″&gt;
<nova:package version=”2015.1.0-3.fc23″/>
<nova:name>vf22-instance</nova:name>
<nova:creationTime>2015-07-31 08:03:54</nova:creationTime>
<nova:flavor name=”m1.small.performance”>
<nova:memory>4096</nova:memory>
<nova:disk>20</nova:disk>
<nova:swap>0</nova:swap>
<nova:ephemeral>0</nova:ephemeral>
<nova:vcpus>4</nova:vcpus>
</nova:flavor>
<nova:owner>
<nova:user uuid=”4ece2385b17a4490b6fc5a01ff53350c”>demo</nova:user>
<nova:project uuid=”14f736e6952644b584b2006353ca51be”>demo</nova:project>
</nova:owner>
<nova:root type=”image” uuid=”7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52″/>
</nova:instance>
</metadata>
<memory unit=’KiB’>4194304</memory>
<currentMemory unit=’KiB’>4194304</currentMemory>
<vcpu placement=’static’>4</vcpu>
<cputune>
<shares>4096</shares>
<vcpupin vcpu=’0′ cpuset=’2’/>
<vcpupin vcpu=’1′ cpuset=’6’/>
<vcpupin vcpu=’2′ cpuset=’3’/>
<vcpupin vcpu=’3′ cpuset=’7’/>
<emulatorpin cpuset=’2-3,6-7’/>
</cputune>
<numatune>
<memory mode=’strict’ nodeset=’0’/>
<memnode cellid=’0′ mode=’strict’ nodeset=’0’/>
</numatune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type=’smbios’>
<system>
<entry name=’manufacturer’>Fedora Project</entry>
<entry name=’product’>OpenStack Nova</entry>
<entry name=’version’>2015.1.0-3.fc23</entry>
<entry name=’serial’>f1b336b1-6abf-4180-865a-b6be5670352e</entry>
<entry name=’uuid’>4b99f3cf-3126-48f3-9e00-94787f040e43</entry>
</system>
</sysinfo>
<os>
<type arch=’x86_64′ machine=’pc-i440fx-2.3′>hvm</type>
<boot dev=’hd’/>
<smbios mode=’sysinfo’/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode=’host-model’>
<model fallback=’allow’/>
<topology sockets=’2′ cores=’1′ threads=’2’/>
<numa>
<cell id=’0′ cpus=’0-3′ memory=’4194304′ unit=’KiB’/>
</numa>
</cpu>
<clock offset=’utc’>
<timer name=’pit’ tickpolicy=’delay’/>
<timer name=’rtc’ tickpolicy=’catchup’/>
<timer name=’hpet’ present=’no’/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<disk type=’file’ device=’disk’>
<driver name=’qemu’ type=’qcow2′ cache=’none’/>
<source file=’/var/lib/nova/instances/4b99f3cf-3126-48f3-9e00-94787f040e43/disk’/>
<backingStore type=’file’ index=’1′>
<format type=’raw’/>
<source file=’/var/lib/nova/instances/_base/6c60a5ed1b3037bbdb2bed198dac944f4c0d09cb’/>
<backingStore/>
</backingStore>
<target dev=’vda’ bus=’virtio’/>
<alias name=’virtio-disk0’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x06′ function=’0x0’/>
</disk>
<controller type=’usb’ index=’0′>
<alias name=’usb0’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x01′ function=’0x2’/>
</controller>
<controller type=’pci’ index=’0′ model=’pci-root’>
<alias name=’pci.0’/>
</controller>
<controller type=’virtio-serial’ index=’0′>
<alias name=’virtio-serial0’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x05′ function=’0x0’/>
</controller>
<interface type=’bridge’>
<mac address=’fa:16:3e:4f:25:03’/>
<source bridge=’qbr567b21fe-52’/>
<target dev=’tap567b21fe-52’/>
<model type=’virtio’/>
<alias name=’net0’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x03′ function=’0x0’/>
</interface>
<serial type=’file’>
<source path=’/var/lib/nova/instances/4b99f3cf-3126-48f3-9e00-94787f040e43/console.log’/>
<target port=’0’/>
<alias name=’serial0’/>
</serial>
<serial type=’pty’>
<source path=’/dev/pts/2’/>
<target port=’1’/>
<alias name=’serial1’/>
</serial>
<console type=’file’>
<source path=’/var/lib/nova/instances/4b99f3cf-3126-48f3-9e00-94787f040e43/console.log’/>
<target type=’serial’ port=’0’/>
<alias name=’serial0’/>
</console>
<channel type=’spicevmc’>
<target type=’virtio’ name=’com.redhat.spice.0′ state=’disconnected’/>
<alias name=’channel0’/>
<address type=’virtio-serial’ controller=’0′ bus=’0′ port=’1’/>
</channel>
<input type=’mouse’ bus=’ps2’/>
<input type=’keyboard’ bus=’ps2’/>
<graphics type=’spice’ port=’5901′ autoport=’yes’ listen=’0.0.0.0′ keymap=’en-us’>
<listen type=’address’ address=’0.0.0.0’/>
</graphics>
<sound model=’ich6′>
<alias name=’sound0’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x04′ function=’0x0’/>
</sound>
<video>
<model type=’qxl’ ram=’65536′ vram=’65536′ vgamem=’16384′ heads=’1’/>
<alias name=’video0’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x02′ function=’0x0’/>
</video>
<memballoon model=’virtio’>
<alias name=’balloon0’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x07′ function=’0x0’/>
<stats period=’10’/>
</memballoon>
</devices>
<seclabel type=’dynamic’ model=’selinux’ relabel=’yes’>
<label>system_u:system_r:svirt_t:s0:c359,c706</label>
<imagelabel>system_u:object_r:svirt_image_t:s0:c359,c706</imagelabel>
</seclabel>
</domain>

Screenshot from 2015-07-31 21-55-33                                              Screenshot from 2015-07-31 15-05-53