Glusterfs Striped volumes based Havana 2013.2 instances on NFS-Like Standalone Storage Server With GlusterFS 3.4.1 Fedora 19

Please,  view first nice article: http://www.mirantis.com/blog/openstack-havana-glusterfs and  https://wiki.openstack.org/wiki/CinderSupportMatrix. Here goes a sample of more complicated structure of  gluster volume for Havana’s instances bootable cinder volumes storage  versus  http://bderzhavets.wordpress.com/2013/10/18/glusterfss-volume-based-havana-rc1-instances-on-nfs-like-standalone-storage-server-with-glusterfs-3-4-1-fedora-19/

[root@localhost boris(keystone_admin)]# gluster volume create \

cinder-volumes stripe 3 \

192.168.1.142:/home/boris/node1 \

192.168.1.142:/home/boris/node2 \

192.168.1.142:/home/boris/node3

volume create: cinder-volumes: success: please start the volume to access data

[root@localhost boris(keystone_admin)]# gluster volume start cinder-volumes

volume start: cinder-volumes: success

[root@localhost boris(keystone_admin)]# gluster volume info

Volume Name: cinder-volume

Type: Distribute

Volume ID: be95ce0d-47ea-47f5-beee-66196c546d20

Status: Started

Number of Bricks: 1

Transport-type: tcp

Bricks:

Brick1: 192.168.1.142:/rhs/brick1/cinder-volume

Volume Name: cinder-volumes

Type: Stripe

Volume ID: 14b7de86-7b0e-4a60-b21b-4990b1222f43

Status: Started

Number of Bricks: 1 x 3 = 3

Transport-type: tcp

Bricks:

Brick1: 192.168.1.142:/home/boris/node1

Brick2: 192.168.1.142:/home/boris/node2

Brick3: 192.168.1.142:/home/boris/node3

[root@localhost boris(keystone_admin)]# systemctl status glusterd

glusterd.service – GlusterFS an clustered file-system server

Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled)

Active: active (running) since Sun 2013-10-20 13:47:11 MSK; 2h 21min ago

Process: 1350 ExecStart=/usr/sbin/glusterd -p /run/glusterd.pid (code=exited, status=0/SUCCESS)

Main PID: 1380 (glusterd)

CGroup: name=systemd:/system/glusterd.service

├─ 1380 /usr/sbin/glusterd -p /run/glusterd.pid

├─ 2217 /usr/sbin/glusterfsd -s 192.168.1.142 –volfile-id cinder-volume.192.168.1.142.rhs-brick1-cinder-volume …

├─11156 /usr/sbin/glusterfsd -s 192.168.1.142 –volfile-id cinder-volumes.192.168.1.142.home-boris-node1 -p /var…

├─11165 /usr/sbin/glusterfsd -s 192.168.1.142 –volfile-id cinder-volumes.192.168.1.142.home-boris-node2 -p /var…

├─11174 /usr/sbin/glusterfsd -s 192.168.1.142 –volfile-id cinder-volumes.192.168.1.142.home-boris-node3 -p /var…

├─11190 /usr/sbin/glusterfs -s localhost –volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/l…

└─11198 /sbin/rpc.statd

Oct 20 13:47:11 localhost.localdomain rpc.statd[2261]: Version 1.2.7 starting

Oct 20 13:47:11 localhost.localdomain systemd[1]: Started GlusterFS an clustered file-system server.

Oct 20 13:58:50 localhost.localdomain rpc.statd[11198]: Version 1.2.7 starting

Oct 20 13:58:50 localhost.localdomain sm-notify[11199]: Version 1.2.7 starting

 Configuring Cinder to Add GlusterFS

  If it has not been done

# openstack-config –set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver

# openstack-config –set /etc/cinder/cinder.conf DEFAULT glusterfs_shares_config /etc/cinder/shares.conf

# openstack-config –set /etc/cinder/cinder.conf DEFAULT glusterfs_mount_point_base /var/lib/cinder/volumes

  Update shares to point to another gluster volume 

   # vi /etc/cinder/shares.conf

    192.168.1.142:cinder-volumes

:wq

  If it has not been done

# iptables-save > iptables.dump

Add to *filter section:

-A INPUT -m state –state NEW -m tcp -p tcp –dport 111 -j ACCEPT

-A INPUT -m state –state NEW -m tcp -p tcp –dport 24007 -j ACCEPT

-A INPUT -m state –state NEW -m tcp -p tcp –dport 24008 -j ACCEP

-A INPUT -m state –state NEW -m tcp -p tcp –dport 24009 -j ACCEPT

-A INPUT -m state –state NEW -m tcp -p tcp –dport 24010 -j ACCEPT

-A INPUT -m state –state NEW -m tcp -p tcp –dport 24011 -j ACCEPT

-A INPUT -m state –state NEW -m tcp -p tcp –dport 38465:38469 -j ACCEPT

# iptables-restore <  iptables.dump

# service iptables restart

[root@localhost boris(keystone_admin)]# df -h

Filesystem                   Size  Used Avail Use% Mounted on

/dev/mapper/fedora00-root    145G   60G   78G  44% /

devtmpfs                     3.9G     0  3.9G   0% /dev

tmpfs                        3.9G  648K  3.9G   1% /dev/shm

tmpfs                        3.9G 1020K  3.9G   1% /run

tmpfs                        3.9G     0  3.9G   0% /sys/fs/cgroup

tmpfs                        3.9G   96K  3.9G   1% /tmp

/dev/sdb3                    477M  104M  344M  24% /boot

/dev/loop0                   928M  1.4M  860M   1% /srv/node/device1

192.168.1.142:cinder-volume  145G   60G   78G  44% /var/lib/cinder/volumes/4b0a6960f94e0c2a28a479c06957e35a

tmpfs                        3.9G 1020K  3.9G   1% /run/netns

Ready to restart cinder services  

  [root@localhost boris(keystone_admin)]# for i in api scheduler volume; do service openstack-cinder-${i} restart; done

Redirecting to /bin/systemctl restart  openstack-cinder-api.service

Redirecting to /bin/systemctl restart  openstack-cinder-scheduler.service

Redirecting to /bin/systemctl restart  openstack-cinder-volume.service

New directory mounted :

[root@localhost boris(keystone_admin)]# df -h

Filesystem                    Size  Used Avail Use% Mounted on

/dev/mapper/fedora00-root     145G   59G   78G  44% /

devtmpfs                      3.9G     0  3.9G   0% /dev

tmpfs                         3.9G  648K  3.9G   1% /dev/shm

tmpfs                         3.9G 1020K  3.9G   1% /run

tmpfs                         3.9G     0  3.9G   0% /sys/fs/cgroup

tmpfs                         3.9G  108K  3.9G   1% /tmp

/dev/sdb3                     477M  104M  344M  24% /boot

/dev/loop0                    928M  1.4M  860M   1% /srv/node/device1

tmpfs                         3.9G 1020K  3.9G   1% /run/netns

192.168.1.142:cinder-volumes  433G  177G  234G  44% /var/lib/cinder/volumes/76078cf31245409735587a509f24ab39

[root@localhost boris(keystone_admin)]# nova image-list

+————————————–+—————————-+——–+——–+

| ID                                   | Name                       | Status | Server |

+————————————–+—————————-+——–+——–+

| 807758c4-20fd-41b2-a943-27c80a651fc7 | Fedora19image              | ACTIVE |        |

| 35098ebe-f04d-4c26-bd8c-0bec0f0fb892 | UbuntuSaucyRelease         | ACTIVE |        |

| e8e426f8-8898-42a5-9b3f-0374d20e318b | UbuntuServer1310.image     | ACTIVE |        |

| 8fdc5c6e-1554-4ad2-bcaf-f4aefee7d690 | Windos Server2012 Std Eval | ACTIVE |        |

| 10295ec8-3d8d-4623-9490-f217f8498878 | cirros                     | ACTIVE |        |

+————————————–+—————————-+——–+——–+

[root@localhost boris(keystone_admin)]# cinder create –image-id 807758c4-20fd-41b2-a943-27c80a651fc7  –display_name Fedora19VL 5

+———————+————————————–+

|       Property      |                Value                 |

+———————+————————————–+

|     attachments     |                  []                  |

|  availability_zone  |                 nova                 |

|       bootable      |                false                 |

|      created_at     |      2013-10-20T10:05:22.586342      |

| display_description |                 None                 |

|     display_name    |              Fedora19VL              |

|          id         | d11f00de-e543-4ed5-9686-a0421639c5e3 |

|       image_id      | 807758c4-20fd-41b2-a943-27c80a651fc7 |

|       metadata      |                  {}                  |

|         size        |                  5                   |

|     snapshot_id     |                 None                 |

|     source_volid    |                 None                 |

|        status       |               creating               |

|     volume_type     |                 None                 |

+———————+————————————–+

[root@localhost boris(keystone_admin)]# cinder list

+————————————–+———–+————–+——+————-+———-+————-+

|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |

+————————————–+———–+————–+——+————-+———-+————-+

| d11f00de-e543-4ed5-9686-a0421639c5e3 | available |  Fedora19VL  |  5   |     None    |   true   |             |

+————————————–+———–+————–+——+————-+———-+————-+

[root@localhost boris(keystone_admin)]# cinder create –image-id 35098ebe-f04d-4c26-bd8c-0bec0f0fb892 –display_name SalamanderVL 5

+———————+————————————–+

|       Property      |                Value                 |

+———————+————————————–+

|     attachments     |                  []                  |

|  availability_zone  |                 nova             |

|       bootable         |                false              |

|      created_at       |      2013-10-20T10:07:01.897362      |

| display_description |                 None        |

|     display_name  |             SalamanderVL  |

|          id                | a0102e10-52cb-49d6-bd58-f0365968f721 |

|       image_id       | 35098ebe-f04d-4c26-bd8c-0bec0f0fb892 |

|       metadata      |                  {}               |

|         size             |                  5                 |

|     snapshot_id   |                 None           |

|     source_volid   |                 None          |

|        status          |               creating       |

|     volume_type   |                 None          |

+———————+————————————–+

[root@localhost boris(keystone_admin)]# cinder list

+————————————–+————-+————–+——+————-+———-+————-+

|                  ID                  |    Status   | Display Name | Size | Volume Type | Bootable | Attached to |

+————————————–+————-+————–+——+————-+———-+————-+

| a0102e10-52cb-49d6-bd58-f0365968f721 | downloading | SalamanderVL |  5   |     None    |   true   |             |

| d11f00de-e543-4ed5-9686-a0421639c5e3 |  available  |  Fedora19VL  |  5   |     None    |   true   |             |

+————————————–+————-+————–+——+————-+———-+————-+

[root@localhost boris(keystone_admin)]# cinder list

+————————————–+———–+————–+——+————-+———-+————-+

|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |

+————————————–+———–+————–+——+————-+———-+————-+

| a0102e10-52cb-49d6-bd58-f0365968f721 | available | SalamanderVL |  5   |     None    |   true   |             |

| d11f00de-e543-4ed5-9686-a0421639c5e3 | available |  Fedora19VL  |  5   |     None    |   true   |             |

+————————————–+———–+————–+——+————-+———-+————-+

[root@localhost boris(keystone_admin)]# ls -l /var/lib/cinder/volumes/76078cf31245409735587a509f24ab39

total 1496292

-rw-rw-rw-. 1 root root 5368709120 Oct 20 14:07 volume-a0102e10-52cb-49d6-bd58-f0365968f721

-rw-rw-rw-. 1 root root 5368709120 Oct 20 14:05 volume-d11f00de-e543-4ed5-9686-a0421639c5e3

[root@localhost boris(keystone_admin)]# nova list

+————————————–+——————-+——–+————+————-+——————————–+

| ID                                   | Name              | Status | Task State | Power State | Networks                       |

+————————————–+——————-+——–+————+————-+——————————–+

| 9754374c-bcf2-4f34-a931-649b417d939d | UbuntuSalamander1 | ACTIVE | None       | Running     | private=10.0.0.3, 192.168.1.61 |

+————————————–+——————-+——–+————+————-+——————————–+

[root@localhost boris(keystone_admin)]# nova show  9754374c-bcf2-4f34-a931-649b417d939d

+————————————–+———————————————————-+

| Property                             | Value                                                    |

+————————————–+———————————————————-+

| status                               | ACTIVE                                                   |

| updated                              | 2013-10-20T10:16:44Z                                     |

| OS-EXT-STS:task_state                | None                                                     |

| OS-EXT-SRV-ATTR:host                 | localhost.localdomain                                    |

| key_name                             | key2                                                     |

| image                                | Attempt to boot from volume – no image supplied          |

| private network                      | 10.0.0.3, 192.168.1.61                                   |

| hostId                               | 8a06ef818edcb74efba027817c5f14cb4d1d38c0fcd1dde73a9356d6 |

| OS-EXT-STS:vm_state                  | active                                                   |

| OS-EXT-SRV-ATTR:instance_name        | instance-00000013                                        |

| OS-SRV-USG:launched_at               | 2013-10-20T10:16:44.000000                               |

| OS-EXT-SRV-ATTR:hypervisor_hostname  | localhost.localdomain                                    |

| flavor                               | m1.small (2)                                             |

| id                                   | 9754374c-bcf2-4f34-a931-649b417d939d                     |

| security_groups                      | [{u'name': u'default'}]                                  |

| OS-SRV-USG:terminated_at             | None                                                     |

| user_id                              | 239b0f75c0514945acb5bb2041ccd89d                         |

| name                                 | UbuntuSalamander1                                        |

| created                              | 2013-10-20T10:16:35Z                                     |

| tenant_id                            | 644515d6bdf14f53a44f226fd7c9f69b                         |

| OS-DCF:diskConfig                    | MANUAL                                                   |

| metadata                             | {}                                                       |

| os-extended-volumes:volumes_attached | [{u'id': u'a0102e10-52cb-49d6-bd58-f0365968f721'}]       |

| accessIPv4                           |                                                          |

| accessIPv6                           |                                                          |

| progress                             | 0                                                        |

| OS-EXT-STS:power_state               | 1                                                        |

| OS-EXT-AZ:availability_zone          | nova                                                     |

| config_drive                         |                                                          |

+————————————–+———————————————————-+

[root@localhost boris(keystone_admin)]# cinder list

+————————————–+———–+————–+——+————-+———-+————————————–+

|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable |             Attached to              |

+————————————–+———–+————–+——+————-+———-+————————————–+

| a0102e10-52cb-49d6-bd58-f0365968f721 |   in-use  | SalamanderVL |  5   |     None    |   true   | 9754374c-bcf2-4f34-a931-649b417d939d |

| d11f00de-e543-4ed5-9686-a0421639c5e3 | available |  Fedora19VL  |  5   |     None    |   true   |                                      |

+————————————–+———–+————–+——+————-+———-+————————————–+

[root@localhost boris(keystone_admin)]# ls -lah /var/lib/cinder/volumes/76078cf31245409735587a509f24ab39

total 1.8G

drwxr-xr-x. 3 root   root   4.0K Oct 20 14:07 .

drwxr-xr-x. 4 cinder cinder 4.0K Oct 20 14:03 ..

-rw-rw-rw-. 1 qemu   qemu   5.0G Oct 20 14:28 volume-a0102e10-52cb-49d6-bd58-f0365968f721

-rw-rw-rw-. 1 root   root   5.0G Oct 20 14:05 volume-d11f00de-e543-4ed5-9686-a0421639c5e3

[root@localhost boris(keystone_admin)]# cd /home/boris

[root@localhost boris(keystone_admin)]# du | grep node.$

637972    ./node3

638368    ./node2

637964    ./node1

    Another snap shots series

Loading Windows 2012 Server via bootable cinder volume located on glusterfs stripe 3 volume.

[root@localhost (keystone_admin)]# nova image-list

+————————————–+——————————+——–+——–+

| ID                                   | Name                         | Status | Server |

+————————————–+——————————+——–+——–+

| 59758edc-da8d-444e-b0a0-d93d323fc026 | F19Image                     | ACTIVE |        |

| df912358-b227-43a5-94a3-edc874c577bc | UbuntuSalamander             | ACTIVE |        |

| 1e26928b-5df0-4097-bbc6-46832dc8361b | Windows Server 2012 Std Eval | ACTIVE |        |

| ae07d1ba-41de-44e9-877a-455f8956d86f | cirros                       | ACTIVE |        |

+————————————–+——————————+——–+——–+

[root@localhost (keystone_admin)]# cinder create –image-id 1e26928b-5df0-4097-bbc6-46832dc8361b  –display_name WinSRV2012 20

+———————+————————————–+

|       Property      |                Value                 |

+———————+————————————–+

|     attachments     |                  []                  |

|  availability_zone  |                 nova                 |

|       bootable      |                false                 |

|      created_at     |      2013-10-22T06:03:05.019987      |

| display_description |                 None                 |

|     display_name    |              WinSRV2012              |

|          id         | 5812c7e5-d071-43cb-8a1c-81346167d72a |

|       image_id      | 1e26928b-5df0-4097-bbc6-46832dc8361b |

|       metadata      |                  {}                  |

|         size        |                  20                  |

|     snapshot_id     |                 None                 |

|     source_volid    |                 None                 |

|        status       |               creating               |

|     volume_type     |                 None                 |

+———————+————————————–+

[root@localhost boris(keystone_admin)]# cinder list

+————————————–+————-+————–+——+————-+———-+————————————–+

|                  ID                  |    Status   | Display Name | Size | Volume Type | Bootable |             Attached to              |

+————————————–+————-+————–+——+————-+———-+————————————–+

| 5812c7e5-d071-43cb-8a1c-81346167d72a | downloading |  WinSRV2012  |  20  |     None    |  false   |                                      |

| be59b5ae-863b-41e2-abf6-de6756398722 |    in-use   | SalamanderVL |  5   |     None    |   true   | 82ba6ffe-dc01-4fcc-b247-7995a5ca7cb8 |

+————————————–+————-+————–+——+————-+———-+————————————–+

[root@localhost boris(keystone_admin)]# ls -lah /var/lib/cinder/volumes/1d42fffa70f6647a45514f6fb5ce40ca

total 16G

drwxr-xr-x. 3 root   root   4.0K Oct 22 10:03 .

drwxr-xr-x. 5 cinder cinder 4.0K Oct 20 16:30 ..

-rw-rw-rw-. 1 root   root    16G Oct 22 10:21 volume-5812c7e5-d071-43cb-8a1c-81346167d72a

-rw-rw-rw-. 1 root   root   5.0G Oct 21 11:12 volume-be59b5ae-863b-41e2-abf6-de6756398722

[root@localhost boris(keystone_admin)]# du | grep node.$

5517480    ./node1

5516684    ./node3

5514680    ./node2

[root@localhost boris(keystone_admin)]# du | grep node.$

6551272    ./node1

6550168    ./node3

6549416    ./node2

[root@localhost boris(keystone_admin)]# cinder list

+————————————–+———–+————–+——+————-+———-+————————————–+

|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable |             Attached to              |

+————————————–+———–+————–+——+————-+———-+————————————–+

| 5812c7e5-d071-43cb-8a1c-81346167d72a | available |  WinSRV2012  |  20  |     None    |   true   |                                      |

| be59b5ae-863b-41e2-abf6-de6756398722 |   in-use  | SalamanderVL |  5   |     None    |   true   | 82ba6ffe-dc01-4fcc-b247-7995a5ca7cb8 |

+————————————–+———–+————–+——+————-+———-+————————————–+

[root@localhost boris(keystone_admin)]# ls -lah /var/lib/cinder/volumes/1d42fffa70f6647a45514f6fb5ce40ca

total 19G

drwxr-xr-x. 3 root   root   4.0K Oct 22 10:03 .

drwxr-xr-x. 5 cinder cinder 4.0K Oct 20 16:30 ..

-rw-rw-rw-. 1 root   root    20G Oct 22 10:24 volume-5812c7e5-d071-43cb-8a1c-81346167d72a

-rw-rw-rw-. 1 root   root   5.0G Oct 21 11:12 volume-be59b5ae-863b-41e2-abf6-de6756398722

[root@localhost boris(keystone_admin)]# df -h

Filesystem                    Size  Used Avail Use% Mounted on

/dev/mapper/fedora-root       164G   70G   86G  45% /

devtmpfs                      3.9G     0  3.9G   0% /dev

tmpfs                         3.9G   92K  3.9G   1% /dev/shm

tmpfs                         3.9G  964K  3.9G   1% /run

tmpfs                         3.9G     0  3.9G   0% /sys/fs/cgroup

tmpfs                         3.9G  112K  3.9G   1% /tmp

/dev/sda1                     477M   87M  362M  20% /boot

/dev/loop0                    928M  1.4M  860M   1% /srv/node/device1

192.168.1.135:cinder-volumes  490G  209G  257G  45% /var/lib/cinder/volumes/1d42fffa70f6647a45514f6fb5ce40ca

tmpfs                         3.9G  964K  3.9G   1% /run/netns

[root@localhost boris(keystone_admin)]# cinder list

+————————————–+——–+————–+——+————-+———-+————————————–+

|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |

+————————————–+——–+————–+——+————-+———-+————————————–+

| 5812c7e5-d071-43cb-8a1c-81346167d72a | in-use |  WinSRV2012  |  20  |     None    |   true   | d6e702c3-16e6-4890-bbe7-22b19ed05263 |

| be59b5ae-863b-41e2-abf6-de6756398722 | in-use | SalamanderVL |  5   |     None    |   true   | 82ba6ffe-dc01-4fcc-b247-7995a5ca7cb8 |

+————————————–+——–+————–+——+————-+———-+————————————–+

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: