Skip to content
This repository was archived by the owner on Oct 16, 2024. It is now read-only.

Commit 6f557fc

Browse files
author
Simon Murray
committed
Support for Centos 7
Support for RedHat based systems. Abstracts away differences between Ubuntu and Centos package names and service providers, which are required for init and upstart to work correctly with monitors and gateways. Service invokation is switched based upon the OS too. The OSD type is updated to activate disks on Centos (this happens implicitly via upstart on Ubuntu). Default parameters have been updated to use the loopback device, this means we aren't reliant on a VM having fixed interface naming (e.g. eth0) for CI. Tests added for local and cloud Centos. Acceptance tests updated to use netstat to check for a working rados gateway as the previous nc command doesn't exist on Centos.
1 parent ae325fb commit 6f557fc

File tree

13 files changed

+216
-62
lines changed

13 files changed

+216
-62
lines changed

‎.travis.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@ rvm:
22
- 1.9.3
33
script:
44
- bundle exec rake validate
5-
- BEAKER_set=openstack BEAKER_debug=yes bundle exec rake beaker
5+
- BEAKER_set=openstack-ubuntu BEAKER_debug=yes bundle exec rake beaker
6+
- BEAKER_set=openstack-centos BEAKER_debug=yes bundle exec rake beaker
67
env:
78
global:
89
- secure: HsYVsN5saRGoAv3MMNHHERP7hRpbYMq6s1wZ6TjLlq5xTU0EaAnoolFx1jELKWQSDxIwFjUBh0kfL6KLUU28hV6AfOWd3FSEI+i0/fbjDMG5wJcuXOIUywQG4CWQFiobGzajRVEpWG1iLsuBztGvKJPX2rxizEHiQzoMYK+rUivL0rV95yC8VLdrYpBUHtboFuCdJ6fhrnOo/89pc9KZhscfkS99kFiSQBocGDrISiS5N99okbnmGmTNzeCytGUmno2MfbjUQh3VBFHTYlldKOEuc4bOfPSpZSTgF85FI0aikdVqv4SpIwEipVwZ7wocQc7LiV46vzB5VmnemVGSmN1OOisJ12sOZmyF9P8mAJHfq+1J5N1Zx/Qc/FXCjXRn2EnuWPOfYAJuS7nWGtaXCdz9O5fIrlVacfKjs/SfOVGq9JOvyvlvAR5uPUb45OncFIWNj8oNz/Zac/7jo66XOognEDbM9/omXitkcIhSjpX5Biil/N3sw3ho5lvbnjKWpeD5U57jpqopvX2zUkrIIk8ZxczYY4h4KcvbfZfZ0onNPMUjtPjHTXQ+K+4gt1wX3xmkt6GZQJU2UXi7x3WKRTGU06ALuGSkzIFn1m4R1f0RNH90JGcU3J5S5QES1ilS1D6ygzQ7wLMbuHPYWMebjyn8AyeaGSpm4pm3VQJBQbA=

‎README.md

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,13 @@ coding device names is bad. However if we model deployment on hardware location
2626
then we can derive the device name, probe the drive partiton type and provision
2727
based on whether ceph-disk has been run.
2828

29+
The OSD provider can also operate on enclosures with SES firmware running on
30+
a SAS expander. In some cases SCSI addresses aren't predicatable and susceptible
31+
to the same enumeration problem as /dev device names. In these cases the devices
32+
can be provisioned with 'Slot 01/Slot 12' which directly correlates with slot names
33+
found in sysfs. The two addressing modes can be used interchangably thus
34+
configuration like 'Slot 01/2:0:0:0' is permissible.
35+
2936
### Usage
3037

3138
The module is exlusively for use with hiera to segregate data from code thus
@@ -88,8 +95,12 @@ ceph::disks:
8895
fstype: 'xfs'
8996
```
9097
98+
It is recommended to enable deep merging so that global configuration can be
99+
defined in common.yaml and role/host specific configuration merged with the
100+
global section.
101+
91102
##Limitations
92103
93-
1. Ubuntu only
94-
2. Keys are implicitly added on the monitor, ergo all keys need to be defined
104+
1. Keys are implicitly added on the monitor, ergo all keys need to be defined
95105
on the monitor node
106+
2. For use with ceph 0.94 (Hammer) or lower

‎lib/puppet/provider/osd/ceph_disk.rb

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,11 @@ def device_prepared?(dev)
6868
def create
6969
command = "ceph-disk prepare --fs-type #{@fstype} /dev/#{@osd_dev} /dev/#{@journal_dev}"
7070
Puppet::Util::Execution.execute(command)
71+
# Upstart automatically does this for us via udev events
72+
if :operatingsystem != 'Ubuntu'
73+
command = "ceph-disk activate /dev/#{@osd_dev}1"
74+
Puppet::Util::Execution.execute(command)
75+
end
7176
end
7277

7378
# Destroy the resource

‎manifests/init.pp

Lines changed: 23 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -57,21 +57,22 @@
5757
# Global configuration
5858
$conf_merge = false,
5959
$conf = {
60-
'global' => {
61-
'fsid' => '62ed9bd6-adf4-11e4-8fb5-3c970ebb2b86',
62-
'mon_initial_members' => $::hostname,
63-
'mon_host' => $::ipaddress,
64-
'public_network' => "${::network_eth0}/24",
65-
'cluster_network' => "${::network_eth0}/24",
66-
'auth_supported' => 'cephx',
67-
'filestore_xattr_use_omap' => true,
68-
'osd_crush_chooseleaf_type' => '0',
60+
'global' => {
61+
'fsid' => '62ed9bd6-adf4-11e4-8fb5-3c970ebb2b86',
62+
'mon_initial_members' => $::hostname,
63+
'mon_host' => '127.0.0.1',
64+
'public_network' => '127.0.0.0/8',
65+
'cluster_network' => '127.0.0.0/8',
66+
'auth_supported' => 'cephx',
67+
'filestore_xattr_use_omap' => true,
68+
'osd_crush_chooseleaf_type' => 0,
6969
},
70-
'osd' => {
70+
'osd' => {
7171
'osd_journal_size' => 100,
7272
},
73-
'client.rgw' => {
74-
'keyring' => '/etc/ceph/ceph.client.rgw.keyring',
73+
'client.radosgw.puppet' => {
74+
'keyring' => '/etc/ceph/ceph.client.radosgw.puppet.keyring',
75+
'rgw frontends' => '"civetweb port=7480"'
7576
},
7677
},
7778
# Monitor configuration
@@ -80,25 +81,25 @@
8081
# Key management
8182
$keys_merge = false,
8283
$keys = {
83-
'/etc/ceph/ceph.client.admin.keyring' => {
84+
'/etc/ceph/ceph.client.admin.keyring' => {
8485
'user' => 'client.admin',
8586
'key' => 'AQBAyNlUmO09CxAA2u2p6s38wKkBXaLWFeD7bA==',
8687
'caps_mon' => 'allow *',
8788
'caps_osd' => 'allow *',
8889
'caps_mds' => 'allow',
8990
},
90-
'/etc/ceph/ceph.client.rgw.keyring' => {
91-
'user' => 'client.rgw',
91+
'/etc/ceph/ceph.client.radosgw.puppet.keyring' => {
92+
'user' => 'client.radosgw.puppet',
9293
'key' => 'AQD+zXZVDljeKRAAKA30V/QvzbI9oUtcxAchog==',
9394
'caps_mon' => 'allow rwx',
9495
'caps_osd' => 'allow rwx',
9596
},
96-
'/var/lib/ceph/bootstrap-osd/ceph.keyring' => {
97+
'/var/lib/ceph/bootstrap-osd/ceph.keyring' => {
9798
'user' => 'client.bootstrap-osd',
9899
'key' => 'AQDLGtpUdYopJxAAnUZHBu0zuI0IEVKTrzmaGg==',
99100
'caps_mon' => 'allow profile bootstrap-osd',
100101
},
101-
'/var/lib/ceph/bootstrap-mds/ceph.keyring' => {
102+
'/var/lib/ceph/bootstrap-mds/ceph.keyring' => {
102103
'user' => 'client.bootstrap-mds',
103104
'key' => 'AQDLGtpUlWDNMRAAVyjXjppZXkEmULAl93MbHQ==',
104105
'caps_mon' => 'allow profile bootstrap-mds',
@@ -117,8 +118,11 @@
117118
},
118119
},
119120
# RGW management
120-
$rgw_id = 'rgw',
121-
) {
121+
$rgw_id = 'radosgw.puppet',
122+
# Parameters
123+
$service_provider = $::ceph::params::service_provider,
124+
$radosgw_package = $::ceph::params::radosgw_package,
125+
) inherits ceph::params {
122126

123127
contain ::ceph::repo
124128
contain ::ceph::install

‎manifests/mon.pp

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,12 @@
1616
# Enable managament by init/upstart
1717
file { [
1818
"/var/lib/ceph/mon/ceph-${::ceph::mon_id}/done",
19-
"/var/lib/ceph/mon/ceph-${::ceph::mon_id}/upstart",
19+
"/var/lib/ceph/mon/ceph-${::ceph::mon_id}/${ceph::service_provider}",
2020
]:
2121
ensure => file,
22+
owner => 'root',
23+
group => 'root',
24+
mode => '0644',
2225
} ->
2326

2427
# Prevent ceph-create-keys from adding in defaults on monitor startup
@@ -36,12 +39,27 @@
3639
} ->
3740

3841
# Finally start the service
39-
service { "ceph-mon-${::ceph::mon_id}":
40-
ensure => running,
41-
provider => 'init',
42-
start => "start ceph-mon id=${::ceph::mon_id}",
43-
status => "status ceph-mon id=${::ceph::mon_id}",
44-
stop => "stop ceph-mon id=${::ceph::mon_id}",
42+
Service['ceph-mon']
43+
44+
case $::operatingsystem {
45+
'Ubuntu': {
46+
service { 'ceph-mon':
47+
ensure => running,
48+
provider => 'init',
49+
start => "start ceph-mon id=${::ceph::mon_id}",
50+
status => "status ceph-mon id=${::ceph::mon_id}",
51+
stop => "stop ceph-mon id=${::ceph::mon_id}",
52+
}
53+
}
54+
default: {
55+
service { 'ceph-mon':
56+
ensure => running,
57+
provider => 'init',
58+
start => "/etc/init.d/ceph start mon.${::ceph::mon_id}",
59+
status => "/etc/init.d/ceph status mon.${::ceph::mon_id}",
60+
stop => "/etc/init.d/ceph stop mon.${::ceph::mon_id}",
61+
}
62+
}
4563
}
4664

4765
}

‎manifests/params.pp

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# == Class: ceph::params
2+
#
3+
# Static platform differences
4+
#
5+
class ceph::params {
6+
7+
case $::operatingsystem {
8+
'Ubuntu': {
9+
$service_provider = 'upstart'
10+
$radosgw_package = 'radosgw'
11+
}
12+
default: {
13+
$service_provider = 'sysvinit'
14+
$radosgw_package = 'ceph-radosgw'
15+
}
16+
}
17+
18+
}

‎manifests/repo.pp

Lines changed: 36 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -8,19 +8,43 @@
88

99
if $::ceph::manage_repo {
1010

11-
include ::apt
12-
13-
apt::source { 'ceph':
14-
location => "http://eu.ceph.com/debian-${::ceph::repo_version}",
15-
release => $::ceph::repo_release,
16-
repos => 'main',
17-
key => {
18-
'id' => '08B73419AC32B4E966C1A330E84AC2C0460F3994',
19-
'source' => 'https://git.ceph.com/release.asc',
20-
},
21-
}
11+
case $::osfamily {
12+
'Debian': {
13+
14+
include ::apt
15+
16+
apt::source { 'ceph':
17+
location => "http://eu.ceph.com/debian-${::ceph::repo_version}",
18+
release => $::ceph::repo_release,
19+
repos => 'main',
20+
key => {
21+
'id' => '08B73419AC32B4E966C1A330E84AC2C0460F3994',
22+
'source' => 'https://git.ceph.com/release.asc',
23+
},
24+
}
25+
26+
Class['::apt'] -> Package <||>
27+
28+
}
2229

23-
Class['::apt'] -> Package <||>
30+
'RedHat': {
31+
32+
yumrepo { 'ceph':
33+
descr => 'Ceph',
34+
baseurl => "http://download.ceph.com/rpm-${::ceph::repo_version}/el\$releasever/x86_64",
35+
priority => 2,
36+
enabled => 1,
37+
gpgcheck => 1,
38+
gpgkey => 'https://download.ceph.com/keys/release.asc',
39+
}
40+
41+
}
42+
43+
default: {
44+
err('Unsupported OS Platform')
45+
}
46+
47+
}
2448

2549
}
2650

‎manifests/rgw.pp

Lines changed: 40 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -8,29 +8,51 @@
88

99
if $::ceph::rgw {
1010

11-
package { 'radosgw':
11+
package { $::ceph::radosgw_package:
1212
ensure => installed,
1313
} ->
1414

15-
file {
16-
"/var/lib/ceph/radosgw/ceph-${::ceph::rgw_id}":
17-
ensure => directory,
18-
owner => 'root',
19-
group => 'root',
20-
mode => '0755';
21-
"/var/lib/ceph/radosgw/ceph-${::ceph::rgw_id}/done":
22-
ensure => file,
23-
owner => 'root',
24-
group => 'root',
25-
mode => '0755',
15+
file { [
16+
'/var/lib/ceph/radosgw',
17+
"/var/lib/ceph/radosgw/ceph-${::ceph::rgw_id}",
18+
]:
19+
ensure => directory,
20+
owner => 'root',
21+
group => 'root',
22+
mode => '0755',
2623
} ->
2724

28-
service { 'radosgw':
29-
ensure => running,
30-
provider => 'init',
31-
start => "start radosgw id=${::ceph::rgw_id}",
32-
status => "status radosgw id=${::ceph::rgw_id}",
33-
stop => "stop radosgw id=${::ceph::rgw_id}",
25+
file { [
26+
"/var/lib/ceph/radosgw/ceph-${::ceph::rgw_id}/done",
27+
"/var/lib/ceph/radosgw/ceph-${::ceph::rgw_id}/${ceph::service_provider}",
28+
]:
29+
ensure => file,
30+
owner => 'root',
31+
group => 'root',
32+
mode => '0644',
33+
} ->
34+
35+
Service['radosgw']
36+
37+
case $::operatingsystem {
38+
'Ubuntu': {
39+
service { 'radosgw':
40+
ensure => running,
41+
provider => 'init',
42+
start => "start radosgw id=${::ceph::rgw_id}",
43+
status => "status radosgw id=${::ceph::rgw_id}",
44+
stop => "stop radosgw id=${::ceph::rgw_id}",
45+
}
46+
}
47+
default: {
48+
service { 'radosgw':
49+
ensure => running,
50+
provider => 'init',
51+
start => '/etc/init.d/ceph-radosgw start',
52+
status => '/etc/init.d/ceph-radosgw status',
53+
stop => '/etc/init.d/ceph-radosgw stop',
54+
}
55+
}
3456
}
3557

3658
}

‎spec/acceptance/all_in_one_spec.rb renamed to ‎spec/acceptance/default_spec.rb

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
require 'spec_helper_acceptance'
22

33
describe 'ceph' do
4-
context 'all-in-one server' do
4+
context 'all services' do
55
it 'provisions with no errors' do
6+
67
# Select the disk layout based on the VM type as this defines
78
# the host bus adaptor and how the disks are presented
89
if default['hypervisor'] == 'vagrant'
@@ -36,6 +37,7 @@
3637
else
3738
raise ArgumentError, "Unsupported hypervisor"
3839
end
40+
3941
pp = <<-EOS
4042
Exec { path => '/bin:/usr/bin:/sbin:/usr/sbin' }
4143
class { 'ceph':
@@ -45,14 +47,15 @@ class { 'ceph':
4547
#{disks}
4648
}
4749
EOS
50+
4851
# Check for clean provisioning and idempotency
4952
apply_manifest(pp, :catch_failures => true)
5053
apply_manifest(pp, :catch_changes => true)
54+
5155
end
5256
it 'accepts http requests' do
5357
# Wait for radosgw to start listening and ensure it works
54-
retry_on(default, 'nc -vz localhost 7480', :max_retries => 30)
55-
shell('curl localhost:7480', :acceptable_exit_codes => 0)
58+
retry_on(default, 'netstat -l | grep 7480', :max_retries => 30)
5659
end
5760
end
5861
end

‎spec/acceptance/nodesets/default-centos.yml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
HOSTS:
3+
puppet:
4+
roles:
5+
- master
6+
- centos
7+
platform: el-7-amd64
8+
distmoduledir: /etc/puppet/modules
9+
box: puppetlabs/centos-7.2-64-nocm
10+
box_url: https://vagrantcloud.com/puppetlabs/boxes/centos-7.2-64-nocm
11+
hypervisor: vagrant
12+
volumes:
13+
osd0:
14+
size: 10000
15+
osd1:
16+
size: 10000
17+
osd2:
18+
size: 10000
19+
journal:
20+
size: 1000
21+
CONFIG:
22+
type: foss

0 commit comments

Comments
 (0)