Collaboratively work on a Puppet module. Vagrant + r10k.

Once a puppet module is in the forge it is quite easy to share it with other people to try it out. But until then, it can be somewhat cumbersome. They are several reasons why releasing a module to the forge can be delayed :

  • a dependent module has not been officialy released yet
  • a pull request being pending on another module one is relying on.
  • a pull request has been merged but one needs to wait for the maintainer to actually do a new release

All those reasons, make the collaboration on a puppet module a litlle bit harder than a simple puppet module install.

In order to tackle this issue and make collaboration / demo of a puppet module easier let’s see how by using Vagrant and r10k this problem can be solved painlessly and in a versionned controlled way.

r10k

r10k is a project that allows one to specify her puppet module dependencies on a file (Puppetfile) and make a proper deployment on a puppet master or a local folder.

Those dependencies can be expressed either as forge puppet module version number or git repository. When specifying git, the checkout process can be bound on master, a specific branch, a specific commit, etc…

mod 'puppetlabs/ntp', '3.0.3'

mod 'stdlib',
:git => 'git://github.com/puppetlabs/puppetlabs-stdlib.git'

Vagrant (provisioner)

I am sure you’ve heard about vagrant, at least once. It’s a great project that makes collaboration easier by scripting the boot + provisionning of a VM. No more excuses like “it works on my machine”.
By simply using the same Vagrant file and the same base box, two users are sure to have the same result. One of the great feature of vagrant is the provisionning on VM creation.
This is the feature that will be used here. To make the point the following Vagrant file will be used :

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

config.vm.box = "MYBASEBOX"
config.vm.provision "puppet" do |puppet|
puppet.module_path = 'modules'
end

end

By default, vagrant will look for a manifet to run in ./manifests/default.pp, hence it is not necessary to specify it here since we will be dropping the file in the correct path.
With the afro mentionned Vagrantfile, vagrant will boot a VM using the MYBASEBOX box and run the manifest/default.pp manifests using the modules/ folder located in the samed directory as your vagrantfile as –modulepath option to puppet.

run.sh (a.k.a “the glue”)

The run.sh script, or however you want to call it, will simply create the appropriate folders and run the r10k install command to retriveve the modules specified in the Puppetfile.
It will then copy an example manifest into manifests/default.pp for vagrant to run it. This is what a basic run.sh could look like :

#!/usr/bin/sh

if [[ ! `gem list r10k` ]];then
gem install r10k
fi
mkdir -p modules
mkdir -p manifests
PUPPETFILE=./Puppetfile PUPPETFILE_DIR=modules r10k --verbose 3 puppetfile install
cp modules/path/to/example.pp manifests/default.pp

It is up to you to customize it to your need.
In this example a file called example.pp was located at the root of the module but it is up to you to have it in an examples folder, simply adapt the script accordingly.

Show time

Long story short, simply run the following command and see the magic happen

vagrant up

Conclusion

By having the Puppetfile and the run.sh script in a git repository, one can easily and repeatedly share the avancement on a given puppet module at any time, without having to wait for upstream release. QED.

Advertisements

Create Puppet modules with solid foundations

Over the last year , the team at PuppetLabs have done a great job making the forge a better place. Also, during this time, they have been pushing puppet module authors to create better modules. By better, three characteristics can be highlighted here; testable – thanks to rspec-puppet, style compliant – thanks to puppet-lint and input-validated – thanks to the validation functions on the stdlib puppetlabs’ module. This post walk you through the process of creating a puppet module taking advantage of these.

Note : It is assumed that you have Puppet, gems and bundler already installed

Create the module

Simply run the following command. If you are root by default the folder will be on /etc/puppet/modules/, if you are logged in as a regular user you’ll find it on ~/.puppet/modules (It relies on $modulepath)

puppet module generate yguenane-mkdir

This will generates boilerplate for a new mkdir module by creating the directory structure and files. Note : the module name needs to have the pattern forgeusername-modulename.

Since the forge last update, new files got pushed as first class citizen file :

  • README : This one still gets displayed as the home page of your module profile
  • CHANGELOG : The changelog gets it’s own tab. One can quickly see the activity of a module without going to the module project page
  • LICENSE : The license file gets it’s own tab also. The license tab contains – as you would guess – the license text

Bring a set of extra features to your module

Gemfile

Create a .gemfile file at the root of your module with the following content

source 'https://rubygems.org'

puppetversion = ENV['PUPPET_VERSION']
gem 'puppet', puppetversion, :require => false
gem 'puppet-lint'
gem 'rspec-puppet'
gem 'puppetlabs_spec_helper', '>= 0.1.0'

Install the bundled gems with the following command

bundle install --gemfile .gemfile

Some of you might ask why .gemfile instead of the traditional and conventional Gemfile, this is PuppetLabs explanation :

Aside: Gemfiles and the Puppet Module Tool. In our modules, we name our Gemfiles .gemfile instead of Gemfile. Dotfiles are automatically ignored when packaging with the Module Tool. We recommend this practice in order to avoid clutter on end-users’ systems, but it is not strictly required.

Everything we need to start working on a puppet module has been installed. Now time for configuration

Rakefile

First of all, a Rakefile needs to be created at the root of your module with the following content

require 'rubygems'
require 'puppetlabs_spec_helper/rake_tasks'
require 'puppet-lint'

Then execute BUNDLE_GEMFILE=.gemfile bundle exec rake help to see what task one can run

Note: Here we preceded our bundle command by setting the BUNDLE_GEMFILE environment variable to tell bundle which file to use. You can rename your .gemfile to Gemfile, since this is the default file you will not need to specify it, but remember to rename it before packaging the module.

rake build            # Build puppet module package
rake clean            # Clean a built module package
rake coverage         # Generate code coverage information
rake help             # Display the list of available rake tasks
rake lint             # Check puppet manifests with puppet-lint
rake spec             # Run spec tests in a clean fixtures directory
rake spec_clean       # Clean up the fixtures directory
rake spec_prep        # Create the fixtures directory
rake spec_standalone  # Run spec tests on an existing fixtures directory

rspec-puppet

rspec-puppet is a project that brings rspec feature to test puppet module. It is lead by one of the puppet community member @rodjek. The necessary gems has been installed when you ran bundle install earlier.

rspec-puppet provide the rspec-puppet-init binary to help you getting started with. Simply run rspec-puppet-init at the root of your module and a bunch of directory will be created under your spec/ folder.

The only thing to configure here is the spec/spec_helper.rb file that should contain this content

require 'rubygems'
require 'puppetlabs_spec_helper/module_spec_helper'

.fixtures.yml

Most probably than not, your module will depend on other modules. Thus your rspec tests will need to know about those modules in order to pass. This is the exact purpose of the .fixtures.yml file located at the root of your module. Before running tests, it will download the module it depends on and set them as fixtures, so your test can actually be run against them.

A .fixtures.yml file looks like this

fixtures:
  repositories:
    stdlib: "git://github.com/puppetlabs/puppetlabs-stdlib"
    dep2: "git://github.com/dep2author/dep2author-dep2module"
  symlinks:
    mymodule: "#{source_dir}"

.travis.yml

This file aims to tell travis-ci.org how to run your test suite. For those who does not know travis-ci it is a hosted continuous integration service that you can use freely to test your open source project. Refer to travis-ci.org official website for more information on how to use it.

This is the travis configuration I use for my own modules

language: ruby
rvm:
  - 1.8.7
  - 1.9.3
  - ruby-head
script:
  - "rake spec SPEC_OPTS='--format documentation'"
env:
  - PUPPET_VERSION="~> 2.7.0"
  - PUPPET_VERSION="~> 3.0.0"
  - PUPPET_VERSION="~> 3.1.0"
matrix:
  allow_failures:
    - rvm: ruby-head
  exclude:
    - rvm: 1.9.3
      env: PUPPET_GEM_VERSION="~> 2.7.0"
    - rvm: ruby-head
      env: PUPPET_GEM_VERSION="~> 2.7.0"
gemfile: .gemfile

Please refer to the official documentation for technical details, explanation of the keywords and key concept to extend this configuration file.

Use them all together

For the following test I will be using yguenane-mkdir module available on github. This sample module has been configured as mention above.

Lint

With the current version of this module executing the following command

BUNDLE_GEMFILE=.gemfile bundle exec rake lint

will output the following result

manifests/dir.pp - WARNING: defined type not documented on line 1
tests/init.pp - WARNING: line has more than 80 characters on line 5
tests/init.pp - WARNING: line has more than 80 characters on line 6
tests/init.pp - WARNING: line has more than 80 characters on line 9

Thanks to this feature you can make sure your module is conformed to PuppetLabs Style Guide. Some constraints can be too much restrictive for some situation (ie. 80chars) in order for puppet-lint to not complain about it you can disable this check by adding this line in your Rakefile – PuppetLint.configuration.send(“disable_check“) – where check is the condition being checked.

For example for the 80 characters constraint the line would be

PuppetLint.configuration.send("disable_80chars")

List of checks is available here

Rspec

With the current version of this module executing the following command

BUNDLE_GEMFILE=.gemfile bundle exec rake spec

will output the following result

..

Finished in 0.28395 seconds
2 examples, 0 failures

It ran the tests under spec/{classes,defines} folders, and 2 tests out of 2 passed.

Once your module is puppet-lint compliant and passes all tests, the module is ready to be uploaded to the forge. Simply use puppet module build to create the tarball and upload it.
Done ! A new puppet style-guide compliant and testable module is on the forge !

Extra

If you are using travis-ci do not forget to add the travis badge on your README that let people now the status of the builds. Information here

Conclusion

Even if it can be seen as a long process at first, it is definitely worth it :

  • Contribution to the module becomes simpler, by running the tests a contributor knows the module status during contributions
  • Testing on multiple puppet and ruby version becomes trivial using travis-ci.org
  • This point is more subjective, but when I see a puppet module with tests and changelog, I have a better feeling about the dedication one is putting into h(er|is) module

The more puppet module authors will do module this way, better the quality of the forge and the quality of the modules you can find there will be. QED

PS: The actual module is available at http://forge.puppetlabs.com/yguenane/mkdir