Provisioning server stack with Chef Solo

  19 Sep 2014


This article will look at how chef solo terminology goes with its directory structure.

The first step is creating a new chef solo project by making new directory names chef_solo_proj and cd into the directory.

    mkdir chef_solo_proj && cd chef_solo_proj

And, we have to install Chef, Knife Solo and Berkshelf. In order to do that, we will use Gemfile to manage gems and their versions installation. For those packages that we mentioned above, we could translate it into Gemfile like so:

     #Gemfile
     source 'https://rubygems.org'
     gem 'knife-solo'
     gem 'chef'
     gem 'chef-zero'
     gem 'berkshelf'

Then, run bundler command: bundle install Once the above installation has done, you can use knife to initial chef solo project. My chef repo names di_chef_repo, so I execute the following command:

    bundle exec knife solo init di_chef_repo

The output should look like this:

    WARNING: No knife configuration file found
    Creating kitchen...
    Creating knife.rb in kitchen...
    Creating cupboards...
    Setting up Berkshelf...

If you cd into di_chef_repo and run tree -a -I '.git*', you should see the directory structure of the initial project:

    .
    ├── .chef
    │   └── knife.rb
    ├── Berksfile
    ├── cookbooks
    ├── data_bags
    ├── environments
    ├── nodes
    ├── roles
    └── site-cookbooks

    7 directories, 2 files

First, run echo ‘source “https://supermarket.getchef.com”’ > Berksfile (to avoid warning when we cook the recipe). You can notice that there is .chef directory that contains knife.rb file. We will use this file to configure cookbook_path, node_path, role_path and data_bag_path, if we’d like. (there are a lot more configuration keys for knife.rb, which is available at https://docs.getchef.com/config_rb_knife.html, used for chef central server)

Also, we will use site-cookbooks directory to store our cookbook and cookbooks for other people’s cookbooks because cookbooks will be wiped out every time when you run knife solo cook <chef repo>

Let’s say we would like to create a new cookbook that contains only one Redis recipe.

Create the recipe by executing this command

knife cookbook create -o site-cookbooks redis-zdk

(specifying cookbook directory path)

Then, remove files and directories that are unnecessary for our purpose.

rm -rf CHANGELOG.md README.md definitions files providers resources

And the structure of the recipe will be following:

    .
    ├── attributes
    │   └── default.rb
    ├── metadata.rb
    ├── recipes
    │   └── default.rb
    └── templates
        └── default
            ├── redis-server.erb
            └── redis.conf.erb

    4 directories, 5 files

Berkshelf will look at metadata.rb file to figure out for cookbook details and what needed dependencies to be installed. And, add relavant contents to metadata.rb. For me, I add this:

    name              "redis-ubuntu"
    maintainer        "zdk"
    maintainer_email  ""
    description       "Redis installation for Ubuntu"
    version           "0.0.1"

    recipe "redis-ubuntu", "Install Redis"

    supports "ubuntu"

name is cookbook name, recipe is the key to specify recipe’s name.

Another important key is depends, we use it to specify external cookbooks. (e.g., depends 'build-essential' for specifying a cookbook that install standard build toolchain.)

In cookbook, there is an important file in recipes directory names default.rb, i.e. /recipes/default.rb It is a default recipe for the cookbook that will be installed if you don't specify anything in node or role definition.

Think about how you would install Redis package on the server. I usually run

add-apt-repository ppa:rwky/redis

That will add a stable redis ppa. Then, we need to

apt-get update

The above is what we have done on the shell. And, we will translate this into chef recipe (in recipes/default.rb) Assume, we ran above commands on bash shell. So, we translate it by creating a block like so:

bash 'add redis repo and update' do
     user 'root'
     code <<-CMD
         add-apt-repository ppa:rwky/redis
         apt-get update
     CMD
end

Notice that in a powerful Ruby heredoc is where we execute a sequence of commands. Also, we run them as ‘root’ user. (Remember that it doesn’t load a predefined shell env, e.g. PATH, HOME etc.) Instead of putting apt-get install redis-server in heredoc block, we will use chef abstract package command, like so:

 package 'redis-server'

This will ensure redis server will install across package manager unless it will try to install from source.

That will make the /recipes/default.rb file look like this:

    bash 'adding stable redis ppa' do
      user 'root'
      code <<-EOC
        add-apt-repository ppa:chris-lea/redis-server
        apt-get update
      EOC
    end
    package 'redis-server'

Next, we have to config redis and its configuration file is /etc/redis/redis.conf. Create a template file for the configuration. For this case, we create templates/default/redis.conf.erb. The default template configuration will be a default for the any distribution. i.e. the default template will be used if there is no other template matched the installed distribution. for instance, you are installing on ubuntu14.04 and you have templates/ubuntu-12.04/redis.conf.erb and templates/default/redis.conf.erb the default template will be used.

Copy the contents of /etc/redis/redis.conf to Copy the contents of templates/default/redis.conf.erb` Remember that the template is an erb, i.e. ruby code.

You can use <% ruby code here %> in chef cookbook template file like so:

The above template read the recipe attribute node[:redis][:dont_bind] from attributes/default.rb

Although, you can create node attribute in node definition, but it will be overrided by the cookbook or recipe attribute. Creating attribute in node definition, you could do by creating a file redis-server.json in nodes directory that has

    {
        "redis": {
            "dont_bind" : true
        },
        "run_list":
        [
            "role[redis-server]"
        ]
    }

The run_list key is a list of roles or recipies that will apply to the node, i.e. install and configure the node. In the previous node run_list, we specified role[redis-server]

So, we need to create a role definition, that is redis-server role, and put it in roles directory.

    cat roles/redis-server.json
    {
        "name": "redis-server",
        "description": "Redis server",
        "default_attributes": {
            "redis": {
              "dont_bind": false
            }
        },
        "json_class": "Chef::Role",
        "run_list": [
            "redis-zdk"
        ],
        "chef_type": "role"
    }

Initial your favourite VPS (I tried on Digital Ocean, Ubuntu 14.04)

To prepare your node, run:

bundle exec knife solo prepare <user>@<host>

If you have already copied public key to the node and named the host as chef-anatomy in .ssh/config. you could use

    bundle exec knife solo prepare chef-anatomy

After you run it, you should see the output that looks similar to:

    Bootstrapping Chef...
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 16519  100 16519    0     0  10557      0  0:00:01  0:00:01 --:--:-- 10562
    Downloading Chef 11.12.8 for ubuntu...
    downloading https://www.opscode.com/chef/metadata?v=11.12.8&prerelease=false&nightlies=false&p=ubuntu&pv=14.04&m=x86_64
      to file /tmp/install.sh.19538/metadata.txt
    trying wget...
    url     https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/13.04/x86_64/chef_11.12.8-2_amd64.deb
    md5     708874f3a14484376ec3ab19ae9703f6
    sha256  0e6a2f89d4231782d72a5cd5fbf9609a1e5a3ee38c7fccbebe6707dd86ac271e
    downloaded metadata file looks valid...
    downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/13.04/x86_64/chef_11.12.8-2_amd64.deb
      to file /tmp/install.sh.19538/chef_11.12.8-2_amd64.deb
    trying wget...
    Comparing checksum with sha256sum...
    Installing Chef 11.12.8
    installing with dpkg...
    (Reading database ... 97929 files and directories currently installed.)
    Preparing to unpack .../chef_11.12.8-2_amd64.deb ...
    Unpacking chef (11.12.8-2) over (11.12.8-2) ...
    Setting up chef (11.12.8-2) ...
    Thank you for installing Chef!
    Generating node config 'nodes/chef-anatomy.json'...

That was the good part of knife, it installed chef on the server and create nodes/chef-anatomy.json file in the repo.

Log on to the chef-anatomy server, then check if redis-server is installed by:

dpkg-query -l redis-server

The result is:

dpkg-query: no packages found matching redis-server

Therefore, there is no redis-server package installed.

Let’s try to apply our recipe to the chef-anatomy nodee, just run the following command:

 bundle exec knife solo cook chef-anatomy

There is a long output that describes everything chef client does, e.g. install and config redis-server etc. If you got the last message like:

    ....
    (Trucated output)
    ....

    Running handlers:
    Running handlers complete

    Chef Client finished, 18/18 resources updated in 39.396222152 seconds

That means your chef client has just cooked the server. You could also check redis-server package if it’s now installed,, e.g. dpkg -l redis-server, cat /etc/redis/redis.conf etc.

If you check /etc/redis/redis.conf file on the chef-anatomy, you will see bind 127.0.0. as it reflects the node redis definition.

Let’s try to change it by setting dont_bind to true

    $cat nodes/chef-anatomy.json
    {
      "redis": {
        "dont_bind": true
      },
      "run_list": [
        "role[redis-server]"
      ],
      "automatic": {
        "ipaddress": "chef-anatomy"
      }
    }

Then, re-cook the node.

 bundle exec knife solo cook chef-anatomy

And look for bind 127.0.0 line in /etc/redis/redis.conf. Though, you won’t find it . The bind line has been removed! (due to node[redis][dont_bind] value) Chef will detects what changes we made, e.g. apt-get command, configuration changes etc. It will not do anything if it doesn’t need..That is pretty good when you have too iterate on those commands, configuration. However, Chef will not automatically detect for removing role or recipe.

Starting to use Chef might be hard because there are chef terminology you have to learn, where should you put stuff, toolchain (knife, knife solo, Berkshelf) and how stuff get overrided. Though, I believe the time taken after a while you’ll get used to it and it will be worth it.

That was a simple use case just to get idea how to use Chef, I will write more on production use cases including adding user, secure server, install relational database, setting programming language environment etc.

comments powered by Disqus