Vagrant is an excellent tool to quickly spin up a (Linux) VM to test all kinds of things: a new distribution, a configuration management tool, a new version of a software, you name it.
But Vagrant can do more: you can describe multiple VMs and their configuration in a single Vagrantfile. I’ve been using Vagrant for many years to develop various applications and test and verify deployments, and being able to spin up a small, but otherwise complete cluster is a godsend.
Manually defining multiple VMs
A simple example might look like this (directly stolen from the documentation):
Vagrant.configure("2") do |config|
config.vm.provision "shell", inline: "echo Hello"
config.vm.define "web" do |web|
web.vm.box = "apache"
end
config.vm.define "db" do |db|
db.vm.box = "mysql"
end
end
This Vagrantfile creates two VMs: one for MySQL, and another for Apache. This works well if the VMs you’d like to create are quite different from each other, and you need to configure them in significantly different ways.
Creating multiple similar VMs, and a problem
But sometimes, the cluster you’re setting up consists of multiple VMs that are very similar, or even identical, in configuration. I am working on trying out k3s together with Rook, MetalLB, and a current version of Traefik. For a minimal cluster with k3s and Rook Ceph, three nodes are needed. The first node, the k3s master, has a lot of configuration, but the additional nodes basically only need k3s installed; all configuration will be handled by k3s. So a naive approach could look like this:
Vagrant.configure("2") do |config|
config.vm.define "master" do |master|
master.vm.box = "debian/buster"
master.vm.network "private_network", ip: "192.168.33.10"
master.vm.provision "shell", path: "provision-master.sh"
end
for i in 1..2
config.vm.define "node-#{i}" do |node|
node.vm.box = "debian/buster"
node.vm.network "private_network", ip: "192.168.33.#{10+i}"
node.vm.provision "shell", path: "provision-node.sh", args: [ i.to_s ]
end
end
end
The first VM is the master, the second and third are created identically with a for loop. In both blocks, a second network interface is added as a VirtualBox Host-Only network, so the cluster nodes can communicate with each other, and the host can access each node directly. The provision-node.sh will also receive the index variable so it can adjust names and objects to the node that is currently being provisioned.
When you vagrant up this configuration, you will be surprised to see that both node-1 and node-2 will received the IP address 192.168.33.12, and provision-node.sh will be called with 2 in both cases.
What’s happening?
To understand why Vagrant is misreading our intentions, we have to dive into Ruby a little bit. Vagrant is implemented in Ruby, and the Vagrantfile is in fact a Ruby program. The line
config.vm.define "node-#{i}" do |node|
defines a Ruby block (everything from do to the following end line) that is passed to the define method of the vm object of the config object. The block parameter |node| is provided by the define method when the block is executed. Inside the block, you can access properties of the node object, such as the vm object, and set values or call methods, to create the configuration that you need.
Also, the block has access to the variables that are defined in the main program. That’s why we can use Ruby string interpolation to format the node name ("node-#{i}") or the IP address ("192.168.33.#{10+i}") by using the loop variable i. So if this is all as it should be, why has the loop variable 2 as a value in both loops? When we look at the output of vagrant status, we see three VMs, each with it’s own unique name:
Current machine states:
master not created (virtualbox)
node-1 not created (virtualbox)
node-2 not created (virtualbox)
The reason the loop variable has the same value inside the block for both iterations is when the block is executed. You can be forgiven thinking that the code should execute right where it appears in the code: inside the loop, when config.vm.define is called. However, that is only when it is defined. Vagrant will run the Vagrantfile program and construct an internal configuration from it, collecting the blocks for various properties. Only when the complete definition of the configuration has been collected, these blocks are executed and their results taken into consideration.
By the time the blocks for each of the two nodes are executed, the for loop has completed, and the loop variable simply has the last value. Since the variable is only resolved when the block is executed (not when it is defined), the value is 3 for both versions of the blocks. This is really surprising for many people not familiar with this Ruby idiom and the way it is often used. While other languages have the capability to store closures for later use, I have not seen this pattern there, only in Ruby-based applications.
Creating a configuration object
So, how can we overcome this? Since the loop variable is referenced in both instances of the block definition, but doesn’t have the right value by the time the code finally is executed, we somehow need to make sure we have a different variable for each round. One way to achieve this is to define our own config class, and create a config object for each iteration that stores all the info that we’re interested in, and can define the block in such a way that only the config class instance is referenced.
Let’s look at the example:
class Cfg
def initialize(i)
@name = "node-#{i}"
@ip = "192.168.33.#{10 + i}"
end
def configure(config)
config.vm.define @name, primary: @primary do |v|
v.vm.hostname = @name
v.vm.network "private_network", ip: @ip
end
end
end
Vagrant.configure("2") do |config|
# ...
for i in 1..2
cfg = Cfg.new(i)
cfg.configure(config)
end
end
First, we define a Ruby class Cfg that receives two properties, name and ip, when the instances is created (the initialize method). Then, the configure method runs the Vagrant code and uses the instance properties. Finally, in the for loop, we create a new instance of the Cfg class for each iteration, and call each instances configuration method.
Further reading
The code in this post is simplified. If you would like to see the real-world code that stands up a k3s cluster with additional components, head over to github.com/stblassitude/k3s-rook.