Root314

Docker multi stage builds

2018-05-20T20:00:00+02:00

With Docker 1.17 the long awaited multi stage builds are now available. It allows you to run several stages in your Dockerfile while reducing the size of the final container.

It is typically used for compiled languages like C, Java or Go, where you need a compiler environment to compile the code, but don’t need it for runtime.

The Dockerfile below is an example showing this in action for a Go application, a single docker build . will:

First compile the source code with golang:alpine container (~250MB)
Copy the compiled artefact into a scratch container

The resulting runtime image is only 1.86MB (!) and should be the absolute minimum to run the application.

app.go

package main

import (
        "fmt"
)

func main() {
        fmt.Println("Hello Docker")
}

Dockerfile

# compiler container
FROM golang:alpine as build
ENV CGO_ENABLED=1
ADD . .
RUN go build -o /main

# runtime container
FROM scratch
COPY --from=build /main /main
ENTRYPOINT ["/main"]

You can find this example in GitHub RootPi314/docker-examples/go-multistage

This can be easily adapted to cover any compiled language, more info available in the Docker documentation

Kubernetes All-In-One in 10 minutes

2018-04-02T20:00:00+02:00

Kubespray is a set of Ansible playbooks to deploy a production ready Kubernetes cluster. But we can also use it for quick and easy test/dev Kubernetes clusters.

With Vagrant

The easiest way is to use Vagrant, simply download the Vagrantfile and run vagrant up.

git clone https://github.com/RootPi314/kubespray-aio.git
cd kubespray-aio
vagrant up

Cloud Init

If you prefer to run it with a cloud provider instead of running things locally simply pass the cloud-init.yml file as --user-data.

Do It Yourself & behind the Scene

Finally, if you want to get your hands dirty you’ll need an Ubuntu 16.04 server or VM, then run:

git clone git clone https://github.com/RootPi314/kubespray-aio.git
cd kubespray-aio
./install.sh

This should take around 10 minutes depending your computer and internet connection. After which kubectl get nodes will show you a ready cluster:

$ kubectl get nodes
NAME        STATUS    ROLES         AGE       VERSION
localhost   Ready     master,node   5m       v1.9.2+coreos.0

More documentation and info available on GitHub.com/RootPi314/kubespray-aio

OpenStack Nordic Days 2017

2017-11-30T19:30:00+01:00

Last month I had made a presentation at the OpenSTack Nordic Day 2017 in the Copenhagen: How to get your private cloud project to take off.

Video recording is now available:

OpenStack Days UK 2017

2017-08-30T20:30:00+02:00

The OpenStack community is organising OpenStack Days UK on September 26, 2017 in London. I’ll be presenting how to work with legacy applications in the clouds (Room: London Wall @15:45).

Check out the full schedule and see you there!

VM live migration across the globe with Ceph and Openstack

2017-06-08T22:00:00+02:00

Have you ever been in a situation where you had to migrate a VMs across the globe with minimum downtime? VM live migration across Openstack regions.

Classic approach

The volume import/export capabilities of Openstack are limited, therefore the classic approach is to turn off the VM, copy it to the destination, then attach the volume to a VM in the new region. It works but the VM is down during the transfer and it could mean days of downtime for your application.

That translates in the following Openstack commands:

DST_REGION=RegionTwo
VOL_SIZE=100
# Stop the VM for safety
openstack server stop myserver
# Send the VM snapshot into glance
openstack image create --volume myvolume mysnapshot
# Pipe the glance image into the DST_REGION
glance image-download mysnapshot | glance --os-region-name "$DST_REGION" image-upload --name mysnapshot
# Switch to destination region for all further commands
export OS_REGION_NAME="$DST_REGION"
# Create volume
openstack volume create --image mysnapshot --size $VOL_SIZE myvolume
# Boot new server in DST_REGION
openstack server create --flavor m1.small --image mybaseimage myserver
# Attach the volume to the new VM
openstack server add volume myserver myvolume

This technique is using exclusively the Openstack API but requires downtime during the transfer.

Here comes Ceph

Ceph’s snapshot feature allows this type of migration to be handled like VM live migrations:

Take a live snapshot of the volume
Transfer the snapshot (many GBs)
Stop the original VM
Transfer the changes since the snapshot (few MBs)
Attach volume and start the VM

# Set source region variables
SRC_REGION=RegionOne
SRC_POOL=volumes
SRC_SERVER=src-server
SRC_VOL_ID=00000000-1111-2222-3333-4444444444444

# Set destination region variables
DST_REGION=RegionTwo
DST_POOL=volumes
DST_SERVER=dst-server

# Keep the volume size and name handy
VOL_SIZE=$(openstack volume show $SRC_VOL_ID --format value --column size)
VOL_NAME=$(openstack volume show $SRC_VOL_ID --format value --column name)

# Take a snapshot of volume and keep snapshot ID
SRC_SNAP_ID=$(openstack --os-region-name "$SRC_REGION" snapshot create testrbd --name testsnapshot --force --format value --column id)

# Create volume in destination region and keep its ID
DST_VOL_ID=$(openstack --os-region-name "$DST_REGION" volume create --size $VOL_SIZE --name $VOL_NAME --format value --column id)
ceph-dst:~# rbd rm $DST_POOL/volume-$DST_VOL_ID

# export | import of the snapshot with an SSH pipe for data transfer
ceph-src:~# rbd export $SRC_POOL/volume-$SRC_VOL_ID@snapshot-$SRC_SNAP_ID - | ssh ceph-dst rbd --image-format 2 import - $DST_POOL/volume-$DST_VOL_ID

The export-import can take a long time (hours or days) but since the VM is still running in the source region there is no downtime during the transfer time. Also note that the data transfer is encrypted by SSH in this example.

Once the base snapshot is imported to its destination, we can start with the incremental:

# Unmount the file system on the volume (downtime starts)
src-server:~# umount /path/to/mount/point
ceph-src:~# openstack server remove volume $SRC_SERVER $SRC_VOL_ID

# Create a matching snapshot in the destination region
ceph-dst:~# rbd snap create $DST_POOL/volume-$DST_VOL_ID@snapshot-$SRC_SNAP_ID

# Then export-diff | import-diff the delta between current state and base snapshot
ceph-src:~# rbd export-diff --from-snap snapshot-$SRC_SNAP_ID $SRC_POOL/volume-$SRC_VOL_ID - | ssh ceph-dst rbd import-diff - $DST_POOL/volume-$DST_VOL_ID

# Clean up the temporary snapshot
ceph-dst:~# rbd snap rm $DST_POOL/volume-$DST_VOL_ID@snapshot-$SRC_SNAP_ID

# Attach volume to destination VM and mount the file system (downtime ends)
ceph-dst:~# openstack server add volume $DST_SERVER $DST_VOL_ID
dst-server:~# mount /dev/disk/by-id/virtio-${DST_VOL_ID:0:20} /path/to/mount/point

To summarize

This example demonstrate how to get it going with 1 incremental iteration for simplicity, but you might repeat the export-diff | import-diff a couple of times to reduce the downtime to less than 10 seconds. This type of live volume migration is great tool to critical move VMs accross continents.

Ceph hybrid storage tiers

2017-04-30T00:00:00+02:00

In a previous post I showed you how to deploy storage tiering for Ceph, today I will explain how to setup hybrid storage tiers.

What is hybrid storage?

Hybrid storage is a combination of two different storage tiers like SSD and HDD. In Ceph terms that means that the copies of each objects are located in different tiers - maybe 1 copy on SSD and 2 copies on HDDs.

The idea is to keep 1 copy of the data on a high performance tier (usually SSD or NVMe) and 2 additional copies on a lower cost tier (usually HDDs) in order to improve the read performance at a lower cost.

The following diagram explains the difference between read and write I/O, when using a hybrid storage tier:

How to set it up?

To get this to work in Ceph, we create a two step storage policy:

First step: choose the primary OSD (firstn 1) in the high performance tier, “root-ssd” in the example
Second step: choose the rest of the OSDs (firstn -1) in the low performance tier, “root-hdd” in the example

Assuming a replication factor of 3, the following Ceph ruleset will place 1 copy of each object on SSD and 2 copies on HDDs.

# Hybrid storage policy
rule hybrid {
  ruleset 2
  type replicated
  min_size 1
  max_size 10
  step take root-ssd
  step chooseleaf firstn 1 type host
  step emit
  step take root-hdd
  step chooseleaf firstn -1 type host
  step emit
}

Now that you know how to set it up, it’s up to you to combine all the storage tiers.

The Schrodinger Ceph cluster

2017-04-05T00:00:00+02:00

Inspired by Schrodinger’s famous thought experiment, this is the the story of a Ceph cluster that was both full and empty until reality kicked in.

The cluster

Let’s imagine a small 3 servers cluster. All servers are identical and contain 10x 2 TB hard drives. All Ceph pools are triple replicated on three different hosts.

The total capacity of the cluster is 60 TB raw and 20 TB usable. All is well.

The increase

After some time you need more capacity and decide to add a new node, but this time with 10x 6 TB drives. The anticipated capacity is then 120 TB raw and 40 TB usable.

After the installation ceph status reports the expected raw capacity of 120 TB.

 health HEALTH_OK
[...]
 osdmap e10: 40 osds: 40 up, 40 in
  pgmap v20: 256 pgs, 2 pools, 30 TB data, 1000 objects
        30 TB used, 90 TB / 120 TB avail

The problem

The problem with this setup is that the usable capacity is lower than expected. If you would try to fill the pool with data you would notice that the maximum usable capacity of this cluster is 30TB, that’s 10 TB lower than anticipated: simply because of triple replication.

The table below shows the space usage when you try to to fill this cluster.

Server	Disks	% of Total	Used	Available
ceph1	10x2TB	16.7%	20 TB	0 TB
ceph2	10x2TB	16.7%	20 TB	0 TB
ceph3	10x2TB	16.7%	20 TB	0 TB
ceph4	10x6TB	50%	30 TB	30 TB

In this situation you see that the last node (ceph4) still has 30 TB raw available: that’s the missing 10 TB usable. Since all other nodes (ceph1-3) are full, there is nowhere to store the additional copies required by the storage policy (3 copies on 3 different hosts), so that space is not usable for a triple replication pool.

The solutions

This is happening when using replication with N copies (N=3 in this example) and a single node is responsible for more than 1/N (33% in this example) of the overall cluster capacity.

Theoretical solution

Some would say:

Just use N=1, then you would be guaranteed that no node will be responsible for more than 100% of the cluster capacity

That’s right, but in practice that would disable the data protection (1 copy = no replication), so definitly not what we want.

Equilibrate the cluster

The most practical way to address this is to equilibrate the servers. We can swap half of the 2TB drives in ceph3 with half of the 6 TB drives in ceph4.

Server	2 TB drive	6 TB drive	% of Total
ceph1	10	0	16.7%
ceph2	10	0	16.7%
ceph3	5	5	33%
ceph4	5	5	33%

I recommend you to first update the CRUSHmap, this will start the data re-balancing and will make the new space available quickly, but will relax your storage policy (3 copies on 2 different servers) until you also swap the disks physically.

# Move 5x2TB OSDs to ceph4
for i in 20 21 22 23 24
do
  ceph osd crush set osd.$i 2.0 root=default host=ceph4
done
# Move 5x6TB OSDs to ceph3
for i in 30 31 32 33 34
do
  ceph osd crush set osd.$i 6.0 root=default host=ceph3
done

The basics of Wavelength Division Multiplexing

2017-03-05T13:00:00+01:00

Wavelength Division Multiplexing (WDM) is a multiplexing method which uses different colors (or wavelengths) of light. Where traditional optics allow a single channel of communication on a fiber pair, WDM deployments allow to carry up to 96 channels on a single fiber pair. Channels can support different technologies and different speeds, for example you can mix 1G Ethernet, 10G Ethernet and 4G Fiber Channel on the same fiber run.

Usage

Since it is a multiplexing technology, we combine (mux) the transmission and we split (demux) the reception. With a typical fiber pairs, this is done by a mux/demux passive box as illustrated below.

The optics used in a WDM setup are called colored optics (tuned to a specific wavelength) therefore the color on each end of the fiber must match.

CWDM vs DWDM

WDM exists in two variety: Coarse (CWDM) and Dense (DWDM), each using its own range of wavelengths. Since their wavelength range overlaps, they can co-exist on the same link to some extent.

Name	Channels	Wavelength	Cost	Use case
CWDM	18	1270-1610nm	Low	Short distance
DWDM	96	1520-1577nm	High	Medium/long distance

Reducing interconnect costs

WDM reduces the interconnection costs by using the same fiber pair for many links. Let’s say we want 60Gbps bandwidth (6x10 Gbps) between two point of presence (POP A and B) with dark fiber between them.

To make this a fair example I will assume the following prices:

Dark fiber: 100 €/mo/fiber pair
Traditional LR optic 10Gbps SFP+: 30 €
CWDM optic 10Gbps SFP+: 100 €/unit
Mux-Demux: 700 €/unit

Traditional approach

We need 6 dark fiber pairs (600 €/mo) and 12 optics (360 €). Costing a total of 21 960 € over 3 years.

CWDM

Since we need 6 channels CWDM (up to 16 channels) over a few hundred meters (different rooms) will work just fine. Of course we will need only 1 fiber pair (100 €/mo), and 12 CWDM optics (1200 €). We will also use 1 Mux-Demux at each location (1400 €).

Yielding a total cost of 6 200 €, 70% less than the traditional setup.

Total Cost of Ownership calculator

You can use the following calculator to estimate the TCO of a WDM deployment based on the number of channels. The example costs are based on online prices as of March 2017.

Caveats

Using WDM requires additional documentation to keep track of which wavelengths are used by which links. Be mindful of the physical layer and do not put redundant links on the same fiber pair.

Conclusion

WDM is a great tool to deliver high speed (up to 960 Gbps) on a single fiber pair and to keep costs down.

Edit the Ceph CRUSHmap

2017-01-29T19:00:00+01:00

The CRUSHmap, as suggested by the name, is a map of your storage cluster. This map is necessary for the CRUSH algorithm to determine data placements. But Ceph’s CRUSHmap is stored in binary form. So how to easily change it?

Native tools

Ceph comes with a couple of native commands to do basic customizations to the CRUSHmap:

Reweight

ceph osd crush reweight {name} {weight}

You use this to adjust the weight of an OSD or a bucket. It’s very useful when some OSDs are getting more used than others, as it allows to lower the weights of the more busy drives or nodes.

Remove

ceph osd remove {name|bucket-name}

You use this to either clean up old buckets, or when you decomission OSDs.

Move or add, set location and weight

ceph osd crush set {id-or-name} {weight} root={pool-name} [{bucket-type}={bucket-name} ...]

This is one of the most interesting commands. It does 3 things at once to the specified OSD or bucket:

If the specified OSD or bucket does not exist - it creates it. So be careful with typos, oosd.0 is probably not what you meant :)
It changes the location
It changes the weight

This command is pretty useful when you are physically moving things in your cluster.

Read and write the map

# Read
ceph osd getcrushmap -o {output file}
# Write
ceph osd setcrushmap -i {input file}

If you want to customize anything else (not covered in ceph osd crush) then you will need to download the CRUSHmap, edit it and then upload the new version. But since the CRUSHmap is in binary format you have to convert it to and from human readable text.

ceph osd getcrushmap -o map.bin returns the map in its binary form and crushtool -d map.bin -o map.txt converts the binary file into a human readable text file.
You can edit the map.txt with your favorite text editor.
To apply your changes, you first need to convert the edited text file to binary with crushtool -c map.txt -o map.bin and then to apply your changes with ceph osd setcrushmap -i map.bin

Helper scripts

To make it easy for you I made a pair of helper scripts that takes care of the conversion transparently:

ceph-get-crushmap

This first one just outputs the current CRUSHmap to stdout. Perfect to combine with a pipeline.

# Get and convert the CRUSHmap
tmp=$(mktemp)
ceph osd getcrushmap -o "$tmp" && crushtool -d "$tmp" -o /dev/stdout
rm "$tmp"

ceph-set-crushmap

This second script applies the CRUSHmap from stdin to the Ceph cluster.

# Convert and set the CRUSHmap
tmp=$(mktemp)
crushtool -c /dev/stdin -o "$tmp" && ceph osd setcrushmap -i "$tmp"
rm "$tmp"

Example

Assuming you have both scripts in your $PATH, you could easily rename the host type to server.

ceph-get-crushmap | sed 's/host/server/' | get-set-crushmap

Deploying Ceph with storage tiering

2017-01-15T21:00:00+01:00

You have several options to deploy storage tiering within Ceph. In this post I will show you a simple yet powerful approach to automatically update the CRUSHmap and create storage policies.

Some basics

Storage tiering means having several tiers available. The classic 3 tiered approach is:

fast: all flash
medium: disks accelerated by some flash journals
slow: archive disks with collocated journals

Tiered CRUSHmap

First we will configure the crush location hook. It is a script invoked on OSD start to determine the OSD’s location in the CRUSHmap. To make things simple I use the size of a disk to find out which tier it should belong to:

Bigger than 6 TB → Archive drive
Between 1.6TB and 6TB → Disk with flash journal
Smaller than 1.6TB → SSD assumed

This works in most environments but you might want to adjust the script to fit your environment.

Append the following code to a copy of /usr/bin/ceph-crush-location, then specify its path with osd crush location hook in ceph.conf.

# more than 6TB for slow
size_limit_slow=6000
# more than 1.6TB for medium
size_limit_medium=1600

size=$(df /var/lib/ceph/osd/ceph-$id | awk '{if(NR > 1){printf "%d", $2/1024/1024}}')
if [ "$size" -gt "$size_limit_slow" ]
then
  tier="slow"
if [ "$size" -gt "$size_limit_medium" ]
then
  tier="medium"
else
  tier="fast"
fi
echo "host=$(hostname -s)-$tier root=$tier"

After a restart your OSDs will show up in a tier specific root, the OSD tree should look like that:

root fast
- host ceph-1-fast
- host ceph-2-fast
- host ceph-3-fast
root medium
- host ceph-1-medium
- host ceph-2-medium
- host ceph-3-medium
root slow
- host ceph-1-slow
- host ceph-2-slow
- host ceph-3-slow

Creating rulesets

Rulesets allow you to describe your storage policies. We will use rulesets to restrict storage pools to each tiers. You can easily do this by editing the CRUSHmap. Below is an example of rulesets for replicated pools with copies stored on different hosts.

rule fast {
  ruleset 1
  type replicated
  min_size 1
  max_size 10
  step take fast
  step chooseleaf firstn 0 type host
  step emit
}
rule medium {
  ruleset 2
  type replicated
  min_size 1
  max_size 10
  step take medium
  step chooseleaf firstn 0 type host
  step emit
}
rule slow {
  ruleset 3
  type replicated
  min_size 1
  max_size 10
  step take slow
  step chooseleaf firstn 0 type host
  step emit
}

Bring it all together

To finish you simply set the appropriate ruleset to each storage pool as shown below and you are ready to go.

# Set fast tier for the rbd-fast
ceph osd pool set rbd-fast crush_ruleset 1
# Set medium tier for the rbd pool
ceph osd pool set rbd crush_ruleset 2
# Set slow tier for the "archives" pool
ceph osd pool set archives crush_ruleset 3

Monitoring

If you are doing Ceph tiering in production, you quickly realize that the output of ceph status shows the combined available and used space of all tiers.

To find the used space of each storage tier use ceph osd df tree. You can reflect that in your monitoring system with the following command:

# Show percentage space used, space used and
ceph@ceph-1:~# sudo ceph osd df tree | grep 'root ' | awk '{print $10 ":", $7 "%" " " $5 "/" $4}'
fast: 50.11% 1169G/2332G
medium: 23.28% 3059G/13142G
slow: 10.19% 6153G/60383G

Edit

Since Ceph Luminious there is native support for storage tiering under the name device classes.