Use LVM on Flatcar Container Linux

LVM - Logical Volume Management - allows you to create logical volumes, for example to use multiple physical disks as one volume. This allows you to make the full use of all attached disks.

Flatcar Linux has built-in support for LVM. This guide covers creation of logical volumes using LVM and how to use them.

Creating LVM

There are two main ways to do this: create everything manually or use an Ignition config. We will first cover the manual way to get a better grip of what is happening, then we will cover the Ignition way.

Manual

You can find all volumes using the lsblk command. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop3     7:3    0  68.3M  1 loop
loop4     7:4    0  39.3M  1 loop
loop5     7:5    0     4K  1 loop
sda       8:0    0 223.6G  0 disk
|-sda1    8:1    0   128M  0 part
|-sda2    8:2    0     2M  0 part
|-sda3    8:3    0     1G  0 part
| `-usr 254:0    0  1016M  1 crypt /usr
|-sda4    8:4    0     1G  0 part
|-sda6    8:6    0   128M  0 part  /oem
|-sda7    8:7    0    64M  0 part
`-sda9    8:9    0 221.3G  0 part  /
sdb       8:16   0 447.1G  0 disk
sdc       8:32   0 447.1G  0 disk
sdd       8:48   0 223.6G  0 disk

Now we know that we have /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd available. However, we cannot use /dev/sda in this scenario, but we can work with the others.

1
2
3
4
5

# pvcreate /dev/sdb /dev/sdc /dev/sdd
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  Physical volume "/dev/sdb" successfully created.
  Physical volume "/dev/sdc" successfully created.
  Physical volume "/dev/sdd" successfully created.

You can verify that everything worked with the following commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

# pvs
  PV         VG Fmt  Attr PSize   PFree
  /dev/sdb      lvm2 ---  447.13g 447.13g
  /dev/sdc      lvm2 ---  447.13g 447.13g
  /dev/sdd      lvm2 ---  223.57g 223.57g
  
# pvdisplay
  "/dev/sdd" is a new physical volume of "223.57 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdd
  VG Name
  PV Size               223.57 GiB
  Allocatable           NO
  PE Size               0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               x0N0k2-5c6j-HlHZ-vuAX-s82V-Yx8v-Pi3rWa

  "/dev/sdc" is a new physical volume of "447.13 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdc
  VG Name
  PV Size               447.13 GiB
  Allocatable           NO
  PE Size               0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               dRa71o-rYk9-gJKC-bJdC-tJVV-yarW-LHTwPu

  "/dev/sdb" is a new physical volume of "447.13 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdb
  VG Name
  PV Size               447.13 GiB
  Allocatable           NO
  PE Size               0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               vu98O9-4UDD-TTGK-PvsY-g4bN-FeL4-lcqLy7

As you can see, you do not yet have a virtual group. You use the vgcreate command to create one and add your PVs.

You need to specify the name of the group and the volumes you want to add to it like so:

1
2

# vgcreate base-layer /dev/sdb /dev/sdc /dev/sdd
  Volume group "base-layer" successfully created

Now you can go ahead and create a logical volume. We recommend to name it according to its purpose as in this example:

1
2

# lvcreate -n vol_docker -l 100%FREE base-layer
 Logical volume "vol_docker" created.

You can verify that everything worked well by issuing the following command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# lvdisplay
  --- Logical volume ---
  LV Path                /dev/base-layer/vol_docker
  LV Name                vol_docker
  VG Name                base-layer
  LV UUID                d0ne0u-zBZQ-29f5-rkd9-XnZv-0vhE-rGmLvA
  LV Write Access        read/write
  LV Creation host, time rrackow-test, 2024-09-18 06:42:09 +0000
  LV Status              available
  # open                 0
  LV Size                1.09 TiB
  Current LE             286163
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

As you can see we now have a total size the sum of the individual disks.

Next we need to use mkfs to create an ext4 filesystem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# mkfs.ext4 /dev/base-layer/vol_docker
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 293030912 4k blocks and 73261056 inodes
Filesystem UUID: eed7e226-87f8-40e0-a49b-21eae4ef9620
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Now you can for example mount the volume for use with docker, by mounting it to /var/lib/docker like so:

1
2

# mkdir /var/lib/docker
# mount /dev/base-layer/vol_docker /var/lib/docker

Ignition

In your Ignition config you will need two units: one to create the volume group and one to mount the volume. Additionally, you will also need a script that executes all the required commands.

We will start with the script. It basically packages everything from the manual part into a script like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/bin/bash
set -euo pipefail


# Function to find all disks
find_volumes(){
  lsblk -d -o NAME,TYPE | awk '$2 == "disk" {print "/dev/" $1}'
}

disks=$(find_volumes)

# Create Physical Volumes
pvcreate "${disks}"

# Create Volume Group
vgcreate vg-root "${disks}"

# Create Logical Volume for data
lvcreate -n vol_root -l 100%FREE vg-root

# Format the data volume with ext4 filesystem
mkfs.ext4 /dev/vg-root/vol_root

As you can see, we use a function to list all available disks. If you do not want to use all disks, you need to adjust the script accordingly. If you want to mount to a different place, e.g. /var/lib/docker instead of /, you need to adjust this bit as well.

The next step is to create the unit file that executes the script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

[Unit]
Description=LVM Setup
ConditionFirstBoot=yes
Before=local-fs-pre.target
[Service]
Type=oneshot
Restart=on-failure
RemainAfterExit=yes
ExecStart=/etc/systemd/multi-user.target/lvm.sh #This is the name and path of the file above
[Install]
WantedBy=multi-user.target

However, we also need to mount the volume we created:

1
2
3
4
5
6
7
8
9

[Unit]
Description=LVM Mount
[Mount]
What=/dev/vg-root/vol_root
Where=/
Type=ext4
Options=defaults
[Install]
WantedBy=local-fs.target

Now we need to put it all together into a butane yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

variant: flatcar
version: 1.0.0
systemd:
  units:
    - name: lvm-setup.service
      enabled: true
      contents: |
        [Unit]
        Description=LVM Setup
        ConditionFirstBoot=yes
        DefaultDependencies=no
        Before=local-fs-pre.target
        [Service]
        Type=oneshot
        Restart=on-failure
        RemainAfterExit=yes
        ExecStart=/opt/lvm.sh
        [Install]
        WantedBy=multi-user.target
    - name: var-lib-docker.mount
      enabled: true
      contents: |
        [Unit]
        Description=Mount LVM to docker dir
        After=lvm-setup.service
        [Mount]
        What=/dev/vg-docker/vol_docker
        Where=/var/lib/docker
        Type=ext4
        Options=defaults
        [Install]
        WantedBy=local-fs.target
    - name: docker.service
      dropins:
        - name: 10-wait-docker.conf
          contents: |
            [Unit]
            After=var-lib-docker.mount
            Requires=var-lib-docker.mount
storage:
  files:
    - path: /opt/lvm.sh
      mode: 0744
      contents:
        inline: |
          #!/bin/bash
          set -euo pipefail


          # Function to find all disks
          find_volumes(){
            lsblk -d -o NAME,TYPE | awk '$2 == "disk" {print "/dev/" $1}'
          }

          disks=$(find_volumes)

          # Create Physical Volumes
          pvcreate "${disks}"

          # Create Volume Group
          vgcreate vg-root "${disks}"

          # Create Logical Volume for data
          lvcreate -n vol_root -l 100%FREE vg-root

          # Format the data volume with ext4 filesystem
          mkfs.ext4 /dev/vg-root/vol_root

As mentioned before, we need to still transpile from a butane yaml to an Ignition config like so:

1

$ docker run --rm -i quay.io/coreos/butane:latest < lvm.yaml > ignition.json

You can verify your config with the following:

1
2

$ cat ignition.json
{"ignition":{"version":"3.3.0"},"storage":{"files":[{"path":"/etc/systemd/system/multi-user.target.wants/lvm.sh","contents":{"compression":"gzip","source":"data:;base64,H4sIAAAAAAAC/3ySQY/TPhTE7+9TzN/Nn20lQrYrbqsiIdTdC6AVQkicKjd5bqy4doidLFW33x05rtosB47Om/mN38Sz/4qttsVW+po8B+TcO7S6ZSW1IaIZHnpbBu0sgoPStoI0BpX2jad43AzO9Hv288WRAOO3pkFeIXf4+vHL+u33n09rvEA+N7jJ7rBaQUSvwLHttA0QRcVDIZAtTzegE1HZsQy8aQc/XyAix6hVNp+GLQjovdwa3qSxEESAct0oh7bIxsE9KkcAoBXaIbGRH9IUdx/eLPGCXcct8l8Qvi9L9l71xhyQxJW4R6jZjpC/Q7PpMTHFKFQ6XtxZjrfisnZ4JY17zvDIIaJhtA9w6sw+V/sqKJtfW1lE66e0yFN98LqUBj9SLXRZMTtOAaeJJ0nx2Lm+pWF31g+7vHJlw90/nJ/dbhKWypZBkrn0ajE4szmDcoPl7e3/D9/W6yt+fFCu28u0evQj/VM861CDf4f3UNqwP/jAe9o3yr8bP47v5MIprkH0JwAA//+D2TD6wwIAAA=="},"mode":484}]},"systemd":{"units":[{"contents":"[Unit]\nDescription=LVM Setup\nConditionFirstBoot=yes\nBefore=local-fs-pre.target\n[Service]\nType=oneshot\nRestart=on-failure\nRemainAfterExit=yes\nExecStart=/etc/systemd/system/multi-user.target.wants/lvm.sh\n[Install]\nWantedBy=multi-user.target\n","enabled":true,"name":"lvm-setup.service"},{"contents":"[Unit]\nDescription=Mount LVM to docker dir\n[Mount]\nWhat=/dev/vg-docker/vol_docker\nWhere=/var/lib/docker\nType=ext4\nOptions=defaults\n[Install]\nAfter=lvm-setup.service\nWantedBy=local-fs.target\n","enabled":true,"name":"var-lib-docker.mount"},{"dropins":[{"contents":"[Unit]\nAfter=var-lib-docker.mount\nRequires=var-lib-docker.mount\n","name":"10-wait-docker.conf"}],"name":"docker.service"}]}}

Add this Ignition config to your cloud provider of choice now as user-data and create the given instance.

Edit this page File documentation issue