btrfs on multiple disks: how a btrfs pool works

Picture a VM with a btrfs filesystem on one 50 GiB data disk. You attach a 100 GiB disk and run btrfs device add. Then one balance can convert the metadata, the filesystem’s record of its files, to raid1.

Of the 150 GiB, roughly 144 GiB is then usable for single data. Mirror the data with raid1, and at most 50 GiB is.

On a cloud VM, each btrfs device is usually a whole data disk: an Azure managed disk, or an Amazon Elastic Block Store (EBS) volume. Every device carries the filesystem’s UUID, which is how btrfs groups them. Where udev runs, each one is registered with the kernel as it appears, so naming any one disk mounts the whole set.

btrfs allocates space in chunks, and each new chunk goes to the disks with the most unallocated space. Traditional RAID works across whole disks of equal size. btrfs applies its RAID rules inside each chunk instead, which is why one btrfs filesystem across multiple disks of different sizes works.

Subvolumes sit on top. Each is a directory tree you can mount on its own with the subvol= option, and all share one pool of free space. That makes a btrfs storage pool useful on a VM: no mount runs out while free space sits on a disk that belongs to another.

A profile sets how many copies btrfs keeps of each chunk and where it puts them. Data and metadata each get their own profile, written data/metadata: single/raid1 means one copy of your files and two of their metadata.

The parity profiles, raid5 and raid6, are left out on purpose. The mkfs.btrfs manual warns that they have known problems and shouldn’t be used in production.

Choosing single or raid1 data on cloud disks

An Azure locally redundant (LRS) managed disk is already written three times in one datacenter. AWS replicates each EBS volume within its Availability Zone. raid1 data adds a copy that btrfs controls, and takes two GiB of disk for every GiB of data.

Scrub reads all the data and checks its checksums. With raid1, it can repair a bad block from the other copy. With single, it finds the damage but has no copy to fix it from.

Keep metadata in raid1 either way, since it describes every file. For data, choose:

• single data when the provider’s replication and your backups are your protection. You get every disk’s full size, and losing a disk loses the data on it.
• raid1 data when you want scrub to repair, or the filesystem to survive losing a whole disk, such as one detached by mistake. It then mounts with the degraded option until you replace the disk.

To make btrfs span multiple disks from the start, name every disk in one command. Check each one with lsblk first, because mkfs.btrfs writes a new, empty filesystem over every device you name:

$ sudo mkfs.btrfs -L data -d single -m raid1 /dev/sdc /dev/sdd /dev/sde
$ sudo mount /dev/sdc /data

mkfs.btrfs picks profiles from the device count when you don’t name them:

• On one device, metadata defaults to dup.
• On several, data defaults to single and metadata to raid1.
• Before btrfs-progs 5.8, the multi-device data default was raid0.

The manual recommends naming both profiles. That keeps the result the same across versions.

You can add a device to the filesystem while it’s mounted. btrfs refuses a disk that already holds a filesystem. The -f flag overrides that by overwriting whatever is on the disk, so use it only on a disk you’ve confirmed is the new one.

1. Attach a new, empty data disk to the VM. An EBS volume has to be in the instance’s Availability Zone.
2. Find its device name with lsblk, and check that it’s the empty disk you just attached.
3. Add it to the pool, then list, disk by disk, what each profile has allocated there:

$ sudo btrfs device add /dev/sdf /data
$ sudo btrfs device usage /data

4. If the filesystem started on one disk, its metadata is still dup, both copies on the same disk. Convert it to raid1 with a balance. The copies then sit on different disks, and the system chunks, which map addresses to places on the disks, follow the metadata:

$ sudo btrfs balance start -mconvert=raid1 /data

The manual says the add itself is instant. btrfs registers the disk and moves no existing data. First, where the disk supports it, btrfs discards (TRIMs) the whole disk, unless you pass -K.

With single data, the new space is usable at once. New chunks go to the disk with the most unallocated space. While that’s the new disk, the next chunks land there.

A balance rewrites existing chunks, and the allocator places them as it places new ones, so old data spreads onto the new disk too. It’s a long job, because every chunk it moves is rewritten with its block pointers updated.

Azure can expand a managed data disk without deallocating the VM, with a size limit:

• A Standard HDD, Standard SSD or Premium SSD disk of 4 TiB or less can’t grow past 4 TiB that way. Deallocate the VM and detach the disk first.
• Premium SSD v2 and Ultra Disk have no such limit.

AWS changes an EBS volume’s size while it’s in use. Grow it in the EC2 console or with the AWS CLI, then continue at step 3.

1. On Azure, grow the disk. Azure rounds the size up to the nearest managed disk size.

$ az disk update --resource-group <resource-group> --name <disk-name> --size-gb 256

2. Rescan the device, because Linux doesn’t see the new size until you do. Replace sdf with your disk’s name. Microsoft allows up to 10 minutes for a Premium SSD v2 or Ultra Disk to show it.

$ echo 1 | sudo tee /sys/class/block/sdf/device/rescan

3. Find the devid of the disk you grew.

$ sudo btrfs filesystem show /data

4. Grow that disk’s share to the whole device with resize <devid>:max. Here sdf, the pool’s fourth disk, is devid 4. Without a devid, resize acts on devid 1, which may not be the disk you grew.

$ sudo btrfs filesystem resize 4:max /data

resize doesn’t touch partitions. If the disk has one, grow the partition first.

Growing goes one way. Microsoft says shrinking an existing disk “isn’t supported and might result in data loss.” AWS can’t shrink an EBS volume either.

A btrfs pool gets smaller a whole disk at a time instead. btrfs device remove moves that disk’s data onto the others. It fails if the disks left can’t meet the profiles, as raid1 can’t on one disk.

Shrinking one Azure disk covers a VM without a pool.

Plain df gives one number for space that btrfs splits by profile. The btrfs(5) glossary says its results aren’t accurate.

btrfs filesystem usage shows the split. Its manual says it’s meant to replace btrfs filesystem df in the long run. Run it as root, since a normal user doesn’t get the per-profile detail:

$ sudo btrfs filesystem usage /data

The Overall section comes first:

The Free (estimated) figure can differ from the Avail column of df, because the two are worked out differently. The min beside it is the smallest the estimate can be when more than one profile is present.

Below the Overall section, root sees each profile with its size and how full it is, then the unallocated space on each disk.

“Multiple profiles: yes” can mean a profile conversion was interrupted. btrfs(5) says to restart the same conversion balance, which carries on until every chunk has the new profile.

Amoeba optimizes block storage. It works on the data disks of running VMs: Azure managed disks and Amazon EBS volumes, on Windows and Linux VMs. It never touches the OS disk or root volume.

• The pool. Amoeba moves a VM’s data disks onto one btrfs pool, sized to your data, with room to grow. On Azure the pool uses Premium SSD v2 where your region offers it, and Premium SSD otherwise. On AWS it uses gp3.
• The move. Amoeba copies your data while your services run. Then it switches each mount to a pool subvolume at the same path, with a short restart of the services that use it. If the switch fails, it puts the original mounts back and tries again.
• The old disks. Once the switch is checked, Amoeba snapshots each old disk and deletes the disk. It keeps the snapshot for a few days, then deletes it too.
• Cost during the move. You pay for both sets of disks only while the move runs. Then you pay for the snapshots while they’re kept.
• Rightsizing. Amoeba grows the pool while it stays mounted, before it fills, and sooner when data arrives fast. When your data goes down and stays down, it shrinks the pool one disk at a time. So you pay for close to what you use.

On Linux, the VM needs a recent kernel with btrfs. Every service that keeps a data mount open must run under systemd. The VM also needs a few free disk slots for the pool.