RAIDZ expansion planner: how much space does adding a disk give?

Version 1.0.3

See how much space you gain by adding disks to an existing RAIDZ vdev one at a time, and how close each step gets to a vdev built at that width from scratch.

Have the vdev's current width, its smallest disk, how much space it already uses, and its recordsize and ashift.

RAIDZ level Required
Smallest existing member unit
Each added disk unit
Current physical allocation unit

Use this vdev's ALLOC value from zpool list -vp.

Workload Required
recordsize Required
ashift Required

What to enter

Enter one RAIDZ vdev, not the whole pool. The planner assumes this vdev has never been expanded.

Current physical allocation is the ALLOC value for this vdev in zpool list -vp. It counts parity too, which is what the planner needs.

Use recordsize for a dataset or volblocksize for a zvol, whichever holds most of your data. zpool get ashift shows the pool's ashift. ashift=12 means 4 KiB sectors.

Share plan puts your inputs in a link you can send or bookmark.

RAIDZ2 from 6 to 8 disks of 8 TB, with 30 TB allocated

Example

With a 128 KiB recordsize and ashift=12, capacity rises from 32 TB to 44.18 TB, a gain of 12.18 TB. A fresh 8-disk RAIDZ2 would hold 45.51 TB, so 1.33 TB is lost to the data written before the expansion.

Why does an expanded vdev hold less than a new one?

Data written before the expansion keeps its old data-to-parity ratio. A 6-wide RAIDZ2 stores 4 data sectors for every 2 parity. After you widen it to 8, those old blocks still use 4 and 2, and only new blocks use 6 and 2. The zpool attach manual describes this.

The planner calls the difference the geometry gap. The fuller the vdev is before you expand, the bigger the gap. Rewriting old data, for example with zfs send and zfs receive, moves it to the new ratio.

Why does ZFS report less free space than the planner?

ZFS keeps estimating space with the vdev's original parity ratio, even after expansion, so zfs list and df show slightly less than the vdev can hold. The manual says so too. The planner shows the space the vdev can hold, not what zfs list will report.

Why does one extra disk sometimes add less than expected?

Block size and padding can make two widths equally efficient. With 128 KiB blocks and 4 KiB sectors, a 7-wide RAIDZ2 stores data no better than a 6-wide one. In the example above, the first 8 TB disk adds 5.33 TB and the second adds 6.85 TB. The ZFS padding calculator shows the allocation for one block at any width.

What should you check before expanding?

CheckWhy
OpenZFS 2.3 or laterRAIDZ expansion arrived in OpenZFS 2.3.0.
The vdev is healthyExpansion pauses if a disk fails until the vdev is repaired.
The new disk is at least as large as the smallest memberSpace above that size goes unused.
A current backupExpansion reads and rewrites every allocated block.

Expansion adds one disk at a time and keeps the parity level, so RAIDZ2 stays RAIDZ2. For a new pool, use the ZFS capacity calculator instead.