Drive Failures

RustFS uses erasure coding to keep data readable and writable when some drives fail. After you replace a failed drive, RustFS heals the missing data onto it automatically.

Unmount the failed diskCopy Anchor Link

Before replacing the physical hard drive, unmount the failed disk at the operating system level to prevent I/O errors in the file system or RustFS during replacement.

# Assume the failed disk is /dev/sdb
umount /dev/sdb

systemctl stop rustfs

Replace the failed diskCopy Anchor Link

After physically replacing the failed disk, you need to partition and format the new disk, and apply the same label as the original disk.

# Format as XFS and apply the same label as the original disk (RustFS requires XFS, like the other disks in the deployment)
mkfs.xfs -L DISK1 /dev/sdb

Requirements

• The new disk is at least as large as the original disk.
• File system type consistent with other disks (XFS, per the installation guide);
• We recommend mounting by label (LABEL) or UUID so that disk order is not affected by a system restart.

Update /etc/fstab or the RustFS configurationCopy Anchor Link

Confirm that the labels or UUIDs in the /etc/fstab entries point to the new disk. The mount point must stay identical to the one listed in RUSTFS_VOLUMES (in /etc/default/rustfs), so the healed disk rejoins the cluster at the same path.

# View current fstab
cat /etc/fstab

# Example fstab entry (no modification needed if labels are the same)
LABEL=DISK1 /data/rustfs0 xfs defaults,noatime 0 2

blkid /dev/sdb
# Get the new partition's UUID, then replace the corresponding field in fstab

mount -a # If no errors, configuration is correct

Remount the new diskCopy Anchor Link

Run the following commands to mount all configured file systems and start RustFS:

mount -a
systemctl start rustfs

Verify that all expected file systems are mounted:

df -h | grep /data/rustfs

Trigger and monitor data healingCopy Anchor Link

Once RustFS detects a freshly formatted disk at a known mount point, its background scanner heals the missing data onto it automatically — no manual command is required. Confirm that recovery has started and track its progress through the service logs:

# For systemd-managed installations
journalctl -u rustfs -f

# Or view the log files under the directory set by RUSTFS_OBS_LOG_DIRECTORY
tail -f /var/log/rustfs/rustfs.log

You can also open the RustFS Console and check the disk status of the affected node.

Follow-up checks and notesCopy Anchor Link

1. Performance Monitoring

• I/O load may fluctuate during healing. We recommend monitoring disk and network utilization.

2. Batch Failures

• If several disks from the same manufacturing batch fail, inspect the remaining disks from that batch more often.

3. Regular failure tests

• Run regular disk failure drills so your team can practice the recovery procedure.

4. Maintenance Windows

• When failure rates are high, schedule dedicated maintenance windows to speed up replacement and healing.