Enlarging a live mdadm ext4 filesystem

My computer runs a several RAID5 arrays on a handful of drives of various sizes.  I recently replaced one of those drives with one of a larger size, which allowed me to add more space to one of my smaller RAID5 arrays.  I tried to enlarge the RAID filesystem without rebooting, which should theoretically been possible, but I couldn't make it work.  I eventually broke down and rebooted so I could finish the process.  Here's the blow by blow steps.

First, add the new disk to the machine.  Use gdisk to partition it just like the others in the machine.

# gdisk /dev/md126

Once partitioned, add the new partition into the existing RAID array, initially as a spare:

# mdadm --manage /dev/md/home --add /dev/sdb2

The above was instant.  Now enlarge the RAID array to actively use the space from the newly added partition.  This will take a while to shuffle around the storage:

# mdadm --grow --raid-devices=4 --backup-file=/root/grow_md_home.bak /dev/md/home

The above took about 6 hours to sync when growing a 3-disk, 1TB RAID5 to a 4-disk, 1.5TB RAID5.  You can monitor its progress with these two commands:

# mdadm --detail /dev/md/home
# watch cat /proc/mdstat

Here's what the output looks like when the enlargement is finished:

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
     md126 : active raid5 sdb2[4] sdj2[3] sdi2[1] sdh2[0]
      1464734208 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
     unused devices: <none>

The GNU disk utility will now show that the RAID array has some spare, unallocated space equal to the size of the newly added partition:

But gdisk still only knows about enough sectors to recreate the original size because the kernel hasn't realized that the "physical" disk device grew.  A reboot would fix it, but try though I might, I couldn't figure out how to do it live.

# gdisk /dev/md/home

Command (? for help): p
Disk /dev/md126: 2929468416 sectors, 1.4 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 249C1F35-206C-4D49-A2BE-C8F426B77BA5
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 2929468382
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      2929468382   1.4 TiB     8300  Linux filesystem

partx tells the linux kernel that the partition has changed size, but it wouldn't recognize the enlarged array:

# partx /dev/md/home
# 1:      2048-1952976895 (1952974848 sectors, 999923 MB)

I tried various things based on the instruction on the Code Silence and Kernel Hardware web sites, but nothing worked.

# gdisk /dev/md126
GPT fdisk (gdisk) version 0.8.8

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/md126: 2929468416 sectors, 1.4 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 249C1F35-206C-4D49-A2BE-C8F426B77BA5
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1952978910
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      1952976895   931.3 GiB   8300 

Command (? for help): d
Using 1

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-1952978910, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-1952978910, default = 1952978910) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'

Command (? for help): p
Disk /dev/md126: 2929468416 sectors, 1.4 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 249C1F35-206C-4D49-A2BE-C8F426B77BA5
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1952978910
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      1952978910   931.3 GiB   8300  Linux filesystem

Command (? for help): w
Warning! Secondary header is placed too early on the disk! Do you want to
correct this problem? (Y/N): y
Have moved second header and partition table to correct location.

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N):
OK; writing new GUID partition table (GPT) to /dev/md126.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.

# partx /dev/md126
# 1:      2048-1952978910 (1952976863 sectors, 999924 MB)

It looks like I just needed to write the partition table to disk to kick the kernel into realizing that there was more space:

# gdisk /dev/md126
GPT fdisk (gdisk) version 0.8.8

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/md126: 2929468416 sectors, 1.4 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 249C1F35-206C-4D49-A2BE-C8F426B77BA5
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 2929468382
Partitions will be aligned on 2048-sector boundaries
Total free space is 976491486 sectors (465.6 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      1952978910   931.3 GiB   8300  Linux filesystem

Command (? for help): d
Using 1

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-2929468382, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-2929468382, default = 2929468382) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'

Command (? for help): p
Disk /dev/md126: 2929468416 sectors, 1.4 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 249C1F35-206C-4D49-A2BE-C8F426B77BA5
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 2929468382
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      2929468382   1.4 TiB     8300  Linux filesystem

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/md126.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.

# partx /dev/md126
# 1:      2048-2929468382 (2929466335 sectors, 1499886 MB)

# resize2fs /dev/md126p1
resize2fs 1.42.9 (4-Feb-2014)
Filesystem at /dev/md126p1 is mounted on /home; on-line resizing required
old_desc_blocks = 59, new_desc_blocks = 59
The filesystem on /dev/md126p1 is now 244121856 blocks long.

Resize2fs claims that the filesystem is now larger, but it's not... at least not yet:

# df -m /home
Filesystem     1M-blocks    Used Available Use% Mounted on
/dev/md126p1      938481  656787    272142  71% /home

After much trial and error, I finally gave up and rebooted the machine.  After rebooting, running resize2fs finally did its thing:

# df -m
Filesystem     1M-blocks    Used Available Use% Mounted on
/dev/md126p1      938481  657142    271787  71% /home
 
# resize2fs /dev/md126p1
resize2fs 1.42.9 (4-Feb-2014)
Filesystem at /dev/md126p1 is mounted on /home; on-line resizing required
old_desc_blocks = 59, new_desc_blocks = 88
The filesystem on /dev/md126p1 is now 366183291 blocks long.
 

# df -m /home
Filesystem     1M-blocks   Used Available Use% Mounted on
/dev/md126p1     1407786 657141    741094  47% /home

So I guess there's no great moral here, and really not even a great instruction set.  Sorry about that one.  If anybody has any pointers on what I did wrong, I'd love to hear it, as I'm sure this won't be the last time I'll have to try this.