Growing mirrored LUN in RedHat

I was putting a RedHat server onto a SAN and I could not find any clear instructions on how to grow a single mirrored LUN on the fly. Anyway, here are some notes on the process. First the setup: Two LUN’s mirrored across two SAN’s with LVM volume on the top of it. I could have easily just presented another set of mirrored LUN’s, add them to VG and go from there. I wanted to avoid that, as that kind of setup can quickly get out of hand as the number of presented LUN’s grows. If there is a more “sensible” and flexible setup, I would most definitely want to know about it.

For sake of completeness, here are steps to recreate the initial setup I had:

  1. Create a mirror from two LUN’s
  2. Use the mirror as PV
  3. Create a VG using the PV
  4. Create LV on the top of the VG
  5. Make ext3 filesystem on the top of LV and mount it

Here are the actual steps with some output:

[root@ultra /]# mdadm --create /dev/md10 --level=1 --raid-devices=2 /dev/mapper/mpath4 /dev/mapper/mpath5
mdadm: array /dev/md10 started.
[root@ultra /]# pvcreate /dev/md10
Physical volume "/dev/md10" successfully created
[root@ultra /]# vgcreate testvg /dev/md10
Volume group "testvg" successfully created
[root@ultra /]# lvcreate -l+100%FREE -n testlv testvg
Logical volume "testlv" created
[root@ultra /]# mkfs -t ext3 /dev/testvg/testlv
[root@ultra /]# mount /dev/testvg/testlv /tmp/test

Now the resizing part. There might be a few steps but the upshot is that the filesystem can stay mounted and in use. High level overview of steps to take:

  1. Grow the two LUN’s using SAN management software
  2. Fail and remove one of the submirrors
  3. Force the kernel to see the size increase of the submirror
  4. Flush and recreate the multipath device map so multipathing sees the new size
  5. Re-add the submirror to the mirror and let it sync
  6. Repeat 2-4 for the second submirror
  7. Resize the PV
  8. Resize the LV
  9. Resize the filesystem

First, you fail and remove the submirror:

[root@ultra /]# mdadm /dev/md10 -f /dev/mapper/mpath4 -r /dev/mapper/mpath4
mdadm: set /dev/mapper/mpath4 faulty in /dev/md10
mdadm: hot removed /dev/mapper/mpath4

Now, note all paths to the LUN. Kernel sees a separate device at the end of each path to a LUN. In this case they are sdj, sdt, sdg and sdq.

[root@ultra /]# multipath -ll mpath4
mpath4 (3600508b400011c300000f000008d0000)
[size=12 GB][features="1 queue_if_no_path"][hwhandler="0"]
_ round-robin 0 [prio=100][active]
._ 1:0:3:1 sdj 8:144 [active][ready]
._ 2:0:3:1 sdt 65:48 [active][ready]
_ round-robin 0 [prio=20][enabled]
._ 1:0:2:1 sdg 8:96 [active][ready]
._ 2:0:2:1 sdq 65:0 [active][ready]

At this point the problem is to get the kernel to recognize the new size without reboot. After a lot of trying and sifting through man pages I found that blockdev command does the magic. Then I googled “blockdev resize” and I found this confirming my finding. So, the next step is to probe all logical paths to the LUN:

[root@ultra /]# blockdev --rereadpt /dev/sdj
[root@ultra /]# blockdev --rereadpt /dev/sdt
[root@ultra /]# blockdev --rereadpt /dev/sdg
[root@ultra /]# blockdev --rereadpt /dev/sdq

You should see messages in /var/log/messages about kernel seeing new size on all paths. If you were to issue multipath -ll right now you would see that multipathing is still reporting old size. To fix that, flush the device map of the LUN and then recreate it:

[root@ultra /]# multipath -f mpath4
[root@ultra /]# multipath -v2
create: mpath4 (3600508b400011c300000f000008d0000)
[size=13 GB][features="0"][hwhandler="0"]
_ round-robin 0 [prio=100]
._ 1:0:3:1 sdj 8:144 [ready]
._ 2:0:3:1 sdt 65:48 [ready]
_ round-robin 0 [prio=20]
._ 1:0:2:1 sdg 8:96 [ready]
._ 2:0:2:1 sdq 65:0 [ready]

Multipathing should be reporting the new size. Now you are ready to put back the grown submirror and let the whole mirror sync:

[root@ultra /]# mdadm /dev/md10 -a /dev/mapper/mpath4
mdadm: hot added /dev/mapper/mpath4

When the mirror has synced up, repeat the above process for the second submirror and wait for the sync to finish. Time to grow the mirror device itself:

[root@ultra /]# mdadm --grow /dev/md10 --size=max

After the completion /proc/mdstat should report increase in size of /dev/md10. Moving on you need to grow the PV that resides on /dev/md10:

[root@ultra /]# pvresize /dev/md10
Physical volume "/dev/md10" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

And finally, you need to resize the LV:

[root@ultra /]# lvresize -l+100%FREE testvg/testlv
Extending logical volume testlv to 13.00 GB
Logical volume testlv successfully resized

Of course, don’t forget to grow the filesystem itself:

[root@ultra /]# ext2online /dev/testvg/testlv
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
[root@ultra /]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
.                    132304280   5104976 120478588   5% /
/dev/md0                132134     32791     92521  27% /boot
none                   8202920         0   8202920   0% /dev/shm
.                     13413488     63516  12668820   1% /tmp/test

That should be it. The sync time for huge volumes is going to be something to keep in mind. The whole setup is clean and neat without clutter. I could have opted to mirror using LVM, but there seems to be a strange requirement for third, log volume. It is possible to keep the log in memory, but that supposedly causes resync on boot.

Posted on February 6, 2009 at 23:52 by somedude · Permalink
In: centos, hp eva, linux, linux tips, linux utilities, lvm2, multipathing, redhat, san, storage

2 Responses

Subscribe to comments via RSS

  1. Written by Henry78
    on March 17, 2009 at 10:54
    Reply · Permalink

    I tried some things on my own here, and came to the same conclusion: there’s no way to do this “online”. If I missed something, I’m thankful for any hints.

Subscribe to comments via RSS

Leave a Reply