Growing mirrored LUN in RedHat

somedude — Fri, 06 Feb 2009 22:52:36 +0000

I was putting a RedHat server onto a SAN and I could not find any clear instructions on how to grow a single mirrored LUN on the fly. Anyway, here are some notes on the process. First the setup: Two LUN’s mirrored across two SAN’s with LVM volume on the top of it. I could have easily just presented another set of mirrored LUN’s, add them to VG and go from there. I wanted to avoid that, as that kind of setup can quickly get out of hand as the number of presented LUN’s grows. If there is a more “sensible” and flexible setup, I would most definitely want to know about it.

For sake of completeness, here are steps to recreate the initial setup I had:

Create a mirror from two LUN’s
Use the mirror as PV
Create a VG using the PV
Create LV on the top of the VG
Make ext3 filesystem on the top of LV and mount it

Here are the actual steps with some output:

[root@ultra /]# mdadm --create /dev/md10 --level=1 --raid-devices=2 /dev/mapper/mpath4 /dev/mapper/mpath5 mdadm: array /dev/md10 started. [root@ultra/]# pvcreate /dev/md10 Physical volume "/dev/md10" successfully created [root@ultra/]# vgcreate testvg /dev/md10 Volume group "testvg" successfully created [root@ultra/]# lvcreate -l+100%FREE -n testlv testvg Logical volume "testlv" created [root@ultra/]# mkfs -t ext3 /dev/testvg/testlv [root@ultra /]# mount /dev/testvg/testlv /tmp/test

Now the resizing part. There might be a few steps but the upshot is that the filesystem can stay mounted and in use. High level overview of steps to take:

Grow the two LUN’s using SAN management software
Fail and remove one of the submirrors
Force the kernel to see the size increase of the submirror
Flush and recreate the multipath device map so multipathing sees the new size
Re-add the submirror to the mirror and let it sync
Repeat 2-4 for the second submirror
Resize the PV
Resize the LV
Resize the filesystem

First, you fail and remove the submirror:

[root@ultra/]# mdadm /dev/md10 -f /dev/mapper/mpath4 -r /dev/mapper/mpath4 mdadm: set /dev/mapper/mpath4 faulty in /dev/md10 mdadm: hot removed /dev/mapper/mpath4
Now, note all paths to the LUN. Kernel sees a separate device at the end of each path to a LUN. In this case they are sdj, sdt, sdg and sdq.

[root@ultra/]# multipath -ll mpath4 mpath4 (3600508b400011c300000f000008d0000) [size=12 GB][features="1 queue_if_no_path"][hwhandler="0"] _ round-robin 0 [prio=100][active] ._ 1:0:3:1 sdj 8:144 [active][ready] ._ 2:0:3:1 sdt 65:48 [active][ready] _ round-robin 0 [prio=20][enabled] ._ 1:0:2:1 sdg 8:96 [active][ready] ._ 2:0:2:1 sdq 65:0 [active][ready]

At this point the problem is to get the kernel to recognize the new size without reboot. After a lot of trying and sifting through man pages I found that blockdev command does the magic. Then I googled “blockdev resize” and I found this confirming my finding. So, the next step is to probe all logical paths to the LUN:

[root@ultra/]# blockdev --rereadpt /dev/sdj [root@ultra/]# blockdev --rereadpt /dev/sdt [root@ultra/]# blockdev --rereadpt /dev/sdg [root@ultra /]# blockdev --rereadpt /dev/sdq

You should see messages in /var/log/messages about kernel seeing new size on all paths. If you were to issue multipath -ll right now you would see that multipathing is still reporting old size. To fix that, flush the device map of the LUN and then recreate it:

[root@ultra/]# multipath -f mpath4 [root@ultra/]# multipath -v2 create: mpath4 (3600508b400011c300000f000008d0000) [size=13 GB][features="0"][hwhandler="0"] _ round-robin 0 [prio=100] ._ 1:0:3:1 sdj 8:144 [ready] ._ 2:0:3:1 sdt 65:48 [ready] _ round-robin 0 [prio=20] ._ 1:0:2:1 sdg 8:96 [ready] ._ 2:0:2:1 sdq 65:0 [ready]

Multipathing should be reporting the new size. Now you are ready to put back the grown submirror and let the whole mirror sync:

[root@ultra/]# mdadm /dev/md10 -a /dev/mapper/mpath4 mdadm: hot added /dev/mapper/mpath4

When the mirror has synced up, repeat the above process for the second submirror and wait for the sync to finish. Time to grow the mirror device itself:

[root@ultra /]# mdadm --grow /dev/md10 --size=max

After the completion /proc/mdstat should report increase in size of /dev/md10. Moving on you need to grow the PV that resides on /dev/md10:

[root@ultra/]# pvresize /dev/md10 Physical volume "/dev/md10" changed 1 physical volume(s) resized / 0 physical volume(s) not resized

And finally, you need to resize the LV:

[root@ultra/]# lvresize -l+100%FREE testvg/testlv Extending logical volume testlv to 13.00 GB Logical volume testlv successfully resized

Of course, don’t forget to grow the filesystem itself:

[root@ultra/]# ext2online /dev/testvg/testlv ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b [root@ultra/]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-rootlv . 132304280 5104976 120478588 5% / /dev/md0 132134 32791 92521 27% /boot none 8202920 0 8202920 0% /dev/shm /dev/mapper/testvg-testlv . 13413488 63516 12668820 1% /tmp/test

That should be it. The sync time for huge volumes is going to be something to keep in mind. The whole setup is clean and neat without clutter. I could have opted to mirror using LVM, but there seems to be a strange requirement for third, log volume. It is possible to keep the log in memory, but that supposedly causes resync on boot.

unixpowered.com » lvm2

Growing mirrored LUN in RedHat