unixpowered.com » linux utilities

Growing mirrored LUN in RedHat

somedude — Fri, 06 Feb 2009 22:52:36 +0000

I was putting a RedHat server onto a SAN and I could not find any clear instructions on how to grow a single mirrored LUN on the fly. Anyway, here are some notes on the process. First the setup: Two LUN’s mirrored across two SAN’s with LVM volume on the top of it. I could have easily just presented another set of mirrored LUN’s, add them to VG and go from there. I wanted to avoid that, as that kind of setup can quickly get out of hand as the number of presented LUN’s grows. If there is a more “sensible” and flexible setup, I would most definitely want to know about it.

For sake of completeness, here are steps to recreate the initial setup I had:

Create a mirror from two LUN’s
Use the mirror as PV
Create a VG using the PV
Create LV on the top of the VG
Make ext3 filesystem on the top of LV and mount it

Here are the actual steps with some output:

[root@ultra /]# mdadm --create /dev/md10 --level=1 --raid-devices=2 /dev/mapper/mpath4 /dev/mapper/mpath5 mdadm: array /dev/md10 started. [root@ultra/]# pvcreate /dev/md10 Physical volume "/dev/md10" successfully created [root@ultra/]# vgcreate testvg /dev/md10 Volume group "testvg" successfully created [root@ultra/]# lvcreate -l+100%FREE -n testlv testvg Logical volume "testlv" created [root@ultra/]# mkfs -t ext3 /dev/testvg/testlv [root@ultra /]# mount /dev/testvg/testlv /tmp/test

Now the resizing part. There might be a few steps but the upshot is that the filesystem can stay mounted and in use. High level overview of steps to take:

Grow the two LUN’s using SAN management software
Fail and remove one of the submirrors
Force the kernel to see the size increase of the submirror
Flush and recreate the multipath device map so multipathing sees the new size
Re-add the submirror to the mirror and let it sync
Repeat 2-4 for the second submirror
Resize the PV
Resize the LV
Resize the filesystem

First, you fail and remove the submirror:

[root@ultra/]# mdadm /dev/md10 -f /dev/mapper/mpath4 -r /dev/mapper/mpath4 mdadm: set /dev/mapper/mpath4 faulty in /dev/md10 mdadm: hot removed /dev/mapper/mpath4
Now, note all paths to the LUN. Kernel sees a separate device at the end of each path to a LUN. In this case they are sdj, sdt, sdg and sdq.

[root@ultra/]# multipath -ll mpath4 mpath4 (3600508b400011c300000f000008d0000) [size=12 GB][features="1 queue_if_no_path"][hwhandler="0"] _ round-robin 0 [prio=100][active] ._ 1:0:3:1 sdj 8:144 [active][ready] ._ 2:0:3:1 sdt 65:48 [active][ready] _ round-robin 0 [prio=20][enabled] ._ 1:0:2:1 sdg 8:96 [active][ready] ._ 2:0:2:1 sdq 65:0 [active][ready]

At this point the problem is to get the kernel to recognize the new size without reboot. After a lot of trying and sifting through man pages I found that blockdev command does the magic. Then I googled “blockdev resize” and I found this confirming my finding. So, the next step is to probe all logical paths to the LUN:

[root@ultra/]# blockdev --rereadpt /dev/sdj [root@ultra/]# blockdev --rereadpt /dev/sdt [root@ultra/]# blockdev --rereadpt /dev/sdg [root@ultra /]# blockdev --rereadpt /dev/sdq

You should see messages in /var/log/messages about kernel seeing new size on all paths. If you were to issue multipath -ll right now you would see that multipathing is still reporting old size. To fix that, flush the device map of the LUN and then recreate it:

[root@ultra/]# multipath -f mpath4 [root@ultra/]# multipath -v2 create: mpath4 (3600508b400011c300000f000008d0000) [size=13 GB][features="0"][hwhandler="0"] _ round-robin 0 [prio=100] ._ 1:0:3:1 sdj 8:144 [ready] ._ 2:0:3:1 sdt 65:48 [ready] _ round-robin 0 [prio=20] ._ 1:0:2:1 sdg 8:96 [ready] ._ 2:0:2:1 sdq 65:0 [ready]

Multipathing should be reporting the new size. Now you are ready to put back the grown submirror and let the whole mirror sync:

[root@ultra/]# mdadm /dev/md10 -a /dev/mapper/mpath4 mdadm: hot added /dev/mapper/mpath4

When the mirror has synced up, repeat the above process for the second submirror and wait for the sync to finish. Time to grow the mirror device itself:

[root@ultra /]# mdadm --grow /dev/md10 --size=max

After the completion /proc/mdstat should report increase in size of /dev/md10. Moving on you need to grow the PV that resides on /dev/md10:

[root@ultra/]# pvresize /dev/md10 Physical volume "/dev/md10" changed 1 physical volume(s) resized / 0 physical volume(s) not resized

And finally, you need to resize the LV:

[root@ultra/]# lvresize -l+100%FREE testvg/testlv Extending logical volume testlv to 13.00 GB Logical volume testlv successfully resized

Of course, don’t forget to grow the filesystem itself:

[root@ultra/]# ext2online /dev/testvg/testlv ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b [root@ultra/]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-rootlv . 132304280 5104976 120478588 5% / /dev/md0 132134 32791 92521 27% /boot none 8202920 0 8202920 0% /dev/shm /dev/mapper/testvg-testlv . 13413488 63516 12668820 1% /tmp/test

That should be it. The sync time for huge volumes is going to be something to keep in mind. The whole setup is clean and neat without clutter. I could have opted to mirror using LVM, but there seems to be a strange requirement for third, log volume. It is possible to keep the log in memory, but that supposedly causes resync on boot.

Getting started with SOL on Sun Fire V20z and V40z

somedude — Sat, 04 Oct 2008 11:40:02 +0000

SP on Sun Fire V40z can be configured so that you can access system console over the network as you would on a UltraSPARC machine with Net Management port. Here is a quick way to get started using V40z and RedHat. Before starting, connect up SP network interface to the network.

Now, edit /etc/inittab and add the following line:

co:12345:respawn:/sbin/agetty -t 60 ttyS0 9600 vt100

This will spawn agetty in runlevels 12345 on serial port 1 with 9600 baud rate and vt100 emulation.

Next you need to edit /etc/grub.conf and comment out splashimage line so boot menu gets rendered properly. Then add following two lines:

serial --unit=0 --speed=9600 terminal --timeout=10 console serial

This will initialize serial port 1 after GRUB startup. If you want to use serial port 2 you would set –unit=1. The terminal we will be using are console and serial in that order, with timeout of 10 seconds. Terminal gets selected depending on where keystroke is detected first, before timeout runs out. If timeout expires, first terminal specified is used.

Finally, append following to the kernel line: console=tty0 console=ttyS0,9600n8. So it will end up looking something like this:

kernel /vmlinuz-2.6.9-67.ELsmp ro root=/dev/VolGroup00/LogVol00 rhgb quiet console=tty0 console=ttyS0,9600n8

Now, you need to add serial port device name to /etc/securetty. This file specifies devices where root can log in. Just append ttyS0 (serial port 1) to the end of the file.

Time to reboot and go to BIOS’s Advanced Settings. Select Console Redirection to Serial Port A and verify you have correct baud rate selected. Reboot the server for all changes to take effect.

At this point SP might not have an IP address assigned, so assign it one using V40z’s front panel. Once you configured SP with IP address, subnet mask and default gateway, ssh to the IP address, from local subnet, using the following:

bash-3.00# ssh setup@IP

You will be asked to setup SP usernames and passwords. When you are done, ssh back to the SP using one of the usernames you have set up, and disable and then re-enable Serial Over LAN:

localhost $ platform set console -s platform localhost $ platform set console -s sp -e -S 9600

After the SP has been re-enabled, it might be a good idea to set up command prompt so you know which server you are logged into:

localhost $ sp set hostname ultra-sp

Now you can access the console using:

ultra-sp $ platform console

After you connect to the console you can get help by pressing CTRL+E followed by c and ?.Here is sample output:

---- ultra-sp $ platform console [Enter `^Ec?' for help] Red Hat Enterprise Linux release 4 Kernel 2.4.21-3.EL on an i686

ultra login: [help] . disconnect ; move to another console a attach read/write b send broadcast message c toggle flow control d down a console e change escape sequence f force attach read/write g group info i information dump L toggle logging on/off l? break sequence list l0 send break per config file l1-9 send specific break sequence m display the message of the day o (re)open the tty and log file p replay the last 60 lines r replay the last 20 lines s spy read only u show host status v show version info w who is on this console x show console baud info z suspend the connection | attach local command ? print this message ignore/abort command ^R replay the last line ooo send character by octal code ----

At this point you should have a usable network console. You might want to make additional setup changes to the SP to fit your environment.

The first time I issued platform console command in SP I got this error:

console: connect: 59372@console: Connection refused

Rebooting SP using sp reboot fixed the issue for me.