unixpowered.com

I was putting a RedHat server onto a SAN and I could not find any clear instructions on how to grow a single mirrored LUN on the fly. Anyway, here are some notes on the process. First the setup: Two LUN’s mirrored across two SAN’s with LVM volume on the top of it. I could have easily just presented another set of mirrored LUN’s, add them to VG and go from there. I wanted to avoid that, as that kind of setup can quickly get out of hand as the number of presented LUN’s grows. If there is a more “sensible” and flexible setup, I would most definitely want to know about it.

For sake of completeness, here are steps to recreate the initial setup I had:

1. Create a mirror from two LUN’s
2. Use the mirror as PV
3. Create a VG using the PV
4. Create LV on the top of the VG
5. Make ext3 filesystem on the top of LV and mount it

Here are the actual steps with some output:

[root@ultra /]# mdadm --create /dev/md10 --level=1 --raid-devices=2 /dev/mapper/mpath4 /dev/mapper/mpath5
mdadm: array /dev/md10 started.
[root@ultra /]# pvcreate /dev/md10
Physical volume "/dev/md10" successfully created
[root@ultra /]# vgcreate testvg /dev/md10
Volume group "testvg" successfully created
[root@ultra /]# lvcreate -l+100%FREE -n testlv testvg
Logical volume "testlv" created
[root@ultra /]# mkfs -t ext3 /dev/testvg/testlv
[root@ultra /]# mount /dev/testvg/testlv /tmp/test

Now the resizing part. There might be a few steps but the upshot is that the filesystem can stay mounted and in use. High level overview of steps to take:

1. Grow the two LUN’s using SAN management software
2. Fail and remove one of the submirrors
3. Force the kernel to see the size increase of the submirror
4. Flush and recreate the multipath device map so multipathing sees the new size
5. Re-add the submirror to the mirror and let it sync
6. Repeat 2-4 for the second submirror
7. Resize the PV
8. Resize the LV
9. Resize the filesystem

First, you fail and remove the submirror:

[root@ultra /]# mdadm /dev/md10 -f /dev/mapper/mpath4 -r /dev/mapper/mpath4
mdadm: set /dev/mapper/mpath4 faulty in /dev/md10
mdadm: hot removed /dev/mapper/mpath4

Now, note all paths to the LUN. Kernel sees a separate device at the end of each path to a LUN. In this case they are sdj, sdt, sdg and sdq.

[root@ultra /]# multipath -ll mpath4
mpath4 (3600508b400011c300000f000008d0000)
[size=12 GB][features="1 queue_if_no_path"][hwhandler="0"]
_ round-robin 0 [prio=100][active]
._ 1:0:3:1 sdj 8:144 [active][ready]
._ 2:0:3:1 sdt 65:48 [active][ready]
_ round-robin 0 [prio=20][enabled]
._ 1:0:2:1 sdg 8:96 [active][ready]
._ 2:0:2:1 sdq 65:0 [active][ready]

At this point the problem is to get the kernel to recognize the new size without reboot. After a lot of trying and sifting through man pages I found that blockdev command does the magic. Then I googled “blockdev resize” and I found this confirming my finding. So, the next step is to probe all logical paths to the LUN:

[root@ultra /]# blockdev --rereadpt /dev/sdj
[root@ultra /]# blockdev --rereadpt /dev/sdt
[root@ultra /]# blockdev --rereadpt /dev/sdg
[root@ultra /]# blockdev --rereadpt /dev/sdq

You should see messages in /var/log/messages about kernel seeing new size on all paths. If you were to issue multipath -ll right now you would see that multipathing is still reporting old size. To fix that, flush the device map of the LUN and then recreate it:

[root@ultra /]# multipath -f mpath4
[root@ultra /]# multipath -v2
create: mpath4 (3600508b400011c300000f000008d0000)
[size=13 GB][features="0"][hwhandler="0"]
_ round-robin 0 [prio=100]
._ 1:0:3:1 sdj 8:144 [ready]
._ 2:0:3:1 sdt 65:48 [ready]
_ round-robin 0 [prio=20]
._ 1:0:2:1 sdg 8:96 [ready]
._ 2:0:2:1 sdq 65:0 [ready]

Multipathing should be reporting the new size. Now you are ready to put back the grown submirror and let the whole mirror sync:

[root@ultra /]# mdadm /dev/md10 -a /dev/mapper/mpath4
mdadm: hot added /dev/mapper/mpath4

When the mirror has synced up, repeat the above process for the second submirror and wait for the sync to finish. Time to grow the mirror device itself:

[root@ultra /]# mdadm --grow /dev/md10 --size=max

After the completion /proc/mdstat should report increase in size of /dev/md10. Moving on you need to grow the PV that resides on /dev/md10:

[root@ultra /]# pvresize /dev/md10
Physical volume "/dev/md10" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

And finally, you need to resize the LV:

[root@ultra /]# lvresize -l+100%FREE testvg/testlv
Extending logical volume testlv to 13.00 GB
Logical volume testlv successfully resized

Of course, don’t forget to grow the filesystem itself:

[root@ultra /]# ext2online /dev/testvg/testlv
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
[root@ultra /]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-rootlv
.                    132304280   5104976 120478588   5% /
/dev/md0                132134     32791     92521  27% /boot
none                   8202920         0   8202920   0% /dev/shm
/dev/mapper/testvg-testlv
.                     13413488     63516  12668820   1% /tmp/test

That should be it. The sync time for huge volumes is going to be something to keep in mind. The whole setup is clean and neat without clutter. I could have opted to mirror using LVM, but there seems to be a strange requirement for third, log volume. It is possible to keep the log in memory, but that supposedly causes resync on boot. Continue Reading

SP on Sun Fire V40z can be configured so that you can access system console over the network as you would on a UltraSPARC machine with Net Management port. Here is a quick way to get started using V40z and RedHat. Before starting, connect up SP network interface to the network.

Now, edit /etc/inittab and add the following line:

co:12345:respawn:/sbin/agetty -t 60 ttyS0 9600 vt100

This will spawn agetty in runlevels 12345 on serial port 1 with 9600 baud rate and vt100 emulation.

Next you need to edit /etc/grub.conf and comment out splashimage line so boot menu gets rendered properly. Then add following two lines:

serial --unit=0 --speed=9600
terminal --timeout=10 console serial

This will initialize serial port 1 after GRUB startup. If you want to use serial port 2 you would set –unit=1. The terminal we will be using are console and serial in that order, with timeout of 10 seconds. Terminal gets selected depending on where keystroke is detected first, before timeout runs out. If timeout expires, first terminal specified is used.

Finally, append following to the kernel line: console=tty0 console=ttyS0,9600n8. So it will end up looking something like this:

kernel /vmlinuz-2.6.9-67.ELsmp ro root=/dev/VolGroup00/LogVol00 rhgb quiet console=tty0 console=ttyS0,9600n8

Now, you need to add serial port device name to /etc/securetty. This file specifies devices where root can log in. Just append ttyS0 (serial port 1) to the end of the file.

Time to reboot and go to BIOS’s Advanced Settings. Select Console Redirection to Serial Port A and verify you have correct baud rate selected. Reboot the server for all changes to take effect.

At this point SP might not have an IP address assigned, so assign it one using V40z’s front panel. Once you configured SP with IP address, subnet mask and default gateway, ssh to the IP address, from local subnet, using the following:

bash-3.00# ssh setup@IP

You will be asked to setup SP usernames and passwords. When you are done, ssh back to the SP using one of the usernames you have set up, and disable and then re-enable Serial Over LAN:

localhost $ platform set console -s platform
localhost $ platform set console -s sp -e -S 9600

After the SP has been re-enabled, it might be a good idea to set up command prompt so you know which server you are logged into:

localhost $ sp set hostname ultra-sp

Now you can access the console using:

ultra-sp $ platform console

After you connect to the console you can get help by pressing CTRL+E followed by c and ?.Here is sample output:

----
ultra-sp $ platform console
[Enter `^Ec?' for help]
Red Hat Enterprise Linux release 4
Kernel 2.4.21-3.EL on an i686

ultra login:
[help]
.    disconnect                        ;    move to another console
a    attach read/write                 b    send broadcast message
c    toggle flow control               d    down a console
e    change escape sequence            f    force attach read/write
g    group info                        i    information dump
L    toggle logging on/off             l?   break sequence list
l0   send break per config file        l1-9 send specific break sequence
m    display the message of the day    o    (re)open the tty and log file
p    replay the last 60 lines          r    replay the last 20 lines
s    spy read only                     u    show host status
v    show version info                 w    who is on this console
x    show console baud info            z    suspend the connection
|    attach local command              ?    print this message
<cr> ignore/abort command              ^R   replay the last line
ooo send character by octal code
----

At this point you should have a usable network console. You might want to make additional setup changes to the SP to fit your environment.

The first time I issued platform console command in SP I got this error:

console: connect: 59372@console: Connection refused

Rebooting SP using sp reboot fixed the issue for me. Continue Reading