* You are viewing the archive for the ‘storage’ Category

Quick and dirty SVM cheatsheet

This list focuses mostly on mirror operations. I use Solaris Volume Manager quite a bit when mirroring internal drives. There are tons of additional features and commands, if you use SVM for  things other than mirroring. In that case you might want to look at check out Solaris Volume Manager Administration Guide.

Create database replicas:
metadb -f -a -c [number_of_replicas] [device]
metadb -f -a -c 3 c0t0d0s7

Delete all database replicas from device:
metadb -d [device]
metadb -d c0t0d0s7

Display status of database replicas:
metadb -i
metadb -i

Display metadevice status:
metastat
metastat

Create simple concat/stripe metadevice:
metainit -f [concat_metadevice] 1 1 [device]
metainit -f d21 1 1 c0t0d0s1

Create a mirror with one submirror:
metainit [mirror_metadevice] -m [submirror_metadevice]
metainit d20 -m d21

Attach a submirror to one sided mirror:
metattach [mirror_metadevice] [submirror_metadevice]
metattach d20 d22

Detach a submirror from a mirror:
metadetach [mirror_metadevice] [submirror_metadevice]
metadetach d20 d22

Clear a metadevice:
metaclear [metadevice]
metaclear d22

Offline a submirror:
metaoffline [mirror_metadevice] [submirror_metadevice]
metaoffline d20 d22

Online a submirror:
metaonline [mirror_metadevice] [submirror_metadevice]
metaonline d20 d22

Enable a failed component:
metareplace -e [metadevice] [device]
metareplace -e d21 c0t0d0s1

Rename a metadevice:
metarename [old_metadevice] [new_metadevice]
metarename d20 d30

Switch metadevice names:
metarename [metadevice_1] [metadevice_2]
metarename -x d20 d30

Configure system for root metadevice:
metaroot [metadevice]
metaroot d10
Continue Reading

Linux multipathing

I use MPxIO in Solaris quite often and it works very well for me. This time I needed to test out I/O multipathing in RedHat. What I really needed to do: have a server with two HBA’s manage a mirror which has submirrors on separate SAN’s; so that the server has multiple paths to each submirror. That way, if an HBA goes the server has still connection to both submirrors through the remaining HBA.

Gear used in this “experiment”:

  • Dell Poweredge server.
  • Two Qlogic QLA2310 HBA’s.
  • RHEL Server 5.3 x86.
  • Two SAN’s presenting one LUN each.

Rough steps I took to get this working:

  1. Make sure device mapper package is installed.
  2. Present two LUN’s from two SAN’s.
  3. Probe HBA’s for presented LUN’s.
  4. Configure multipathing.

First and foremost, make sure qla2xxx driver is loaded. You also have to make sure you have device-mapper-multipath-0.4.7-23.el5 installed. Next, configure multipathing daemon so that it starts on boot:

[root@carbon ~]# chkconfig multipathd on

When that’s done you need to make the system aware of the presented LUN’s. One way to do so is to reboot the server. Another option is to force HBA scan:

[root@carbon ~]# echo "- - -" > /sys/class/scsi_host/host1/scan

During this you should watch /var/log/messages to see if your LUN’s are detected. When done, make multipathd aware of the LUN’s:

[root@carbon ~]# multipath -v2 -d

The above command is a “dry run”. There will be no device map changes committed. You will only be shown device mapper changes that will be made. To commit device map changes run:

[root@carbon ~]# multipath -v2

Once this is done you can see what multipathd is seeing:

[root@carbon ~]# multipath -ll
mpath2 (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][enabled]
 \_ 1:0:3:1 sde 8:64 [active][ready]
 \_ 2:0:3:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:2:1 sdd 8:48 [active][ready]
 \_ 2:0:2:1 sdg 8:96 [active][ready]
mpath1 (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][enabled]
 \_ 1:0:0:1 sdb 8:16 [active][ready]
 \_ 2:0:4:1 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:1 sdc 8:32 [active][ready]
 \_ 2:0:1:1 sdf 8:80 [active][ready]

If everything looks good, you can create configuration file for multipathd. You will need to edit /etc/multipath.conf and depending on your environment, add or modify some parameters. The configuration file contains enough comments and examples to figure out what different parameters mean. When in doubt, consult the man pages.

First, add a blacklist section, which will make certain device exempt from multipathing. I have my internal drives listed in blacklist section:

blacklist {
        devnode "^sd[a-b].*"
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
}

Next, you are going to need device section. This is going to be specific to your SAN. The one below is for EVA5000. I got the parameters from HP’s device mapper package:

device {
        vendor                  "HP|COMPAQ"
        product                 "HSV1[01]1 \(C\)COMPAQ|HSV[2][01]0|HSV300"
        path_grouping_policy    group_by_prio
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        path_checker            tur
        path_selector           "round-robin 0"
        prio_callout            "/sbin/mpath_prio_alua /dev/%n"
        rr_weight               uniform
        failback                immediate
        hardware_handler        "0"
        no_path_retry           12
        rr_min_io               100
}

You should also look at defaults section to make sure it is configured for your setup. Again, the parameters in mine are specific to EVA5000:

defaults {
        udev_dir                /dev
        polling_interval        10
        selector                "round-robin 0"
        path_grouping_policy    failover
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            "/bin/true"
        path_checker            tur
        rr_min_io               100
        rr_weight               uniform
        failback                immediate
        no_path_retry           12
        user_friendly_names     yes
        bindings_file           "/var/lib/multipath/bindings"
}

Finally, you will need to specify configuration for the presented LUN’s. This applies to the multipaths section of multipath.conf file:

multipath {
        wwid                    3600508b4001031250000900001490000
        alias                   san1data
}
multipath {
        wwid                    3600508b400011c300000f00001a90000
        alias                   san2data
}

After you are done, restart multipathd and check output of multipath -ll command:

[root@carbon ~]# multipath -ll
san2data (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:3:1 sde 8:64 [active][ready]
 \_ 2:0:3:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:2:1 sdd 8:48 [active][ready]
 \_ 2:0:2:1 sdg 8:96 [active][ready]
san1data (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][enabled]
 \_ 1:0:0:1 sdb 8:16 [active][ready]
 \_ 2:0:4:1 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:1 sdc 8:32 [active][ready]
 \_ 2:0:1:1 sdf 8:80 [active][ready]

That should be it. You should test the setup by disabling paths to see if your LUN’s stay up. Continue Reading

Growing mirrored LUN in RedHat

I was putting a RedHat server onto a SAN and I could not find any clear instructions on how to grow a single mirrored LUN on the fly. Anyway, here are some notes on the process. First the setup: Two LUN’s mirrored across two SAN’s with LVM volume on the top of it. I could have easily just presented another set of mirrored LUN’s, add them to VG and go from there. I wanted to avoid that, as that kind of setup can quickly get out of hand as the number of presented LUN’s grows. If there is a more “sensible” and flexible setup, I would most definitely want to know about it.

For sake of completeness, here are steps to recreate the initial setup I had:

  1. Create a mirror from two LUN’s
  2. Use the mirror as PV
  3. Create a VG using the PV
  4. Create LV on the top of the VG
  5. Make ext3 filesystem on the top of LV and mount it

Here are the actual steps with some output:

[root@ultra /]# mdadm --create /dev/md10 --level=1 --raid-devices=2 /dev/mapper/mpath4 /dev/mapper/mpath5
mdadm: array /dev/md10 started.
[root@
ultra /]# pvcreate /dev/md10
Physical volume "/dev/md10" successfully created
[root@
ultra /]# vgcreate testvg /dev/md10
Volume group "testvg" successfully created
[root@
ultra /]# lvcreate -l+100%FREE -n testlv testvg
Logical volume "testlv" created
[root@
ultra /]# mkfs -t ext3 /dev/testvg/testlv
[root@
ultra /]# mount /dev/testvg/testlv /tmp/test

Now the resizing part. There might be a few steps but the upshot is that the filesystem can stay mounted and in use. High level overview of steps to take:

  1. Grow the two LUN’s using SAN management software
  2. Fail and remove one of the submirrors
  3. Force the kernel to see the size increase of the submirror
  4. Flush and recreate the multipath device map so multipathing sees the new size
  5. Re-add the submirror to the mirror and let it sync
  6. Repeat 2-4 for the second submirror
  7. Resize the PV
  8. Resize the LV
  9. Resize the filesystem

First, you fail and remove the submirror:

[root@ultra /]# mdadm /dev/md10 -f /dev/mapper/mpath4 -r /dev/mapper/mpath4
mdadm: set /dev/mapper/mpath4 faulty in /dev/md10
mdadm: hot removed /dev/mapper/mpath4

Now, note all paths to the LUN. Kernel sees a separate device at the end of each path to a LUN. In this case they are sdj, sdt, sdg and sdq.

[root@ultra /]# multipath -ll mpath4
mpath4 (3600508b400011c300000f000008d0000)
[size=12 GB][features="1 queue_if_no_path"][hwhandler="0"]
_ round-robin 0 [prio=100][active]
._ 1:0:3:1 sdj 8:144 [active][ready]
._ 2:0:3:1 sdt 65:48 [active][ready]
_ round-robin 0 [prio=20][enabled]
._ 1:0:2:1 sdg 8:96 [active][ready]
._ 2:0:2:1 sdq 65:0 [active][ready]

At this point the problem is to get the kernel to recognize the new size without reboot. After a lot of trying and sifting through man pages I found that blockdev command does the magic. Then I googled “blockdev resize” and I found this confirming my finding. So, the next step is to probe all logical paths to the LUN:

[root@ultra /]# blockdev --rereadpt /dev/sdj
[root@
ultra /]# blockdev --rereadpt /dev/sdt
[root@
ultra /]# blockdev --rereadpt /dev/sdg
[root@
ultra /]# blockdev --rereadpt /dev/sdq

You should see messages in /var/log/messages about kernel seeing new size on all paths. If you were to issue multipath -ll right now you would see that multipathing is still reporting old size. To fix that, flush the device map of the LUN and then recreate it:

[root@ultra /]# multipath -f mpath4
[root@
ultra /]# multipath -v2
create: mpath4 (3600508b400011c300000f000008d0000)
[size=13 GB][features="0"][hwhandler="0"]
_ round-robin 0 [prio=100]
._ 1:0:3:1 sdj 8:144 [ready]
._ 2:0:3:1 sdt 65:48 [ready]
_ round-robin 0 [prio=20]
._ 1:0:2:1 sdg 8:96 [ready]
._ 2:0:2:1 sdq 65:0 [ready]

Multipathing should be reporting the new size. Now you are ready to put back the grown submirror and let the whole mirror sync:

[root@ultra /]# mdadm /dev/md10 -a /dev/mapper/mpath4
mdadm: hot added /dev/mapper/mpath4

When the mirror has synced up, repeat the above process for the second submirror and wait for the sync to finish. Time to grow the mirror device itself:

[root@ultra /]# mdadm --grow /dev/md10 --size=max

After the completion /proc/mdstat should report increase in size of /dev/md10. Moving on you need to grow the PV that resides on /dev/md10:

[root@ultra /]# pvresize /dev/md10
Physical volume "/dev/md10" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

And finally, you need to resize the LV:

[root@ultra /]# lvresize -l+100%FREE testvg/testlv
Extending logical volume testlv to 13.00 GB
Logical volume testlv successfully resized

Of course, don’t forget to grow the filesystem itself:

[root@ultra /]# ext2online /dev/testvg/testlv
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
[root@
ultra /]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-rootlv
.                    132304280   5104976 120478588   5% /
/dev/md0                132134     32791     92521  27% /boot
none                   8202920         0   8202920   0% /dev/shm
/dev/mapper/testvg-testlv
.                     13413488     63516  12668820   1% /tmp/test

That should be it. The sync time for huge volumes is going to be something to keep in mind. The whole setup is clean and neat without clutter. I could have opted to mirror using LVM, but there seems to be a strange requirement for third, log volume. It is possible to keep the log in memory, but that supposedly causes resync on boot. Continue Reading

Quick and dirty ZFS cheatsheet

Create simple striped pool:
zpool create [pool_name] [device] [device] ...
zpool create datapool c5t433127A900011C370000C00003210000d0 c5t433127B4001031250000900000540000d0

Create mirrored pool:
zpool create [pool_name] mirror [device] [device] ...
zpool create datapool mirror c5t433127A900011C370000C00003210000d0 c5t433127B4001031250000900000540000d0

Create Raid-Z pool:
zpool create [pool_name] raidz [device] [device] [device] ...
zpool create datapool raidz c5t433127A900011C370000C00003210000d0 c5t433127B4001031250000900000540000d0 c5t439257C4000019250000900000540000d0

Transform simple pool to a mirror:
zpool create [pool_name] [device]
zpool attach [pool_name] [existing_device] [new_device]
zpool create datapool c5t433127A900011C370000C00003210000d0
zpool attach datapool c5t433127A900011C370000C00003210000d0 c5t433127B4001031250000900000540000d0

Expand simple pool:
zpool create [pool_name] [device]
zpool add [pool_name] [new_device]
zpool create datapool c5t433127A900011C370000C00003210000d0
zpool add datapool c5t433127B4001031250000900000540000d0

Expand mirrored pool by attaching additional mirror:
zpool add [pool_name] mirror [new_device] [new_device]
zpool add datapool mirror c5t433127A900011C370000C00003460000d0 c5t433127B400011C370000C00003410000d0

Replace device in a pool:
zpool replace [pool_name] [old_device] [new_device]
zpool replace datapool c5t433127A900011C370000C00003410000d0 c5t433127B4001031250000900000540000d0

Destroy pool:
zpool destroy [pool_name]
zpool destroy datapool

Set pool mountpoint:
zfs set mountpoint=/path [pool_name]
zfs set mountpoint=/export/zfs datapool

Display configured pools:
zpool list
zpool list

Display pool status info:
zpool status [-v] [pool_name]
zpool status -v datapool

Display pool I/O statistics:
zpool iostat [pool_name]
zpool iostat datapool

Display pool command history:
zpool history [pool_name]
zpool history datapool

Export a pool:
zpool export [pool_name]
zpool export datapool

Import a pool:
zpool import [pool_name]
zpool import datapool

Create a filesystem:
zfs create [pool_name]/[fs_name]
zfs create datapool/filesystem

Destroy a filesystem:
zfs destroy [pool_name]/[fs_name]
zfs destroy datapool/filesystem

Rename a filesystem:
zfs rename [pool_name]/[fs_name] [pool_name]/[fs_name]
zfs rename datapool/filesystem datapool/newfilesystem

Move a filesystem:
zfs rename [pool_name]/[fs_name] [pool_name]/[fs_name]/[fs_name]
zfs rename datapool/filesystem datapool/users/filesystem

Display properties of a filesystem:
zfs get all [pool_name]/[fs_name]
zfs get all datapool/filesystem

Make a snapshot:
zfs snapshot [pool_name]/[fs_name]@[time]
zfs snapshot datapool/filesystem@friday

Roll back filesystem to its snapshot:
zfs rollback [pool_name]/[fs_name]@[time]
zfs rollback datapool/filesystem@friday

Clone a filesystem:
zfs snapshot [pool_name]/[fs_name]@[time]
zfs clone [pool_name]/[fs_name]@[time] [pool_name]/[fs_name]
zfs snapshot datapool/filesystem@today
zfs clone datapool/filesystem@today datapool/filesystemclone

Backup filesystem to a file:
zfs send [pool_name]/[fs_name] > /path/to/file
zfs send datapool/filesystem@friday > /tmp/filesystem.bkp

Restore filesystem from a file:
zfs receive [pool_name]/[fs_name] < /path/to/file
zfs receive datapool/restoredfilesystem < /tmp/filesystem.bkp

Create ZFS volume:
zfs create -V [size] [pool_name]/[vol_name]
zfs create -V 100mb datapool/zvolume
newfs /dev/zvol/dsk/datapool/zvolume
Continue Reading

Growing Solaris Volume Manager mirrors online

You have a SVM mirror and you need to grow it. Online. All steps are straight forward except for the last one. So, in a nutshell:

  1. Grow sizes of both submirror LUN’s on SAN
  2. Detach the first submirror metadevice
  3. Clear the submirror metadevice
  4. Make the system see new LUN size
  5. Recreate the submirror metadevice, attach it to the mirror and let it sync
  6. Repeat the above for the second submirror
  7. Not so obvious step
  8. Grow the filesystem

You have a mirror metadevice d200 consisting of submirrors d201 and d202. Both submirrors are sliced in such way that all space is allocated in slice 0. At this point both LUN’s should be already grown, but you need to make the system to see the size increase. Here is what d201, one of the submirrors of d200, should look like:

bash-3.00# metastat d201
d201: Submirror of d200
State: Okay
Size: 2064384 blocks (1008 MB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

First, detach d201 from its parent and clear it:

bash-3.00# metadetach d200 d201
bash-3.00# metaclear d201

The next step is likely to alter the LUN slice configuration, so make sure you have it noted.

You need to run format command to get the system to see the grown LUN. So, run format then select type and then select Autoconfigure. At this point format should be reporting the new LUN size. Make sure your slice configuration is correct and that all the new space is tacked onto the end of correct slice. In this case, it’s slice 0.

Now recreate d201, attach it to d200 and let the mirror sync. When checking status of d200 note that overall mirror size is still the same but the size of d201 increased:

bash-3.00# metastat d200
d200: Mirror
Submirror 0: d202
State: Okay
Submirror 1: d201
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 2064384 blocks (1008 MB)

d202: Submirror of d200
State: Okay
Size: 2064384 blocks (1008 MB)
Stripe 0:
Device                                             Start Block  Dbase        State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003460000d0s0          0     No            Okay   Yes

d201: Submirror of d200
State: Okay
Size: 4128768 blocks (2.0 GB)
Stripe 0:
Device                                             Start Block  Dbase        State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003410000d0s0          0     No            Okay   Yes

Now repeat the same process with d202. After you are done you will have both submirrors bigger, but the size of overall mirror will still be the same:

bash-3.00# metastat d200
d200: Mirror
Submirror 0: d201
State: Okay
Submirror 1: d202
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 2064384 blocks (1008 MB)

d201: Submirror of d200
State: Okay
Size: 4128768 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

d202: Submirror of d200
State: Okay
Size: 4161536 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003460000d0s0 0 No Okay Yes

Now comes the “Not so obvious” step. In order to increase overall mirror size you need to run metattach on the mirror itself:

bash-3.00# metattach d200
bash-3.00# metastat d200
d200: Mirror
Submirror 0: d201
State: Okay
Submirror 1: d202
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 4128768 blocks (2.0 GB)

This will grow the size of the entire mirror to the size of submirror devices. Now you can grow the filesystem on it without unmounting it. Keep in mind that for the duration of growfs command there will be no writes to the volume.

bash-3.00# growfs -M /export/home/users /dev/md/rdsk/d200
/dev/md/rdsk/d200: 4128768 sectors in 126 cylinders of 128 tracks, 256 sectors
2016.0MB in 63 cyl groups (2 c/g, 32.00MB/g, 15040 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 65824, 131616, 197408, 263200, 328992, 394784, 460576, 526368, 592160,
3487008, 3552800, 3618592, 3684384, 3750176, 3815968, 3881760, 3947552,
4013344, 4079136

That’s all there is to it. The “Not so obvious” step is mentioned in the metattach man page. But unless you do this stuff every day, this very useful piece of information can be easily forgotten. Continue Reading

Page 1 of 3123