unixpowered.com » storage

Quick and dirty SVM cheatsheet

somedude — Thu, 03 Dec 2009 12:50:43 +0000

This list focuses mostly on mirror operations. I use Solaris Volume Manager quite a bit when mirroring internal drives. There are tons of additional features and commands, if you use SVM for things other than mirroring. In that case you might want to look at check out Solaris Volume Manager Administration Guide.

Create database replicas:
metadb -f -a -c [number_of_replicas] [device] metadb -f -a -c 3 c0t0d0s7

Delete all database replicas from device:
metadb -d [device] metadb -d c0t0d0s7

Display status of database replicas:
metadb -i metadb -i

Display metadevice status:
metastat metastat

Create simple concat/stripe metadevice:
metainit -f [concat_metadevice] 1 1 [device] metainit -f d21 1 1 c0t0d0s1

Create a mirror with one submirror:
metainit [mirror_metadevice] -m [submirror_metadevice] metainit d20 -m d21

Attach a submirror to one sided mirror:
metattach [mirror_metadevice] [submirror_metadevice] metattach d20 d22

Detach a submirror from a mirror:
metadetach [mirror_metadevice] [submirror_metadevice] metadetach d20 d22

Clear a metadevice:
metaclear [metadevice] metaclear d22

Offline a submirror:
metaoffline [mirror_metadevice] [submirror_metadevice] metaoffline d20 d22

Online a submirror:
metaonline [mirror_metadevice] [submirror_metadevice] metaonline d20 d22

Enable a failed component:
metareplace -e [metadevice] [device] metareplace -e d21 c0t0d0s1

Rename a metadevice:
metarename [old_metadevice] [new_metadevice] metarename d20 d30

Switch metadevice names:
metarename [metadevice_1] [metadevice_2] metarename -x d20 d30

Configure system for root metadevice:
metaroot [metadevice] metaroot d10

Linux multipathing

somedude — Fri, 27 Nov 2009 22:40:46 +0000

I use MPxIO in Solaris quite often and it works very well for me. This time I needed to test out I/O multipathing in RedHat. What I really needed to do: have a server with two HBA’s manage a mirror which has submirrors on separate SAN’s; so that the server has multiple paths to each submirror. That way, if an HBA goes the server has still connection to both submirrors through the remaining HBA.

Gear used in this “experiment”:

Dell Poweredge server.
Two Qlogic QLA2310 HBA’s.
RHEL Server 5.3 x86.
Two SAN’s presenting one LUN each.

Rough steps I took to get this working:

Make sure device mapper package is installed.
Present two LUN’s from two SAN’s.
Probe HBA’s for presented LUN’s.
Configure multipathing.

First and foremost, make sure qla2xxx driver is loaded. You also have to make sure you have device-mapper-multipath-0.4.7-23.el5 installed. Next, configure multipathing daemon so that it starts on boot:

[root@carbon ~]# chkconfig multipathd on

When that’s done you need to make the system aware of the presented LUN’s. One way to do so is to reboot the server. Another option is to force HBA scan:

[root@carbon ~]# echo "- - -" > /sys/class/scsi_host/host1/scan

During this you should watch /var/log/messages to see if your LUN’s are detected. When done, make multipathd aware of the LUN’s:

[root@carbon ~]# multipath -v2 -d

The above command is a “dry run”. There will be no device map changes committed. You will only be shown device mapper changes that will be made. To commit device map changes run:

[root@carbon ~]# multipath -v2

Once this is done you can see what multipathd is seeing:

[root@carbon ~]# multipath -ll mpath2 (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][enabled] \_ 1:0:3:1 sde 8:64 [active][ready] \_ 2:0:3:1 sdh 8:112 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:2:1 sdd 8:48 [active][ready] \_ 2:0:2:1 sdg 8:96 [active][ready] mpath1 (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][enabled] \_ 1:0:0:1 sdb 8:16 [active][ready] \_ 2:0:4:1 sdi 8:128 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:1:1 sdc 8:32 [active][ready] \_ 2:0:1:1 sdf 8:80 [active][ready]

If everything looks good, you can create configuration file for multipathd. You will need to edit /etc/multipath.conf and depending on your environment, add or modify some parameters. The configuration file contains enough comments and examples to figure out what different parameters mean. When in doubt, consult the man pages.

First, add a blacklist section, which will make certain device exempt from multipathing. I have my internal drives listed in blacklist section:

blacklist { devnode "^sd[a-b].*" devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]" }

Next, you are going to need device section. This is going to be specific to your SAN. The one below is for EVA5000. I got the parameters from HP’s device mapper package:

device { vendor "HP|COMPAQ" product "HSV1[01]1 \(C\)COMPAQ|HSV[2][01]0|HSV300" path_grouping_policy group_by_prio getuid_callout "/sbin/scsi_id -g -u -s /block/%n" path_checker tur path_selector "round-robin 0" prio_callout "/sbin/mpath_prio_alua /dev/%n" rr_weight uniform failback immediate hardware_handler "0" no_path_retry 12 rr_min_io 100 }

You should also look at defaults section to make sure it is configured for your setup. Again, the parameters in mine are specific to EVA5000:

defaults { udev_dir /dev polling_interval 10 selector "round-robin 0" path_grouping_policy failover getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout "/bin/true" path_checker tur rr_min_io 100 rr_weight uniform failback immediate no_path_retry 12 user_friendly_names yes bindings_file "/var/lib/multipath/bindings" }

Finally, you will need to specify configuration for the presented LUN’s. This applies to the multipaths section of multipath.conf file:

multipath { wwid 3600508b4001031250000900001490000 alias san1data } multipath { wwid 3600508b400011c300000f00001a90000 alias san2data }

After you are done, restart multipathd and check output of multipath -ll command:

[root@carbon ~]# multipath -ll san2data (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][active] \_ 1:0:3:1 sde 8:64 [active][ready] \_ 2:0:3:1 sdh 8:112 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:2:1 sdd 8:48 [active][ready] \_ 2:0:2:1 sdg 8:96 [active][ready] san1data (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=50][enabled] \_ 1:0:0:1 sdb 8:16 [active][ready] \_ 2:0:4:1 sdi 8:128 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:1:1 sdc 8:32 [active][ready] \_ 2:0:1:1 sdf 8:80 [active][ready]

That should be it. You should test the setup by disabling paths to see if your LUN’s stay up.

Growing mirrored LUN in RedHat

somedude — Fri, 06 Feb 2009 22:52:36 +0000

I was putting a RedHat server onto a SAN and I could not find any clear instructions on how to grow a single mirrored LUN on the fly. Anyway, here are some notes on the process. First the setup: Two LUN’s mirrored across two SAN’s with LVM volume on the top of it. I could have easily just presented another set of mirrored LUN’s, add them to VG and go from there. I wanted to avoid that, as that kind of setup can quickly get out of hand as the number of presented LUN’s grows. If there is a more “sensible” and flexible setup, I would most definitely want to know about it.

For sake of completeness, here are steps to recreate the initial setup I had:

Create a mirror from two LUN’s
Use the mirror as PV
Create a VG using the PV
Create LV on the top of the VG
Make ext3 filesystem on the top of LV and mount it

Here are the actual steps with some output:

[root@ultra /]# mdadm --create /dev/md10 --level=1 --raid-devices=2 /dev/mapper/mpath4 /dev/mapper/mpath5 mdadm: array /dev/md10 started. [root@ultra/]# pvcreate /dev/md10 Physical volume "/dev/md10" successfully created [root@ultra/]# vgcreate testvg /dev/md10 Volume group "testvg" successfully created [root@ultra/]# lvcreate -l+100%FREE -n testlv testvg Logical volume "testlv" created [root@ultra/]# mkfs -t ext3 /dev/testvg/testlv [root@ultra /]# mount /dev/testvg/testlv /tmp/test

Now the resizing part. There might be a few steps but the upshot is that the filesystem can stay mounted and in use. High level overview of steps to take:

Grow the two LUN’s using SAN management software
Fail and remove one of the submirrors
Force the kernel to see the size increase of the submirror
Flush and recreate the multipath device map so multipathing sees the new size
Re-add the submirror to the mirror and let it sync
Repeat 2-4 for the second submirror
Resize the PV
Resize the LV
Resize the filesystem

First, you fail and remove the submirror:

[root@ultra/]# mdadm /dev/md10 -f /dev/mapper/mpath4 -r /dev/mapper/mpath4 mdadm: set /dev/mapper/mpath4 faulty in /dev/md10 mdadm: hot removed /dev/mapper/mpath4
Now, note all paths to the LUN. Kernel sees a separate device at the end of each path to a LUN. In this case they are sdj, sdt, sdg and sdq.

[root@ultra/]# multipath -ll mpath4 mpath4 (3600508b400011c300000f000008d0000) [size=12 GB][features="1 queue_if_no_path"][hwhandler="0"] _ round-robin 0 [prio=100][active] ._ 1:0:3:1 sdj 8:144 [active][ready] ._ 2:0:3:1 sdt 65:48 [active][ready] _ round-robin 0 [prio=20][enabled] ._ 1:0:2:1 sdg 8:96 [active][ready] ._ 2:0:2:1 sdq 65:0 [active][ready]

At this point the problem is to get the kernel to recognize the new size without reboot. After a lot of trying and sifting through man pages I found that blockdev command does the magic. Then I googled “blockdev resize” and I found this confirming my finding. So, the next step is to probe all logical paths to the LUN:

[root@ultra/]# blockdev --rereadpt /dev/sdj [root@ultra/]# blockdev --rereadpt /dev/sdt [root@ultra/]# blockdev --rereadpt /dev/sdg [root@ultra /]# blockdev --rereadpt /dev/sdq

You should see messages in /var/log/messages about kernel seeing new size on all paths. If you were to issue multipath -ll right now you would see that multipathing is still reporting old size. To fix that, flush the device map of the LUN and then recreate it:

[root@ultra/]# multipath -f mpath4 [root@ultra/]# multipath -v2 create: mpath4 (3600508b400011c300000f000008d0000) [size=13 GB][features="0"][hwhandler="0"] _ round-robin 0 [prio=100] ._ 1:0:3:1 sdj 8:144 [ready] ._ 2:0:3:1 sdt 65:48 [ready] _ round-robin 0 [prio=20] ._ 1:0:2:1 sdg 8:96 [ready] ._ 2:0:2:1 sdq 65:0 [ready]

Multipathing should be reporting the new size. Now you are ready to put back the grown submirror and let the whole mirror sync:

[root@ultra/]# mdadm /dev/md10 -a /dev/mapper/mpath4 mdadm: hot added /dev/mapper/mpath4

When the mirror has synced up, repeat the above process for the second submirror and wait for the sync to finish. Time to grow the mirror device itself:

[root@ultra /]# mdadm --grow /dev/md10 --size=max

After the completion /proc/mdstat should report increase in size of /dev/md10. Moving on you need to grow the PV that resides on /dev/md10:

[root@ultra/]# pvresize /dev/md10 Physical volume "/dev/md10" changed 1 physical volume(s) resized / 0 physical volume(s) not resized

And finally, you need to resize the LV:

[root@ultra/]# lvresize -l+100%FREE testvg/testlv Extending logical volume testlv to 13.00 GB Logical volume testlv successfully resized

Of course, don’t forget to grow the filesystem itself:

[root@ultra/]# ext2online /dev/testvg/testlv ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b [root@ultra/]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-rootlv . 132304280 5104976 120478588 5% / /dev/md0 132134 32791 92521 27% /boot none 8202920 0 8202920 0% /dev/shm /dev/mapper/testvg-testlv . 13413488 63516 12668820 1% /tmp/test

That should be it. The sync time for huge volumes is going to be something to keep in mind. The whole setup is clean and neat without clutter. I could have opted to mirror using LVM, but there seems to be a strange requirement for third, log volume. It is possible to keep the log in memory, but that supposedly causes resync on boot.

Quick and dirty ZFS cheatsheet

somedude — Sat, 20 Dec 2008 15:53:38 +0000

Create simple striped pool:
zpool create [pool_name] [device] [device] ... zpool create datapool c5t433127A900011C370000C00003210000d0 c5t433127B4001031250000900000540000d0

Create mirrored pool:
zpool create [pool_name] mirror [device] [device] ... zpool create datapool mirror c5t433127A900011C370000C00003210000d0 c5t433127B4001031250000900000540000d0

Create Raid-Z pool:
zpool create [pool_name] raidz [device] [device] [device] ... zpool create datapool raidz c5t433127A900011C370000C00003210000d0 c5t433127B4001031250000900000540000d0 c5t439257C4000019250000900000540000d0

Transform simple pool to a mirror:
zpool create [pool_name] [device] zpool attach [pool_name] [existing_device] [new_device] zpool create datapool c5t433127A900011C370000C00003210000d0 zpool attach datapool c5t433127A900011C370000C00003210000d0 c5t433127B4001031250000900000540000d0

Expand simple pool:
zpool create [pool_name] [device] zpool add [pool_name] [new_device] zpool create datapool c5t433127A900011C370000C00003210000d0 zpool add datapool c5t433127B4001031250000900000540000d0

Expand mirrored pool by attaching additional mirror:
zpool add [pool_name] mirror [new_device] [new_device] zpool add datapool mirror c5t433127A900011C370000C00003460000d0 c5t433127B400011C370000C00003410000d0

Replace device in a pool:
zpool replace [pool_name] [old_device] [new_device] zpool replace datapool c5t433127A900011C370000C00003410000d0 c5t433127B4001031250000900000540000d0

Destroy pool:
zpool destroy [pool_name] zpool destroy datapool

Set pool mountpoint:
zfs set mountpoint=/path [pool_name] zfs set mountpoint=/export/zfs datapool

Display configured pools:
zpool list zpool list

Display pool status info:
zpool status [-v] [pool_name] zpool status -v datapool

Display pool I/O statistics:
zpool iostat [pool_name] zpool iostat datapool

Display pool command history:
zpool history [pool_name] zpool history datapool

Export a pool:
zpool export [pool_name] zpool export datapool

Import a pool:
zpool import [pool_name] zpool import datapool

Create a filesystem:
zfs create [pool_name]/[fs_name] zfs create datapool/filesystem

Destroy a filesystem:
zfs destroy [pool_name]/[fs_name] zfs destroy datapool/filesystem

Rename a filesystem:
zfs rename [pool_name]/[fs_name] [pool_name]/[fs_name] zfs rename datapool/filesystem datapool/newfilesystem

Move a filesystem:
zfs rename [pool_name]/[fs_name] [pool_name]/[fs_name]/[fs_name] zfs rename datapool/filesystem datapool/users/filesystem

Display properties of a filesystem:
zfs get all [pool_name]/[fs_name] zfs get all datapool/filesystem

Make a snapshot:
zfs snapshot [pool_name]/[fs_name]@[time] zfs snapshot datapool/filesystem@friday

Roll back filesystem to its snapshot:
zfs rollback [pool_name]/[fs_name]@[time] zfs rollback datapool/filesystem@friday

Clone a filesystem:
zfs snapshot [pool_name]/[fs_name]@[time] zfs clone [pool_name]/[fs_name]@[time] [pool_name]/[fs_name] zfs snapshot datapool/filesystem@today zfs clone datapool/filesystem@today datapool/filesystemclone

Backup filesystem to a file:
zfs send [pool_name]/[fs_name] > /path/to/file zfs send datapool/filesystem@friday > /tmp/filesystem.bkp

Restore filesystem from a file:
zfs receive [pool_name]/[fs_name] < /path/to/file zfs receive datapool/restoredfilesystem < /tmp/filesystem.bkp

Create ZFS volume:
zfs create -V [size] [pool_name]/[vol_name] zfs create -V 100mb datapool/zvolume newfs /dev/zvol/dsk/datapool/zvolume

Growing Solaris Volume Manager mirrors online

somedude — Sat, 08 Nov 2008 14:41:10 +0000

You have a SVM mirror and you need to grow it. Online. All steps are straight forward except for the last one. So, in a nutshell:

Grow sizes of both submirror LUN’s on SAN
Detach the first submirror metadevice
Clear the submirror metadevice
Make the system see new LUN size
Recreate the submirror metadevice, attach it to the mirror and let it sync
Repeat the above for the second submirror
Not so obvious step
Grow the filesystem

You have a mirror metadevice d200 consisting of submirrors d201 and d202. Both submirrors are sliced in such way that all space is allocated in slice 0. At this point both LUN’s should be already grown, but you need to make the system to see the size increase. Here is what d201, one of the submirrors of d200, should look like:

bash-3.00# metastat d201 d201: Submirror of d200 State: Okay Size: 2064384 blocks (1008 MB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

First, detach d201 from its parent and clear it:

bash-3.00# metadetach d200 d201 bash-3.00# metaclear d201

The next step is likely to alter the LUN slice configuration, so make sure you have it noted.

You need to run format command to get the system to see the grown LUN. So, run format then select type and then select Autoconfigure. At this point format should be reporting the new LUN size. Make sure your slice configuration is correct and that all the new space is tacked onto the end of correct slice. In this case, it’s slice 0.

Now recreate d201, attach it to d200 and let the mirror sync. When checking status of d200 note that overall mirror size is still the same but the size of d201 increased:

bash-3.00# metastat d200 d200: Mirror Submirror 0: d202 State: Okay Submirror 1: d201 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 2064384 blocks (1008 MB)

d202: Submirror of d200 State: Okay Size: 2064384 blocks (1008 MB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003460000d0s0 0 No Okay Yes

d201: Submirror of d200 State: Okay Size: 4128768 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

Now repeat the same process with d202. After you are done you will have both submirrors bigger, but the size of overall mirror will still be the same:

bash-3.00# metastat d200 d200: Mirror Submirror 0: d201 State: Okay Submirror 1: d202 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 2064384 blocks (1008 MB)

d201: Submirror of d200 State: Okay Size: 4128768 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

d202: Submirror of d200 State: Okay Size: 4161536 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003460000d0s0 0 No Okay Yes

Now comes the “Not so obvious” step. In order to increase overall mirror size you need to run metattach on the mirror itself:

bash-3.00# metattach d200 bash-3.00# metastat d200 d200: Mirror Submirror 0: d201 State: Okay Submirror 1: d202 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 4128768 blocks (2.0 GB)

This will grow the size of the entire mirror to the size of submirror devices. Now you can grow the filesystem on it without unmounting it. Keep in mind that for the duration of growfs command there will be no writes to the volume.

bash-3.00# growfs -M /export/home/users /dev/md/rdsk/d200 /dev/md/rdsk/d200: 4128768 sectors in 126 cylinders of 128 tracks, 256 sectors 2016.0MB in 63 cyl groups (2 c/g, 32.00MB/g, 15040 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 65824, 131616, 197408, 263200, 328992, 394784, 460576, 526368, 592160, 3487008, 3552800, 3618592, 3684384, 3750176, 3815968, 3881760, 3947552, 4013344, 4079136

That’s all there is to it. The “Not so obvious” step is mentioned in the metattach man page. But unless you do this stuff every day, this very useful piece of information can be easily forgotten.

Disabling MPxIO for onboard disks

somedude — Mon, 01 Sep 2008 18:36:21 +0000

I was configuring MPxIO on Sun Fire V490. After issuing stmsboot -e command and rebooting the server, internal disks were under MPxIO’s control.

This was not something I wanted and I needed to disable it. In order to do so I had to edit /kernel/drv/fp.conf file and tell Solaris not to enable MPxIO on internal disks. So I added the following line at the end of fp.conf file:

name="fp" parent="/pci@8,600000/SUNW,qlc@4" port=0 mpxio-disable="yes";

This line tells Solaris to disable MPxIO on port 0 for all devices whose parent device is /pci@8,600000/SUNW,qlc@4. Of course, similar line should be added for all HBA’s and ports you do not want to have under MPxIO’s control. You can get the parent device from your /var/adm/messages file or from device links pointing to the internal disks. Here is a partial list of device links:

<-----------------SNIP---------------->

lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s0 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:a lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s1 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:b lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s2 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:c lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s3 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:d lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s4 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:e lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s5 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:f lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s6 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:g lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s7 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:h

<-----------------SNIP---------------->

Gathering fibre channel info using fcinfo

somedude — Tue, 19 Aug 2008 19:47:25 +0000

I came across fcinfo command reading some man page. So, I decided to see what kind of useful fibre channel information it could give me. It turns out fcinfo knows quite a bit.

We can list all local fibre channel ports:

bash-3.00# fcinfo hba-port HBA Port WWN: 21000003ba16dbd2 OS Device Name: /dev/cfg/c1 Manufacturer: QLogic Corp. Model: 2200 Firmware Version: 2.1.144 FCode/BIOS Version: ISP2200 FC-AL Host Adapter Driver: 1.12 01/01/16 Type: L-port State: online Supported Speeds: 1Gb Current Speed: 1Gb Node WWN: 20000003ba16dbd2 HBA Port WWN: 2100001b320e3853 OS Device Name: /dev/cfg/c3 Manufacturer: QLogic Corp. Model: QLA2462 Firmware Version: 4.0.27 FCode/BIOS Version: QLA2462 Host Adapter Driver(SPARC): 1.24 11/15/06 Type: N-port State: online Supported Speeds: 1Gb 2Gb 4Gb Current Speed: 2Gb Node WWN: 2000001b320e3853 HBA Port WWN: 2101001b322e3853 OS Device Name: /dev/cfg/c4 Manufacturer: QLogic Corp. Model: QLA2462 Firmware Version: 4.0.27 FCode/BIOS Version: QLA2462 Host Adapter Driver(SPARC): 1.24 11/15/06 Type: N-port State: online Supported Speeds: 1Gb 2Gb 4Gb Current Speed: 2Gb Node WWN: 2001001b322e3853

We can take a look what remote ports are seen by particular local fibre channel port, in this case 2100001b320e3853:

bash-3.00# fcinfo remote-port -p 2100001b320e3853 Remote Port WWN: 100000e00216aef3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e01124b88f Remote Port WWN: 50001fe15003b384 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Remote Port WWN: 100000e0022744f3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e0020744f3 Remote Port WWN: 50001fe150216de9 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53 Remote Port WWN: 100000e00228f492 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e00208f492 Remote Port WWN: 50001fe15076b59b Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Remote Port WWN: 50001fe150216ded Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53

Here, we get link statistics for remote fibre channel device whose WWN is 100000e0020744f3:

bash-3.00# fcinfo remote-port -l -p 2100001b320e3853 100000e0020744f3 Remote Port WWN: 100000e0022744f3 Active FC4 Types: SCSI SCSI Target: unknown Node WWN: 100000e0020744f3 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0

We can also get link statistics and SCSI target information for all remote fibre channel devices see on local port whose WWN is 2100001b320e3853:

bash-3.00# fcinfo remote-port -sl -p 2100001b320e3853 Remote Port WWN: 100000e00216aef3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e01124b88f Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: HP Product: MSL6000 Series OS Device Name: /devices/pci@8,600000/SUNW,qlc@1/fp@0,0/sgen@w100000e00216aef3,0 LUN: 1 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/2n Remote Port WWN: 50001fe15003b384 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B4001031250000900000540000d0s2 Remote Port WWN: 100000e0022744f3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e0020744f3 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: HP Product: MSL6000 Series OS Device Name: /devices/pci@8,600000/SUNW,qlc@1/fp@0,0/sgen@w100000e0022744f3,0 LUN: 1 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/1n LUN: 2 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/0n Remote Port WWN: 50001fe150216de9 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B400011C370000C00003210000d0s2 Remote Port WWN: 100000e00228f492 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e00208f492 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/4n LUN: 1 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/3n Remote Port WWN: 50001fe15076b59b Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B4001031250000900000540000d0s2 Remote Port WWN: 50001fe150216ded Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B400011C370000C00003210000d0s2

This command is quite handy when troubleshooting fibre channel. Very cool…

Mirroring root disk using SVM

somedude — Wed, 02 Jul 2008 19:39:37 +0000

There are about 487359 documents on the Internet about how to mirror root disk in Solaris. So, here is 487360th.

The assumptions are following: the first disk has Solaris already installed, root slice is slice 1, and the disks are identical with the same size and geometry. If they have different cylinder, head, sector count or different size you will have to fiddle with sizing slices more.

The first step is to recreate the same slice arrangement on the second disk:

bash-3.00# prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2 fmthard: New volume table of contents now in place.

You can check both disks have the same VTOC using prtvtoc command

bash-3.00# prtvtoc /dev/rdsk/c1t0d0s2 * /dev/rdsk/c1t0d0s2 partition map * * Dimensions: * 512 bytes/sector * 424 sectors/track * 24 tracks/cylinder * 10176 sectors/cylinder * 14089 cylinders * 14087 accessible cylinders * * Flags: * 1: unmountable * 10: read-only * * First Sector Last * Partition Tag Flags Sector Count Sector Mount Directory 0 3 01 0 4100928 4100927 1 2 00 4100928 20484288 24585215 2 5 00 0 143349312 143349311 3 8 00 24585216 118672512 143257727 7 0 00 143257728 91584 143349311

Now we have to create state database replicas on slice 7. We will be adding two replicas to each slice:

bash-3.00# metadb -a -f -c3 /dev/dsk/c1t0d0s7 bash-3.00# metadb -a -f -c3 /dev/dsk/c1t1d0s7

Database replicas are crucial part of SVM. Here is some important information about how they work, how many you need, etc.

Since the database replicas are in place we can start creating metadevices. The following commands will create metadevice d31 from slice c1t0d0s3, and metadevice d32 from slice c1t1d0s3. Then we create mirror d30 with d31 attached as a submirror. Finally we will attach submirror d32 to mirror d30. Once d32 is attached, the mirror d30 will automatically start syncing.

bash-3.00# metainit -f d31 1 1 c1t0d0s3 d31: Concat/Stripe is setup bash-3.00# metainit -f d32 1 1 c1t1d0s3 d32: Concat/Stripe is setup bash-3.00# metainit d30 -m d31 d30: Mirror is setup bash-3.00# metattach d30 d32 d30: submirror d32 is attached

The procedure is the same for all other mirrors you might want to create. Root filesystem is slightly different. First you will have to create your submirrors. Then you will have to attach submirror with existing root filesystem, in this case d11, to the new mirror metadevice d10. Then you will have to run metaroot command. It will alter / entry in /etc/vfstab. Finally, you flush the filesystem using lockfs command and reboot.

bash-3.00# metainit -f d11 1 1 c1t0d0s1 d31: Concat/Stripe is setup bash-3.00# metainit -f d12 1 1 c1t1d0s1 d32: Concat/Stripe is setup bash-3.00# metainit d10 -m d11 d30: Mirror is setup bash-3.00# metaroot d10 bash-3.00# lockfs -fa bash-3.00# reboot

When the system reboots, you can attach the second submirror to d10 as follows:

bash-3.00# metattach d10 d12

You can check the sync progress using metastat command. Once all mirrors are synced up the next step is to configure the new swap metadevice, in my case d0, to be crash dump device. This is done using dumpadm command:

bash-3.00# dumpadm Dump content: kernel pages Dump device: /dev/dsk/c1t0d0s0 (dedicated) Savecore directory: /var/crash/ultra Savecore enabled: yes bash-3.00# dumpadm -d /dev/md/dsk/d0

The final step is to modify PROM. First we need to find out which two physical devices c1t0d0 and c1t1d0 refer to:

bash-3.00# ls -l /dev/dsk/c1t0d0s1 lrwxrwxrwx 1 root root 43 Mar 4 14:38 /dev/dsk/c1t0d0s1 -> ../../devices/pci@1c,600000/scsi@2/sd@0,0:b bash-3.00# ls -l /dev/dsk/c1t1d0s1 lrwxrwxrwx 1 root root 43 Mar 4 14:38 /dev/dsk/c1t1d0s1 -> ../../devices/pci@1c,600000/scsi@2/sd@1,0:b

The physical device path is everything starting from /pci…. Please make a note of sd towards the end of the device string. When creating device aliases below, sd will have to be changed to disk.

Now we create two device aliases called root and backup_root. Then we set boot-device to be root and backup_root. The :b refers to slice 1(root) on that particular disk.

bash-3.00# eeprom "use-nvramrc?=true" bash-3.00# eeprom "nvramrc=devalias root /pci@1c,600000/scsi@2/disk@0,0 devalias backup_root /pci@1c,600000/scsi@2/disk@1,0" bash-3.00# eeprom "boot-device=root:b backup_root:b net"

Now we can test that the system boot from both root and backup_root devices.

There is one more optional step. Adding two kernel tunables to /etc/system file. The first one is md_mirror:md_resync_bufsz which will speed up mirror resync. The second one is md:mirrored_root_flag. When this flag is enabled the system will boot even if less than majority of database replicas is available. Personally, I do not use the second tunable. More on these can be found in Solaris Tunable Parameters Reference Guide.

Moving boot disk to a different target or controller

somedude — Wed, 18 Jun 2008 18:11:07 +0000

I had a need for a system that can boot two different versions of Solaris from two different disks. Both disks were on target 0 when Solaris was installed on them. But when I moved one of the disks to be target 1, obviously there was going to be a problem with booting from that disk.

There is a simple way to get the disk booting from the new target. Here are the steps to take:

Boot the system off a DVD or CD or jumpstart server in single user mode
Mount the root filesystem on the disk in question and edit vfstab to reflect the new controller/target setup

The last step is to regenerate /etc/path_to_inst and device links in /dev. Searching the internet for some unrelated info I found out devfsadm has an undocumented -p switch that recreates path_to_inst file. The -r switch specifies location of root filesystem.

bash-3.00# devfsadm -r /mnt -p /mnt/etc/path_to_inst

Now you can reboot and the drive should be bootable again.

Moving LUN's between hosts using metarecover

somedude — Wed, 02 Apr 2008 07:11:32 +0000

Sometimes you might need to move a LUN between hosts X and Y on a SAN. Solaris has cool command, metarecover, that lets you do just that. First, you will need to stop all processes on X that might be using the given LUN. Then unmount the LUN and present it to the host Y. Now you can use metarecover to recover metadevice information from the LUN. Couple of things to remember:

There is -n switch that will cause dry run, i.e. metarecover will output what it would do – it will not actually perform recovery operation. Also, be careful about metadevice names on host Y and the ones you are trying to recover. I have not tried it so I am not sure what would happen. It would probably bomb out. You could use metarename to rename Y’s existing metadevices and try to eliminate possible conflict before recovery.

root@ultra# metarecover -v c3t600508B400011C370000C00002970000d0s2 -p -d Verifying on-disk structures on c3t600508B400011C370000C00002970000d0s2. The following extent headers were found on c3t600508B400011C370000C00002970000d0s2. Name Seq# Type Offset Length d100 0 ALLOC 205635593 25165825 d110 0 ALLOC 16384 32769 d120 0 ALLOC 49153 32769 d130 0 ALLOC 81922 32769 d140 0 ALLOC 114691 41943041 d150 0 ALLOC 42057732 146800641 d160 0 ALLOC 188858373 2097153 d170 0 ALLOC 190955526 6291457 d180 0 ALLOC 197246983 6291457 d190 0 ALLOC 203538440 2097153 NONE 0 END 251609087 1 NONE 0 FREE 230801418 20807669 Found 10 soft partition(s) on c3t600508B400011C370000C00002970000d0s2. Checking sequence numbers. mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 100 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 25165824 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 205635594 25165824 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 110 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 32768 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 16385 32768 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 120 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 32768 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 49154 32768 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 130 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 32768 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 81923 32768 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 140 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 41943040 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 114692 41943040 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 150 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 146800640 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 42057733 146800640 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 160 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 2097152 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 188858374 2097152 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 170 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 6291456 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 190955527 6291456 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 180 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 6291456 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 197246984 6291456 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 190 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 2097152 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 203538441 2097152 The following soft partitions were found and will be added to your metadevice configuration. Name Size No. of Extents d100 25165824 1 d110 32768 1 d120 32768 1 d130 32768 1 d140 41943040 1 d150 146800640 1 d160 2097152 1 d170 6291456 1 d180 6291456 1 d190 2097152 1 WARNING: You are about to add one or more soft partition metadevices to your metadevice configuration. If there appears to be an error in the soft partition(s) displayed above, do NOT proceed with this recovery operation. Are you sure you want to do this (yes/no)? yes c3t600508B400011C370000C00002970000d0s2: Soft Partitions recovered from device. root@ultra#