unixpowered.com » solaris volume manager

Quick and dirty SVM cheatsheet

somedude — Thu, 03 Dec 2009 12:50:43 +0000

This list focuses mostly on mirror operations. I use Solaris Volume Manager quite a bit when mirroring internal drives. There are tons of additional features and commands, if you use SVM for things other than mirroring. In that case you might want to look at check out Solaris Volume Manager Administration Guide.

Create database replicas:
metadb -f -a -c [number_of_replicas] [device] metadb -f -a -c 3 c0t0d0s7

Delete all database replicas from device:
metadb -d [device] metadb -d c0t0d0s7

Display status of database replicas:
metadb -i metadb -i

Display metadevice status:
metastat metastat

Create simple concat/stripe metadevice:
metainit -f [concat_metadevice] 1 1 [device] metainit -f d21 1 1 c0t0d0s1

Create a mirror with one submirror:
metainit [mirror_metadevice] -m [submirror_metadevice] metainit d20 -m d21

Attach a submirror to one sided mirror:
metattach [mirror_metadevice] [submirror_metadevice] metattach d20 d22

Detach a submirror from a mirror:
metadetach [mirror_metadevice] [submirror_metadevice] metadetach d20 d22

Clear a metadevice:
metaclear [metadevice] metaclear d22

Offline a submirror:
metaoffline [mirror_metadevice] [submirror_metadevice] metaoffline d20 d22

Online a submirror:
metaonline [mirror_metadevice] [submirror_metadevice] metaonline d20 d22

Enable a failed component:
metareplace -e [metadevice] [device] metareplace -e d21 c0t0d0s1

Rename a metadevice:
metarename [old_metadevice] [new_metadevice] metarename d20 d30

Switch metadevice names:
metarename [metadevice_1] [metadevice_2] metarename -x d20 d30

Configure system for root metadevice:
metaroot [metadevice] metaroot d10

Growing Solaris Volume Manager mirrors online

somedude — Sat, 08 Nov 2008 14:41:10 +0000

You have a SVM mirror and you need to grow it. Online. All steps are straight forward except for the last one. So, in a nutshell:

Grow sizes of both submirror LUN’s on SAN
Detach the first submirror metadevice
Clear the submirror metadevice
Make the system see new LUN size
Recreate the submirror metadevice, attach it to the mirror and let it sync
Repeat the above for the second submirror
Not so obvious step
Grow the filesystem

You have a mirror metadevice d200 consisting of submirrors d201 and d202. Both submirrors are sliced in such way that all space is allocated in slice 0. At this point both LUN’s should be already grown, but you need to make the system to see the size increase. Here is what d201, one of the submirrors of d200, should look like:

bash-3.00# metastat d201 d201: Submirror of d200 State: Okay Size: 2064384 blocks (1008 MB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

First, detach d201 from its parent and clear it:

bash-3.00# metadetach d200 d201 bash-3.00# metaclear d201

The next step is likely to alter the LUN slice configuration, so make sure you have it noted.

You need to run format command to get the system to see the grown LUN. So, run format then select type and then select Autoconfigure. At this point format should be reporting the new LUN size. Make sure your slice configuration is correct and that all the new space is tacked onto the end of correct slice. In this case, it’s slice 0.

Now recreate d201, attach it to d200 and let the mirror sync. When checking status of d200 note that overall mirror size is still the same but the size of d201 increased:

bash-3.00# metastat d200 d200: Mirror Submirror 0: d202 State: Okay Submirror 1: d201 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 2064384 blocks (1008 MB)

d202: Submirror of d200 State: Okay Size: 2064384 blocks (1008 MB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003460000d0s0 0 No Okay Yes

d201: Submirror of d200 State: Okay Size: 4128768 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

Now repeat the same process with d202. After you are done you will have both submirrors bigger, but the size of overall mirror will still be the same:

bash-3.00# metastat d200 d200: Mirror Submirror 0: d201 State: Okay Submirror 1: d202 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 2064384 blocks (1008 MB)

d201: Submirror of d200 State: Okay Size: 4128768 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

d202: Submirror of d200 State: Okay Size: 4161536 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003460000d0s0 0 No Okay Yes

Now comes the “Not so obvious” step. In order to increase overall mirror size you need to run metattach on the mirror itself:

bash-3.00# metattach d200 bash-3.00# metastat d200 d200: Mirror Submirror 0: d201 State: Okay Submirror 1: d202 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 4128768 blocks (2.0 GB)

This will grow the size of the entire mirror to the size of submirror devices. Now you can grow the filesystem on it without unmounting it. Keep in mind that for the duration of growfs command there will be no writes to the volume.

bash-3.00# growfs -M /export/home/users /dev/md/rdsk/d200 /dev/md/rdsk/d200: 4128768 sectors in 126 cylinders of 128 tracks, 256 sectors 2016.0MB in 63 cyl groups (2 c/g, 32.00MB/g, 15040 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 65824, 131616, 197408, 263200, 328992, 394784, 460576, 526368, 592160, 3487008, 3552800, 3618592, 3684384, 3750176, 3815968, 3881760, 3947552, 4013344, 4079136

That’s all there is to it. The “Not so obvious” step is mentioned in the metattach man page. But unless you do this stuff every day, this very useful piece of information can be easily forgotten.

Mirroring root disk using SVM

somedude — Wed, 02 Jul 2008 19:39:37 +0000

There are about 487359 documents on the Internet about how to mirror root disk in Solaris. So, here is 487360th.

The assumptions are following: the first disk has Solaris already installed, root slice is slice 1, and the disks are identical with the same size and geometry. If they have different cylinder, head, sector count or different size you will have to fiddle with sizing slices more.

The first step is to recreate the same slice arrangement on the second disk:

bash-3.00# prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2 fmthard: New volume table of contents now in place.

You can check both disks have the same VTOC using prtvtoc command

bash-3.00# prtvtoc /dev/rdsk/c1t0d0s2 * /dev/rdsk/c1t0d0s2 partition map * * Dimensions: * 512 bytes/sector * 424 sectors/track * 24 tracks/cylinder * 10176 sectors/cylinder * 14089 cylinders * 14087 accessible cylinders * * Flags: * 1: unmountable * 10: read-only * * First Sector Last * Partition Tag Flags Sector Count Sector Mount Directory 0 3 01 0 4100928 4100927 1 2 00 4100928 20484288 24585215 2 5 00 0 143349312 143349311 3 8 00 24585216 118672512 143257727 7 0 00 143257728 91584 143349311

Now we have to create state database replicas on slice 7. We will be adding two replicas to each slice:

bash-3.00# metadb -a -f -c3 /dev/dsk/c1t0d0s7 bash-3.00# metadb -a -f -c3 /dev/dsk/c1t1d0s7

Database replicas are crucial part of SVM. Here is some important information about how they work, how many you need, etc.

Since the database replicas are in place we can start creating metadevices. The following commands will create metadevice d31 from slice c1t0d0s3, and metadevice d32 from slice c1t1d0s3. Then we create mirror d30 with d31 attached as a submirror. Finally we will attach submirror d32 to mirror d30. Once d32 is attached, the mirror d30 will automatically start syncing.

bash-3.00# metainit -f d31 1 1 c1t0d0s3 d31: Concat/Stripe is setup bash-3.00# metainit -f d32 1 1 c1t1d0s3 d32: Concat/Stripe is setup bash-3.00# metainit d30 -m d31 d30: Mirror is setup bash-3.00# metattach d30 d32 d30: submirror d32 is attached

The procedure is the same for all other mirrors you might want to create. Root filesystem is slightly different. First you will have to create your submirrors. Then you will have to attach submirror with existing root filesystem, in this case d11, to the new mirror metadevice d10. Then you will have to run metaroot command. It will alter / entry in /etc/vfstab. Finally, you flush the filesystem using lockfs command and reboot.

bash-3.00# metainit -f d11 1 1 c1t0d0s1 d31: Concat/Stripe is setup bash-3.00# metainit -f d12 1 1 c1t1d0s1 d32: Concat/Stripe is setup bash-3.00# metainit d10 -m d11 d30: Mirror is setup bash-3.00# metaroot d10 bash-3.00# lockfs -fa bash-3.00# reboot

When the system reboots, you can attach the second submirror to d10 as follows:

bash-3.00# metattach d10 d12

You can check the sync progress using metastat command. Once all mirrors are synced up the next step is to configure the new swap metadevice, in my case d0, to be crash dump device. This is done using dumpadm command:

bash-3.00# dumpadm Dump content: kernel pages Dump device: /dev/dsk/c1t0d0s0 (dedicated) Savecore directory: /var/crash/ultra Savecore enabled: yes bash-3.00# dumpadm -d /dev/md/dsk/d0

The final step is to modify PROM. First we need to find out which two physical devices c1t0d0 and c1t1d0 refer to:

bash-3.00# ls -l /dev/dsk/c1t0d0s1 lrwxrwxrwx 1 root root 43 Mar 4 14:38 /dev/dsk/c1t0d0s1 -> ../../devices/pci@1c,600000/scsi@2/sd@0,0:b bash-3.00# ls -l /dev/dsk/c1t1d0s1 lrwxrwxrwx 1 root root 43 Mar 4 14:38 /dev/dsk/c1t1d0s1 -> ../../devices/pci@1c,600000/scsi@2/sd@1,0:b

The physical device path is everything starting from /pci…. Please make a note of sd towards the end of the device string. When creating device aliases below, sd will have to be changed to disk.

Now we create two device aliases called root and backup_root. Then we set boot-device to be root and backup_root. The :b refers to slice 1(root) on that particular disk.

bash-3.00# eeprom "use-nvramrc?=true" bash-3.00# eeprom "nvramrc=devalias root /pci@1c,600000/scsi@2/disk@0,0 devalias backup_root /pci@1c,600000/scsi@2/disk@1,0" bash-3.00# eeprom "boot-device=root:b backup_root:b net"

Now we can test that the system boot from both root and backup_root devices.

There is one more optional step. Adding two kernel tunables to /etc/system file. The first one is md_mirror:md_resync_bufsz which will speed up mirror resync. The second one is md:mirrored_root_flag. When this flag is enabled the system will boot even if less than majority of database replicas is available. Personally, I do not use the second tunable. More on these can be found in Solaris Tunable Parameters Reference Guide.

Moving LUN's between hosts using metarecover

somedude — Wed, 02 Apr 2008 07:11:32 +0000

Sometimes you might need to move a LUN between hosts X and Y on a SAN. Solaris has cool command, metarecover, that lets you do just that. First, you will need to stop all processes on X that might be using the given LUN. Then unmount the LUN and present it to the host Y. Now you can use metarecover to recover metadevice information from the LUN. Couple of things to remember:

There is -n switch that will cause dry run, i.e. metarecover will output what it would do – it will not actually perform recovery operation. Also, be careful about metadevice names on host Y and the ones you are trying to recover. I have not tried it so I am not sure what would happen. It would probably bomb out. You could use metarename to rename Y’s existing metadevices and try to eliminate possible conflict before recovery.

root@ultra# metarecover -v c3t600508B400011C370000C00002970000d0s2 -p -d Verifying on-disk structures on c3t600508B400011C370000C00002970000d0s2. The following extent headers were found on c3t600508B400011C370000C00002970000d0s2. Name Seq# Type Offset Length d100 0 ALLOC 205635593 25165825 d110 0 ALLOC 16384 32769 d120 0 ALLOC 49153 32769 d130 0 ALLOC 81922 32769 d140 0 ALLOC 114691 41943041 d150 0 ALLOC 42057732 146800641 d160 0 ALLOC 188858373 2097153 d170 0 ALLOC 190955526 6291457 d180 0 ALLOC 197246983 6291457 d190 0 ALLOC 203538440 2097153 NONE 0 END 251609087 1 NONE 0 FREE 230801418 20807669 Found 10 soft partition(s) on c3t600508B400011C370000C00002970000d0s2. Checking sequence numbers. mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 100 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 25165824 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 205635594 25165824 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 110 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 32768 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 16385 32768 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 120 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 32768 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 49154 32768 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 130 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 32768 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 81923 32768 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 140 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 41943040 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 114692 41943040 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 150 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 146800640 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 42057733 146800640 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 160 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 2097152 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 188858374 2097152 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 170 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 6291456 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 190955527 6291456 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 180 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 6291456 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 197246984 6291456 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 190 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 2097152 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 203538441 2097152 The following soft partitions were found and will be added to your metadevice configuration. Name Size No. of Extents d100 25165824 1 d110 32768 1 d120 32768 1 d130 32768 1 d140 41943040 1 d150 146800640 1 d160 2097152 1 d170 6291456 1 d180 6291456 1 d190 2097152 1 WARNING: You are about to add one or more soft partition metadevices to your metadevice configuration. If there appears to be an error in the soft partition(s) displayed above, do NOT proceed with this recovery operation. Are you sure you want to do this (yes/no)? yes c3t600508B400011C370000C00002970000d0s2: Soft Partitions recovered from device. root@ultra#

Replacing disk controlled by SVM

somedude — Tue, 11 Mar 2008 20:44:52 +0000

The following scenario assumes two mirrored disks, with two state database replicas located on slice 7 of both disks. High level steps for this are as follows:

determine failed disk
detach failed submirrors
clear failed submirror metadevices and database replicas from failed disk
unconfigure the failed disk and replace it
configure the new disk and recreate VTOC
add new database replicas
recreate the submirrors and reattach them to the respective mirrors

This is the current /etc/vfstab:

bash-3.00# more /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd - /dev/fd fd - no - /proc - /proc proc - no - /dev/md/dsk/d0 - - swap - no - /dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no logging /dev/md/dsk/d30 /dev/md/rdsk/d30 /export/home ufs 2 yes logging /devices - /devices devfs - no - ctfs - /system/contract ctfs - no - objfs - /system/object objfs - no - swap - /tmp tmpfs - yes -

From here on I will use d0 and its submirrors as an example. d0 consists of d1 and d2. d2 is on the failed disk.

d0: Mirror Submirror 0: d1 State: Okay Submirror 1: d2 State: Needs maintenance Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 4100928 blocks (2.0 GB) d1: Submirror of d0 State: Okay Size: 4100928 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c1t0d0s0 0 No Okay Yes d2: Submirror of d0 State: Needs maintenance Invoke: metareplace d0 c1t1d0s0 Size: 4100928 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c1t1d0s0 0 No Maintenance Yes

First we detach d2. The same has to be repeated for d32 and d12:

bash-3.00# metadetach -f d0 d2 d0: submirror d2 is detached

We need to clear d2. Again, the same is repeated for d32 and d12:

bash-3.00# metaclear d2 d2: Concat/Stripe is cleared

Now we delete database replicas from the failed disk. It’s also very important to make sure we have at least half of state database replicas available before we start removing them from the failed disk. Here is a Sun document that explains Majority Consensus Algorithm Solaris Volume Manager uses. You can determine number and location of the replicas using metadb -i command.

bash-3.00# metadb -d c1t1d0s7

Now we can unconfigure the failed disk using cfgadm, replace it and configure the new disk:

bash-3.00# cfgadm -al Ap_Id Type Receptacle Occupant Condition c0 scsi-bus connected configured unknown c0::dsk/c0t0d0 CD-ROM connected configured unknown c1 scsi-bus connected configured unknown c1::dsk/c1t0d0 disk connected configured unknown c1::dsk/c1t1d0 disk connected configured unknown c2 scsi-bus connected unconfigured unknown usb0/1 unknown empty unconfigured ok usb0/2 unknown empty unconfigured ok bash-3.00# cfgadm -c unconfigure c1::dsk/c1t1d bash-3.00# cfgadm -c configure c1::dsk/c1t1d0

Now we replicate VTOC from the good disk:

bash-3.00# prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2

Add database replicas to the new disk:

bash-3.00# metadb -a -c2 c1t1d0s7

Finally, we can recreate failed submirrors and attach them to their respective mirrors and let them sync up. Again, the same is applies for d32 and d12:

bash-3.00# metainit d2 1 1 c1t1d0s0 d2: Concat/Stripe is setup bash-3.00# metattach d0 d2 d0: submirror d2 is attached

Few notes: This setup contains total of 4 state database replicas. During a disk failure half of the replicas will be gone. If the server gets rebooted for whatever reason, it will not come up in multiuser mode. If you have less than half of the replicas, the system will panic. For more info on all that check out docs.sun.com.

When using cfgadm to unconfigure disk, there can be no resources using that disk. Otherwise, unconfigure will fail. Quite possibly swap metadevice is set to be dedicated dump device. To view or change dedicated dump device settings use dumpadm command.