unixpowered.com

Starting with Solaris 9 there is a very handy tool called logadm that makes management of any log files a breeze. Syslog and messages files, among others, are managed by logadm which is called from root’s crontab.
Logadm reads /etc/logadm.conf file to figure out what it needs to do. By default there are following entries in logadm.conf:

/var/log/syslog -C 8 -P 'Wed Apr 8 02:10:22 2009' -a 'kill -HUP `cat /var/run/syslog.pid`'
/var/adm/messages -C 4 -P 'Fri Apr 10 02:10:15 2009' -a 'kill -HUP `cat /var/run/syslog.pid`'
/var/cron/log -c -s 512k -t /var/cron/olog
/var/lp/logs/lpsched -C 2 -N -t '$file.$N'
/var/fm/fmd/errlog -M '/usr/sbin/fmadm -q rotate errlog && mv /var/fm/fmd/errlog.0- $nfile' -N -s 2m
smf_logs -C 8 -s 1m /var/svc/log/*.log
/var/adm/pacct -C 0 -N -a '/usr/lib/acct/accton pacct' -g adm -m 664 -o adm -p never
/var/log/pool/poold -N -a 'pkill -HUP poold; true' -s 512k

Logadm provides -w switch which will write an entry into logadm.conf file that reflects current command line arguments. Of course logadm.conf can be edited using text editor, if that is the preferred method. If that’s the case, -V option can validate syntax of logadm.conf for you. Another handy option is -n which will cause logadm to do a dry run without actually performing the log rotation.

Other useful switches are:

• -b and -a which allow specification of pre and post rotation commands to execute
• -e sends error messages to a specific address instead of sending it to the owner of the crontab
• -r removes entry from logadm.conf for a specific log file
• -o sets different owner for the new log file from the original
• -g sets different group for the new log file from the original
• -m sets different permissions for the new log file from the original

For the whole story on logadm check out logadm man page.

Continue Reading

Link aggregation takes a bunch of network interfaces and creates a big pipe out of them.

Aggregation also provides redundancy. If all interfaces but one go down, the server will remain connected to the network.

Before starting make sure that:

• interfaces to be aggregated are of the following type: xge, e1000g, and bge
• interfaces to be aggregated are not plumbed
• they run in full duplex mode at the same speeds
• eeprom’s local-mac-address? variable is set to true

The following will create aggr1 interface with bge1 as one of its members:

bash-3.00# dladm create-aggr -d bge1 1

Next plumb the aggregate interface, configure an IP address on it and bring it up:

bash-3.00# ifconfig aggr1 plumb 192.168.1.5 netmask 255.255.255.0 up

At this point you can list aggregations:

bash-3.00# dladm show-aggr
key: 1 (0x0001) policy: L4      address: 0:3:ba:56:7f:ba (auto)
           device       address                 speed           duplex  link    state
           bge1         0:3:ba:56:7f:ba   0     Mbps    unknown down    standby

Now add bge0 as a second member of aggr1 aggregation interface and list aggregate interfaces:

bash-3.00# dladm add-aggr -d bge0 1
bash-3.00# dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:3:ba:56:7f:ba (auto)
           device       address                 speed           duplex  link    state
           bge1         0:3:ba:56:7f:ba   0     Mbps    unknown down    standby
           bge0         0:3:ba:56:7f:b9   1000  Mbps    full    up      attached

To keep the configuration persistent across reboots, create /etc/hostname.aggr1 with appropriate content and remove any hostname.* files pertaining to the interfaces that are now members of aggr1.

To have link aggregation working properly you need to have the switch to which server is connected to properly configured with LACP.

Another thing to consider is load balancing policy for outgoing traffic. You can load balance on layers 2,3 and 4. Load balancing policy can be changed using dladm command. Here is a quick example that will modify load balancing policy to combination of L3 and L4:

bash-3.00# dladm modify-aggr -P L3,L4 1
bash-3.00# dladm show-aggr -L
key: 1 (0x0001) policy: L3,L4 address: 0:3:ba:56:7f:ba (auto)
                LACP mode: off  LACP timer: short
    device    activity timeout aggregatable sync  coll dist defaulted expired
    bge1      passive  short   yes          no    no   no   no        no
    bge0      passive  short   yes          no    no   no   no        no

And finally, command that will allow you to see utilisation of individual links within aggregation. Note the %ipkts column, I did not have LACP turned on on the switch at that time:

bash-3.00# dladm show-aggr -s
key: 1  ipackets  rbytes      opackets   obytes          %ipkts %opkts
           Total        2723785   2287233197  1481682   710633551
           bge1 618712    115674760   870443    636559150       22.7    58.7
           bge0 2105073   2171558437  611239    74074401        77.3    41.3

More info on link aggregation is here. Continue Reading

Finally, I got tired of remembering which network interfaces is configured on my Netra test box. So I do not have to remember which interface to plug cable into I configured IPMP on the box. IPMP provides link redundancy among multiple network interfaces in multipathing group. IPMP is not meant to be full fledged load balancing solution, though it will spread outgoing traffic across the interfaces.

I have put my two hme interfaces into a multipathing group. The group has a failover IP address assigned to it. Initially this address will be assigned to hme0. If hme0 fails, the address will automatically move to the other interface in the failover group.

First I edited /etc/hostname.hme0:

unreal-hme0 netmask + broadcast + deprecated -failover group unrealgrp1 up addif unreal netmask + broadcast + failover up

This configures physical hme0 with IP address of 192.168.11.6, which will not fail over and puts hme0 in unrealgrp1 multipathing group. It will additionally configure a virtual IP address of 192.168.11.5 which will failover when hme0 link goes down. Deprecated means the IP address 192.168.11.5 will not be used as source address for any outgoing packets.

Then I edited /etc/hostname.hme1:

unreal-hme1 netmask + broadcast + deprecated -failover group unrealgrp1 up

Similarly, hme1 will be configured with IP address of 192.168.11.7 and as a member of unrealgrp1 multipathing group. Again, 192.168.11.7 is marked as deprecated so it will not be used for outgoing packets. Finally I made sure my hosts file is correct:

bash-3.00# cat /etc/hosts
127.0.0.1 localhost
192.168.11.5 unreal loghost
192.168.11.6 unreal-hme0
192.168.11.7 unreal-hme1

And here is the result:

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=9040843 mtu 1500 index 2
inet 192.168.11.6 netmask ffffff00 broadcast 192.168.11.255
groupname unrealgrp1
ether 8:0:20:d9:ac:c
hme0:1: flags=1000843 mtu 1500 index 2
inet 192.168.11.5 netmask ffffff00 broadcast 192.168.11.255
hme1: flags=19040803 mtu 1500 index 3
inet 192.168.11.7 netmask ffffff00 broadcast 192.168.11.255
groupname unrealgrp1
ether 8:0:20:d9:ac:d

Essentially IP address 192.168.11.5 “floats” among interfaces. If I were to unplug hme0, 192.168.11.5 would fail over to hme1. Failure is detected on link loss. There are some tunable parameters in /etc/default/mpathd.

This is all I needed. Of course, there is much more to IPMP: you can setup test system, that your system will test for reachability, detection of interfaces missing on boot, etc. Sun has much more info on it here. Continue Reading

You have a SVM mirror and you need to grow it. Online. All steps are straight forward except for the last one. So, in a nutshell:

1. Grow sizes of both submirror LUN’s on SAN
2. Detach the first submirror metadevice
3. Clear the submirror metadevice
4. Make the system see new LUN size
5. Recreate the submirror metadevice, attach it to the mirror and let it sync
6. Repeat the above for the second submirror
7. Not so obvious step
8. Grow the filesystem

You have a mirror metadevice d200 consisting of submirrors d201 and d202. Both submirrors are sliced in such way that all space is allocated in slice 0. At this point both LUN’s should be already grown, but you need to make the system to see the size increase. Here is what d201, one of the submirrors of d200, should look like:

bash-3.00# metastat d201
d201: Submirror of d200
State: Okay
Size: 2064384 blocks (1008 MB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

First, detach d201 from its parent and clear it:

bash-3.00# metadetach d200 d201
bash-3.00# metaclear d201

The next step is likely to alter the LUN slice configuration, so make sure you have it noted.

You need to run format command to get the system to see the grown LUN. So, run format then select type and then select Autoconfigure. At this point format should be reporting the new LUN size. Make sure your slice configuration is correct and that all the new space is tacked onto the end of correct slice. In this case, it’s slice 0.

Now recreate d201, attach it to d200 and let the mirror sync. When checking status of d200 note that overall mirror size is still the same but the size of d201 increased:

bash-3.00# metastat d200
d200: Mirror
Submirror 0: d202
State: Okay
Submirror 1: d201
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 2064384 blocks (1008 MB)

d202: Submirror of d200
State: Okay
Size: 2064384 blocks (1008 MB)
Stripe 0:
Device                                             Start Block  Dbase        State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003460000d0s0          0     No            Okay   Yes

d201: Submirror of d200
State: Okay
Size: 4128768 blocks (2.0 GB)
Stripe 0:
Device                                             Start Block  Dbase        State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003410000d0s0          0     No            Okay   Yes


Now repeat the same process with d202. After you are done you will have both submirrors bigger, but the size of overall mirror will still be the same:

bash-3.00# metastat d200
d200: Mirror
Submirror 0: d201
State: Okay
Submirror 1: d202
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 2064384 blocks (1008 MB)

d201: Submirror of d200
State: Okay
Size: 4128768 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

d202: Submirror of d200
State: Okay
Size: 4161536 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
/dev/dsk/c5t511714A510011C370000C00003460000d0s0 0 No Okay Yes

Now comes the “Not so obvious” step. In order to increase overall mirror size you need to run metattach on the mirror itself:

bash-3.00# metattach d200
bash-3.00# metastat d200
d200: Mirror
Submirror 0: d201
State: Okay
Submirror 1: d202
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 4128768 blocks (2.0 GB)

This will grow the size of the entire mirror to the size of submirror devices. Now you can grow the filesystem on it without unmounting it. Keep in mind that for the duration of growfs command there will be no writes to the volume.

bash-3.00# growfs -M /export/home/users /dev/md/rdsk/d200
/dev/md/rdsk/d200: 4128768 sectors in 126 cylinders of 128 tracks, 256 sectors
2016.0MB in 63 cyl groups (2 c/g, 32.00MB/g, 15040 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 65824, 131616, 197408, 263200, 328992, 394784, 460576, 526368, 592160,
3487008, 3552800, 3618592, 3684384, 3750176, 3815968, 3881760, 3947552,
4013344, 4079136

That’s all there is to it. The “Not so obvious” step is mentioned in the metattach man page. But unless you do this stuff every day, this very useful piece of information can be easily forgotten. Continue Reading

I was configuring MPxIO on Sun Fire V490. After issuing stmsboot -e command and rebooting the server, internal disks were under MPxIO’s control.

This was not something I wanted and I needed to disable it. In order to do so I had to edit /kernel/drv/fp.conf file and tell Solaris not to enable MPxIO on internal disks. So I added the following line at the end of fp.conf file:

name="fp" parent="/pci@8,600000/SUNW,qlc@4" port=0 mpxio-disable="yes";

This line tells Solaris to disable MPxIO on port 0 for all devices whose parent device is /pci@8,600000/SUNW,qlc@4. Of course, similar line should be added for all HBA’s and ports you do not want to have under MPxIO’s control. You can get the parent device from your /var/adm/messages file or from device links pointing to the internal disks. Here is a partial list of device links:
<—————–SNIP—————->
lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s0 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:a
lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s1 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:b
lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s2 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:c
lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s3 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:d
lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s4 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:e
lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s5 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:f
lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s6 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:g
lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s7 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:h
<—————–SNIP—————-> Continue Reading