svc.configd: smf(5) database integrity check of:
/etc/svc/repository.db
failed. The database might be damaged or a media error might have
prevented it from being verified. Additional information useful to
your service provider is in:
/etc/svc/volatile/db_errors
The system will not be able to boot until you have restored a working
database. svc.startd(1M) will provide a sulogin(1M) prompt for recovery
purposes. The command:
/lib/svc/bin/restore_repository
can be run to restore a backup version of your repository. See
http://sun.com/msg/SMF-8000-MY for more information.
Having never seen this error, I was thinking: “this is gonna be interesting…”. Thankfully the error was pretty verbose so I started to disect it section by section. Yeah, service repository got hosed, somehow, and I can potentially find some usefull info in /etc/svc/volatile/db_errors. Unfortunatelly, there was nothing of use in there.
The restore_repository script mentioned gave me little more hope. I also went and checked out the page URL. After reading the page I decided to go ahead and try to restore the service repository.
I logged in to the box in single user mode and took a look at the restore script to get an idea of what it might do. Then, I ran it. Fortunatelly, the script was pretty good at doing checks and told me that I can not proceed any further because / filesystem is mounted RO. To fix this I was asked to run:
bash-3.00# /lib/svc/method/fs-root
bash-3.00# /lib/svc/method/fs-usr
Once the filesystems were fixed up I ran the restore_repository script. I was asked which backup copy I wanted to restore and that was it. The system rebooted and came back up fine. This turned out to be a pretty good learning experience and http://www.sun.com/msg/SMF-8000-MY is very well worth reading.
]]>core# test cable-diagnostics tdr interface gigabitethernet0/17
Unfortunately it’s not available on all switches. I dug this up on here Cisco site. And here is the command reference…
]]>Create database replicas:
metadb -f -a -c [number_of_replicas] [device]
metadb -f -a -c 3 c0t0d0s7
Delete all database replicas from device:
metadb -d [device]
metadb -d c0t0d0s7
Display status of database replicas:
metadb -i
metadb -i
Display metadevice status:
metastat
metastat
Create simple concat/stripe metadevice:
metainit -f [concat_metadevice] 1 1 [device]
metainit -f d21 1 1 c0t0d0s1
Create a mirror with one submirror:
metainit [mirror_metadevice] -m [submirror_metadevice]
metainit d20 -m d21
Attach a submirror to one sided mirror:
metattach [mirror_metadevice] [submirror_metadevice]
metattach d20 d22
Detach a submirror from a mirror:
metadetach [mirror_metadevice] [submirror_metadevice]
metadetach d20 d22
Clear a metadevice:
metaclear [metadevice]
metaclear d22
Offline a submirror:
metaoffline [mirror_metadevice] [submirror_metadevice]
metaoffline d20 d22
Online a submirror:
metaonline [mirror_metadevice] [submirror_metadevice]
metaonline d20 d22
Enable a failed component:
metareplace -e [metadevice] [device]
metareplace -e d21 c0t0d0s1
Rename a metadevice:
metarename [old_metadevice] [new_metadevice]
metarename d20 d30
Switch metadevice names:
metarename [metadevice_1] [metadevice_2]
metarename -x d20 d30
Configure system for root metadevice:
metaroot [metadevice]
metaroot d10
Gear used in this “experiment”:
Rough steps I took to get this working:
First and foremost, make sure qla2xxx driver is loaded. You also have to make sure you have device-mapper-multipath-0.4.7-23.el5 installed. Next, configure multipathing daemon so that it starts on boot:
[root@carbon ~]# chkconfig multipathd on
When that’s done you need to make the system aware of the presented LUN’s. One way to do so is to reboot the server. Another option is to force HBA scan:
[root@carbon ~]# echo "- - -" > /sys/class/scsi_host/host1/scan
During this you should watch /var/log/messages to see if your LUN’s are detected. When done, make multipathd aware of the LUN’s:
[root@carbon ~]# multipath -v2 -d
The above command is a “dry run”. There will be no device map changes committed. You will only be shown device mapper changes that will be made. To commit device map changes run:
[root@carbon ~]# multipath -v2
Once this is done you can see what multipathd is seeing:
[root@carbon ~]# multipath -ll
mpath2 (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][enabled]
\_ 1:0:3:1 sde 8:64 [active][ready]
\_ 2:0:3:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 1:0:2:1 sdd 8:48 [active][ready]
\_ 2:0:2:1 sdg 8:96 [active][ready]
mpath1 (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][enabled]
\_ 1:0:0:1 sdb 8:16 [active][ready]
\_ 2:0:4:1 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 1:0:1:1 sdc 8:32 [active][ready]
\_ 2:0:1:1 sdf 8:80 [active][ready]
If everything looks good, you can create configuration file for multipathd. You will need to edit /etc/multipath.conf and depending on your environment, add or modify some parameters. The configuration file contains enough comments and examples to figure out what different parameters mean. When in doubt, consult the man pages.
First, add a blacklist section, which will make certain device exempt from multipathing. I have my internal drives listed in blacklist section:
blacklist {
devnode "^sd[a-b].*"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
Next, you are going to need device section. This is going to be specific to your SAN. The one below is for EVA5000. I got the parameters from HP’s device mapper package:
device {
vendor "HP|COMPAQ"
product "HSV1[01]1 \(C\)COMPAQ|HSV[2][01]0|HSV300"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
path_checker tur
path_selector "round-robin 0"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
rr_weight uniform
failback immediate
hardware_handler "0"
no_path_retry 12
rr_min_io 100
}
You should also look at defaults section to make sure it is configured for your setup. Again, the parameters in mine are specific to EVA5000:
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy failover
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/bin/true"
path_checker tur
rr_min_io 100
rr_weight uniform
failback immediate
no_path_retry 12
user_friendly_names yes
bindings_file "/var/lib/multipath/bindings"
}
Finally, you will need to specify configuration for the presented LUN’s. This applies to the multipaths section of multipath.conf file:
multipath {
wwid 3600508b4001031250000900001490000
alias san1data
}
multipath {
wwid 3600508b400011c300000f00001a90000
alias san2data
}
After you are done, restart multipathd and check output of multipath -ll command:
[root@carbon ~]# multipath -ll
san2data (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:3:1 sde 8:64 [active][ready]
\_ 2:0:3:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 1:0:2:1 sdd 8:48 [active][ready]
\_ 2:0:2:1 sdg 8:96 [active][ready]
san1data (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][enabled]
\_ 1:0:0:1 sdb 8:16 [active][ready]
\_ 2:0:4:1 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 1:0:1:1 sdc 8:32 [active][ready]
\_ 2:0:1:1 sdf 8:80 [active][ready]
That should be it. You should test the setup by disabling paths to see if your LUN’s stay up.
]]>I have done this on Solaris 10 8/07. Before going any further, it is important that both host A and host B are running the same release of Solaris and they are both at the same patch level. Otherwise, you will almost certainly run into a situation where the container will refuse to attach to the new host.
I have created a cloned container called mx2. First, I have detached the container:
bash-3.00# zoneadm -z mx2 detach
Then I compressed the container directory so I could move it to host B. It does not really matter which tool you use to compress the directory. Just make sure you preserve permissions, ownership or ACL’s . For me, for some reason tar had a little of an ordeal compressing the container directory:
bash-3.00# cd /export/home/zones
bash-3.00# tar cf mx2.tar mx2
tar: mx2/root/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/cpu/sparcv9+vis/sparcv9/libclib_jiio.so: symbolic link too long
tar: mx2/root/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/cpu/sparcv9+vis2/sparcv9/libclib_jiio.so: symbolic link too long
Once I had the directory compressed I scp-ied it to host B. Then I decompressed it in the zones directory:
bash-3.00# cd /export/home/zones
bash-3.00# tar xf mx2.tar
I have recreated the problematic symlinks mentioned above manually. If you are having trouble attaching the container check that container configurations are the same on both systems.
Before you can attach a container, you need to have container configuration in place. Without it you will not be able to attach the container.
Make sure your configuration is correct. I was attaching full root container and the system was complaining that the container being attached is missing some packages. In reality, my container configuration was for sparse root container. It turned out that when I was importing container configuration, some inherit-package-dir statements were added which has caused attach operation to fail. I had to remove those manually.
So, again, before attaching the container make sure your container configuration is good, you are inheriting correct package directories, etc. Once you have that right, you can attach the new container:
bash-3.00# zoneadm -z mx2 attach
That’s it. Depending on your needs you might want to change hostname, ip address and so on.
Some interesting linkage:
]]>bash-3.00# mount -F nfs carbon:/media/cdrecorder /mnt/carbon
nfs mount: mount: /mnt/carbon: Not owner
So, I checked and rechecked my settings with no success. Then, I remembered reading something somewhere about NFS v4 in Linux being not so great at one time. Since the Linux box was running RHEL 4 I tought this might be my problem. So, I decided to force mount using NFS v3, since Solaris 10 will try to mount the Linux share using NFS v4 first.
bash-3.00# mount -F nfs -o vers=3 carbon:/media/cdrecorder /mnt/carbon
bash-3.00# cd /mnt/carbon
bash-3.00#
That worked well. Since this was one time mounting job, I did not bother any further. If I would be doing this on regular basis I would probably edit /etc/default/nfs on the Solaris box and force maximum NFS client version to be v3.
]]>Here is how it looks in format command:
partition> p
Current partition table (original):
Total disk sectors available: 73383902 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 34 34.99GB 73383902
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
8 reserved wm 73383903 8.00MB 73400286
If you use straight format command there is no way to get rid of EFI label. Instead, you have to use -e option with format to get to expert features:
bash-3.00# format -e
With -e options when you label the LUN you will be asked about the type of label you want to put on the disk:
format> label
[0] SMI Label
[1] EFI Label
Specify Label type[1]: 0
Warning: This disk has an EFI label. Changing to SMI label will erase all
current partitions.
Continue? y
Auto configuration via format.dat[no]? yes
I suppose you could alternatively use dd command and blank out the whole disk.
]]>To get alternate superblock locations do:
newfs -Nv /dev/rdsk/c0t0d0s3
This will not actually format the slice. It will only print what it would do, if it was actually formating the slice. Basically, it is newfs dry run. It will also give you locations of alternate superblocks on the slice. I guess it should be mentioned that if you used non-default values while formating the slice originally, and you did not make a note of those values, this might not give you valid superblock numbers.
After you have your alternate superblock numbers you can use them to help out fsck (in this case superblock I am using is in block 8192):
fsck -y -F ufs -o b=8192 /dev/rdsk/c0t0d0s3
It might not hurt to run the above with -n option instead of -y to make sure it produces the results you are looking for.
Hopefully, the filesystem is not hosed so badly that it can not be repaired. This was more of an issue when Solaris did not have ufs logging. Nevertheless, it does not hurt to know this for one of those emergencies.
]]>First you need configuration for the new container. You can either make one from scratch or export one from existing container and customize it.
The following command will export the configuration from “gold” master container and save it into /tmp/mx1.cfg file:
bash-3.00# zonecfg -z gold export -f /tmp/mx1.cfg
Now you can edit the configuration. Word of caution, importing seems to be a little flaky. For example, if an existing container has bootargs variable set, you will need to put quotes around them in the config file. So, if my bootargs are set to -m verbose, the string will have to be enclosed in “”. Otherwise the new container configuration will not be imported properly.
Once you have the configuration ready you can import it:
bash-3.00# zonecfg -z mx1 -f /tmp/mx1.cfg
Now you can perform actual clone operation:
bash-3.00# zoneadm -z mx1 clone -m copy gold
Cloning zonepath /export/home/zones/gold...
This should not take very long. Once the operation is done you can see that the new container has been installed:
bash-3.00# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
- mx1 installed /export/home/zones/mx1 native shared
- gold installed /export/home/zones/gold native shared
At this point you can boot the new container. You will be taken through initial configuration of the container. This initial container configuration can be pain if you are rolling out a lot of containers. You can simplify the process by creating a sysidcfg file and putting it into /etc directory of the container before performing the first boot.
In my case I would put the following sysidcfg into /export/home/zones/mx1/root/etc:
system_locale=C
terminal=vt100
network_interface=primary {
hostname=mx1
}
security_policy=NONE
name_service=DNS {
domain_name=example.com
name_server=192.168.1.1
search=example.com
}
nfs4_domain=dynamic
timezone=US/Central
root_password=n2GSHfh
The nice thing about cloning is the fact that it’s fairly quick. So if you hose up a container, you can simply run
bash-3.00# zoneadm -z mx1 uninstall
Are you sure you want to uninstall zone mx1 (y/[n])? y
followed by
bash-3.00# zonecfg -z mx1 delete
Are you sure you want to delete zone mx1 (y/[n])? y
and start all over again.
]]>I did a little bit of searching, and all I found were conflicting reports: it should work, it does not work, it works, but not really…
First of all the container that will host JetDirect has to be full-root, as JetDirect install will put some files into /usr/spool. I suppose it would be possible to work around it and use sparse-root, even though it might be a little messy.
The problem I ran into was while creating print queues using hppi tool. During the process mknod command is called to create printer device file in /dev directory of the container. You can not use mknod inside a container and creation of the device file will fail. Sure enough, greping for mknod in JetDirect’s admin directory yields the following:
bash-3.00$ grep mknod *
addptrtoq: mknod $DFILE c 13 2
Closer look at addptrtoq reveals the following:
HPNPMODEL=$HPNP/sh/hpnp.model
#DFILE=$HPNP/etc/$SPOOLNAME
DFILE=/dev/$SPOOLNAME
if [ ! -c $DFILE ]
then
mknod $DFILE c 13 2
fi
chmod 666 $DFILE
So to get around the problem, first you manually have to create the special file in the global zone inside /dev directory of the print server container. Then you can proceed with installing the printqueue using hppi tool.
bash-3.00# cd /export/home/zones/prtsvr/dev/
bash-3.00# mknod laser c 13 2
bash-3.00# chmod 666 laser
Now you should be able to add a printer queue without any problems. So JetDirect works just fine inside Solaris 10 Container. For me anyways…
]]>