unixpowered.com

One of the apps running on Sun Cluster was randomly crashing. So, I decided to take a look what was happening. Yeah, there is DTrace in Solaris 10. Since I am pretty comfortable with truss I decided to give that a shot first:

root@node1 # truss -p 27462
truss: process is traced: 27462
root@node1 #

That’s it. No truss output, nothing. That was weird. truss will not work if there is a debugger attached to the process to be traced, which was not the case. So, I figured it might have something to do with the fact that the process is handled by the cluster software.

Finaly, NOTES section of pmfadm manpage gave me the answer:
To avoid collisions with other controlling processes. truss(1) does not allow tracing a process that it detects as being controlled by another process by way of the /proc interface. Since rpc.pmfd(1M) uses the /proc interface to monitor processes and their descendents, those processes that are submitted to rpc.pmfd by way of pmfadm cannot be traced or debugged.
So, Dtrace it was. Thankfully, Brendan Gregg already did the hard work for me, by creating DTrace version of truss. The more you know… Continue Reading

Using scconf you can view some useful information about Sun Cluster configuration such as information about cluster transport, disksets, etc.
Using multiple v’s you can increase output verbosity. It’s pretty useful especially if you inherited the cluster.
I added this command to my previous list, as well.

root@node1 # scconf -p -v
Cluster name: cluster
Cluster ID: 0x4538908A
Cluster install mode: disabled
Cluster private net: 172.16.0.0
Cluster private netmask: 255.255.0.0
Cluster new node authentication: unix
Cluster new node list: <. - Exclude all nodes>
Cluster transport heart beat timeout: 10000
Cluster transport heart beat quantum: 1000
Cluster nodes: node1 node2

Cluster node name: node1
(node1) Node ID: 1
(node1) Node enabled: yes
(node1) Node private hostname: clusternode1-priv
(node1) Node quorum vote count: 1
(node1) Node reservation key: 0x4538908A00000001
(node1) Node transport adapters: ce2 ce0

(node1) Node transport adapter: ce2
(node1:ce2) Adapter enabled: yes
(node1:ce2) Adapter transport type: dlpi
(node1:ce2) Adapter property: device_name=ce
(node1:ce2) Adapter property: device_instance=2
(node1:ce2) Adapter property: lazy_free=1
(node1:ce2) Adapter property: dlpi_heartbeat_timeout=10000
(node1:ce2) Adapter property: dlpi_heartbeat_quantum=1000
(node1:ce2) Adapter property: nw_bandwidth=80
(node1:ce2) Adapter property: bandwidth=70
(node1:ce2) Adapter property: netmask=255.255.255.128
(node1:ce2) Adapter property: ip_address=172.16.0.129
(node1:ce2) Adapter port names: 0

(node1:ce2) Adapter port: 0
(node1:ce2@0) Port enabled: yes

(node1) Node transport adapter: ce0
(node1:ce0) Adapter enabled: yes
(node1:ce0) Adapter transport type: dlpi
(node1:ce0) Adapter property: device_name=ce
(node1:ce0) Adapter property: device_instance=0
(node1:ce0) Adapter property: lazy_free=1
(node1:ce0) Adapter property: dlpi_heartbeat_timeout=10000
(node1:ce0) Adapter property: dlpi_heartbeat_quantum=1000
(node1:ce0) Adapter property: nw_bandwidth=80
(node1:ce0) Adapter property: bandwidth=70
(node1:ce0) Adapter property: netmask=255.255.255.128
(node1:ce0) Adapter property: ip_address=172.16.1.1
(node1:ce0) Adapter port names: 0

(node1:ce0) Adapter port: 0
(node1:ce0@0) Port enabled: yes

Cluster node name: node2
(node2) Node ID: 2
(node2) Node enabled: yes
(node2) Node private hostname: clusternode2-priv
(node2) Node quorum vote count: 1
(node2) Node reservation key: 0x4538908A00000002
(node2) Node transport adapters: ce2 ce0

(node2) Node transport adapter: ce2
(node2:ce2) Adapter enabled: yes
(node2:ce2) Adapter transport type: dlpi
(node2:ce2) Adapter property: device_name=ce
(node2:ce2) Adapter property: device_instance=2
(node2:ce2) Adapter property: lazy_free=1
(node2:ce2) Adapter property: dlpi_heartbeat_timeout=10000
(node2:ce2) Adapter property: dlpi_heartbeat_quantum=1000
(node2:ce2) Adapter property: nw_bandwidth=80
(node2:ce2) Adapter property: bandwidth=70
(node2:ce2) Adapter property: netmask=255.255.255.128
(node2:ce2) Adapter property: ip_address=172.16.0.130
(node2:ce2) Adapter port names: 0

(node2:ce2) Adapter port: 0
(node2:ce2@0) Port enabled: yes

(node2) Node transport adapter: ce0
(node2:ce0) Adapter enabled: yes
(node2:ce0) Adapter transport type: dlpi
(node2:ce0) Adapter property: device_name=ce
(node2:ce0) Adapter property: device_instance=0
(node2:ce0) Adapter property: lazy_free=1
(node2:ce0) Adapter property: dlpi_heartbeat_timeout=10000
(node2:ce0) Adapter property: dlpi_heartbeat_quantum=1000
(node2:ce0) Adapter property: nw_bandwidth=80
(node2:ce0) Adapter property: bandwidth=70
(node2:ce0) Adapter property: netmask=255.255.255.128
(node2:ce0) Adapter property: ip_address=172.16.1.2
(node2:ce0) Adapter port names: 0

(node2:ce0) Adapter port: 0
(node2:ce0@0) Port enabled: yes

Cluster transport junctions: switch1 switch2

Cluster transport junction: switch1
(switch1) Junction enabled: yes
(switch1) Junction type: switch
(switch1) Junction port names: 1 2

(switch1) Junction port: 1
(switch1@1) Port enabled: yes

(switch1) Junction port: 2
(switch1@2) Port enabled: yes

Cluster transport junction: switch2
(switch2) Junction enabled: yes
(switch2) Junction type: switch
(switch2) Junction port names: 1 2

(switch2) Junction port: 1
(switch2@1) Port enabled: yes

(switch2) Junction port: 2
(switch2@2) Port enabled: yes

Cluster transport cables

Endpoint Endpoint State
-------- -------- -----
Transport cable: node1:ce2@0 switch1@1 Enabled
Transport cable: node1:ce0@0 switch2@1 Enabled
Transport cable: node2:ce2@0 switch1@2 Enabled
Transport cable: node2:ce0@0 switch2@2 Enabled

Quorum devices: d9

Quorum device name: d9
(d9) Quorum device votes: 1
(d9) Quorum device enabled: yes
(d9) Quorum device name: /dev/did/rdsk/d9s2
(d9) Quorum device hosts (enabled): node1 node2
(d9) Quorum device hosts (disabled):
(d9) Quorum device access mode: scsi2

Device group name: dskset1-dg
(appset1-dg) Device group type: SVM
(appset1-dg) Device group failback enabled: no
(appset1-dg) Device group node list: node1, node2
(appset1-dg) Device group ordered node list: yes
(appset1-dg) Device group desired number of secondaries: 1
(appset1-dg) Device group diskset name: dskset1-dg Continue Reading

By default Solaris 10 starts 16 NFS servers to handle NFS requests. You can tune this by editing /etc/default/nfs file.
<—————–SNIP—————->
# Maximum number of concurrent NFS requests.
# Equivalent to last numeric argument on nfsd command line.
NFSD_SERVERS=16
<—————–SNIP—————->
Changing above variables did not seem to have any effects on how many NFS server Sun Cluster started. Poking around I found nfs_start_daemons script which is part of SUNWscnfs package. In my case it was in /opt/SUNWscnfs/bin directory. It turns out that this script is looking at pre-Solaris 10 nfs.server init script to determine if more than 16 NFS servers are supposed to be started. In Solaris 10 NFS server as most of the services is handled by SMF. The /etc/init.d/nfs.server script is still present, probably due to legacy reasons, but it simply calls svcadm command to start NFS. Here is the relevant section of nfs_start_daemons script:
<—————–SNIP—————->
DEFAULT_NFSDCMD="/usr/lib/nfs/nfsd -a 16"
if [ -f /etc/init.d/nfs.server ]; then
NFSDCMD="`egrep '^[^#]*/usr/lib/nfs/nfsd' \
/etc/init.d/nfs.server \
2>/dev/null | head -1`"
fi
<—————–SNIP—————->
In order to increase number of NFS server that get started by Sun Cluster, you must change the number 16 above to something higher like 1024. Continue Reading

Some useful Sun Cluster commands

Shut down a resource group:
scswitch -F -g [RESOURCE_GROUP_NAME]

Bring up a resource group:
scswitch -Z -g [RESOURCE_GROUP_NAME]

Move failover resource group to node_name:
scswitch -z -g [RESOURCE_GROUP_NAME] -h [NODE_NAME]

Restart resource group:
scswitch -R -h [NODE_NAME] -g [RESOURCE_GROUP_NAME]

Evacuate all resources from node_name:
scswitch -S -h [NODE_NAME]

Disable resource:
scswitch -n -j [RESOURCE]

Enable resource:
scswitch -e -j [RESOURCE]

Clear STOP_FAILED on resource:
scswitch -c -j [RESOURCE] -h [NODE_NAME] -f STOP_FAILED

Disable resource’s fault monitor:
scswitch -n -M -j [RESOURCE]

Enable resource’s fault monitor:
scswitch -e -M -j [RESOURCE]

Enable resource’s fault monitor:
scswitch -e -M -j [RESOURCE]

Lists currently configured DID’s:
scdidadm -L

Put a new device under cluster control:
scgdevs

Displays status of the cluster, resources, resource groups, etc.:
scstat

Display useful setup info about cluster nodes, cluster transport, disksets, etc.:
scconf -p -v Continue Reading