fsck: Error determining size of the physical device: File too large

So, I got done patching this fairly important server, running RedHat 5.10. For reasons I will not go into, the server has about 30TB of direct attached storage. After the reboot, one of the filesystems, in this case it had 16TB of storage, failed to pass automatic fsck check on boot.

So, I booted it into single user mode to take a look:

[root@prdnb0001 dev]# fsck -n /dev/vgmd1200/optlv
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
Error determining size of the physical device: File too large

That did not look promising. I was little dumbfounded. I googled a few things, saw some not so helpful pages, but nothing 100% definite.

Nevertheless, I got a little bit of inspiration. By the way, my version of e4fsprogs was 1.41.12-3.el5. The filesystem was ext4 filesystem so I gave this a shot:

[root@prdnb0001 dev]# fsck.ext4 -n /dev/vgmd1200/optlv

…and filesystem check proceeded to run just fine. I did not have time to dig into why this worked and automatic fsck check on boot failed. Maybe someone can shed some light on this.

In any case, reading man pages always pays and once in a while you find out something cool such as:

SIGUSR1 This signal causes e2fsck to start displaying a completion bar or emitting progress information.

So by doing:

[root@prdnb0001 ~]# pkill -SIGUSR1 fsck.ext4

in another terminal window, I was able to get progress bar for the running check.

Posted on September 30, 2013 at 09:42 by somedude · Permalink · Leave a comment
In: centos, ext4, linux, linux tips, redhat, storage

Unfinished transactions in yum

I was patching this “pseudo server” the other day. Resources on the server were not monitored. Of course / was low on space and yum failed. Subsequent run of yum update threw the following message:

[root@web0001 sbin]# yum update
Loaded plugins: rhnplugin, security
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Install Process
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them.
The program yum-complete-transaction is found in the yum-utils package.

Not wanting to mess anything up at that moment, I installed yum-utils package. yum-utils contains yum-complete-transaction command. Using this tool, you can either finish incomplete transactions or you can clean up transaction files. In either case the directory of interest is /var/lib/yum.

All that was needed to be done to restart update process was to clean up transaction files:

[root@web0001 sbin]# /usr/sbin/yum-complete-transaction --cleanup-only
Loaded plugins: rhnplugin
This system is receiving updates from RHN Classic or RHN Satellite.
Cleaning up unfinished transaction journals
Cleaning up 2013-08-14.09:04.41

Posted on August 28, 2013 at 15:07 by somedude · Permalink · Leave a comment
In: centos, linux, linux tips, linux utilities, redhat

Oracle DB failed to start: ORA-27102: out of memory

Oracle not being exactly my turf, I ran into this last week. After some patching and reboot of RHEL 5.9 server, Oracle DB failed to start. So, I took a peek inside /opt/oracle/product/11.2.0/db_1/startup.log and found this:

ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device
SYS@PRD001 > Disconnected

Lovely. Unfortunately, no DBA was around so I had to do some searching. Thankfully I came across this page that pretty much fit my case.

[root@db0001 db_1]# getconf PAGE_SIZE
4096
[root@db0001 db_1]# sysctl -a | grep shmall
kernel.shmall = 5243392

Going by the suggestion from the above link shmall for my server with 32GB of memory should be: 1024 * 1024 * 1024 * 32 / 4096 which comes to 8388608. This was clearly at odds with what sysctl reported. So…

[root@db0001 db_1]# sysctl kernel.shmall=8388608

After that database startup was successful. Of course, I corrected the parameter in /etc/sysctl.conf so it persists across reboots.

Later I found out whose handy work the incorrect value was… No, it was not me…

Posted on July 27, 2013 at 08:57 by somedude · Permalink · Leave a comment
In: centos, linux, oracle, redhat

vShield Manager CLI password change

This just seems nonsensical to me. Apparently, you cannot change user passwords via CLI in vShield Manager 5.1.2, but you have to go through the rigmarole of removing and recreating accounts. Specifically, I needed to change password for admin account.

Moreover, CLI admin account is separate entity from admin account used in conjunction with web interface!

So, first create a temporary admin account and log out:

manager# config t
manager(config)# user tempadmin password plaintext pass1
manager(config)# exit
manager# write mem
Building Configuration...
Configuration saved.
[OK]
manager# exit

Then log back in using tempadmin account, delete admin account and re-create it using desired password:

manager# config t
manager(config)# no user admin
manager(config)# user admin password plaintext pass2
manager(config)# exit
manager# write mem
Building Configuration...
Configuration saved.
[OK]
manager#

…and finally, logout as tempadmin, log back in as admin and remove tempadmin account:

manager# config t
manager(config)# no user tempadmin
manager(config)# exit
manager# write mem
Building Configuration...
Configuration saved.
[OK]
manager#

More on this is here… And yes, article recommends removing admin completely.

Posted on June 30, 2013 at 10:31 by somedude · Permalink · 2 Comments
In: virtualization, vmware, vshield

Copying directories with tar

This is one of those, “can’t seem to memorize something” posts… Sometimes copying files with cp is just painful. Instead:

[root@sparky ~]# mkdir /tmp/newopt && mount /dev/mapper/system-opt /tmp/newopt
[root@sparky ~]# ( cd /opt && tar --xattrs -cf - . ) | ( cd /tmp/newopt && tar xvfp - )

This will copy files and directories from /opt to /newopt, including extended attributes like acl’s and SELinux contexts. And it’s faster than using cp as well…

Posted on May 27, 2013 at 09:48 by somedude · Permalink · Leave a comment
In: linux, linux tips, shell, solaris, solaris tips