Another take on Dynamic Volume Resize in RHEL 6

Being able to resize an LVM volume without reboot is quite handy. I had described one way here. This is a slightly different take on the same topic, this time on RHEL 6.

I had resized /dev/sdb from 48GB to 140GB in vCenter. However, the operating system is still seeing the old size:

[root@build ~]# fdisk -l /dev/sdb
 
Disk /dev/sdb: 48.3 GB, 48318382080 bytes
64 heads, 32 sectors/track, 46080 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000a

To get RHEL to see the new size, device rescan is needed:

[root@build ~]# cd /sys/class/block/sdb/device
[root@build ~]# echo "- - -" > rescan

The difference this time is that you do not need to look for the correct HBA and LUN number. You can just go by the device name. Now the operating system should see the correct disk size:

[root@build device]# fdisk -l /dev/sdb
 
Disk /dev/sdb: 198.6 GB, 198642237440 bytes
255 heads, 63 sectors/track, 24150 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Now you can proceed with resizing the physical volume and the logical volume. You can throw in -t switch to do a dry run:

[root@build device]# pvresize /dev/sdb
  Physical volume "/dev/sdb" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
[root@build device]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda2  system lvm2 a--   19.47g   3.00g
  /dev/sdb   vg_app lvm2 a--  185.00g 140.00g

The size is correct now. Finally, resize the logical volume. Again, you can throw in -t switch to do a dry run:

[root@build device]# lvresize -r -l100%FREE /dev/mapper/vg_app-lv_app
  Size of logical volume vg_app/lv_app changed from 45.00 GiB (11519 extents) to 140.00 GiB (35840 extents).
  Logical volume lv_app successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_app-lv_app is mounted on /var/www/html/repo; on-line resizing required
old desc_blocks = 3, new_desc_blocks = 9
Performing an on-line resize of /dev/mapper/vg_app-lv_app to 36700160 (4k) blocks.
The filesystem on /dev/mapper/vg_app-lv_app is now 36700160 blocks long.

Done

Posted on December 17, 2013 at 09:37 by somedude · Permalink · Leave a comment
In: centos, fibre channel, linux, linux tips, lvm2, san, storage

There is no screen to be resumed…

I was running a script on a server that took a while to complete. Since I did not want to lose any output I ran it inside a screen session. Of course, during the course of this, my SSH connection dropped and when I logged in again, and tried to reattached to the session I got this:

[root@ultra ~]# screen -list
There is a screen on:
21884.pts-0.ultra (Attached)
1 Socket in /var/run/screen/S-root.
[root@ultra ~]# screen -r 21884.pts-0.ultra
There is a screen on:
21884.pts-0.ultra (Attached)
There is no screen to be resumed matching 21884.pts-0.ultra.

As it turns out, you have I had to first detach and logout first, to be able to attach to the session:

[root@ultra ~]# screen -D -r 21884.pts-0.ultra

Useful to know…

Posted on November 15, 2013 at 08:56 by somedude · Permalink · Leave a comment
In: linux, linux tips, linux utilities, screen, shell

Hung Linux system: Now what?

Finally, I had remembered to try out Magic SysRq key in real life. One of the XenServer hosts decided to refuse to reboot and hung during shutdown. Magic SysRq is really a keyboard combo that allows user to force kernel to perform certain function, even though machine is unresponsive. In my case I needed to safely reboot the host.

For Magic SysRq to work, it must be enabled in kernel:

[root@db0004 ~]# cat /proc/sys/kernel/sysrq
1

If output is 1, then you are good to go. Thankfully, XenServer had this enabled. So, while holding down Alt and SysRq keys I pressed the following keys: r e i s u b.

So, what do all the letters mean? I ripped the explanation from Wikipedia. Article is worth checking out, especially the section on how to deal with keyboards other than QWERTY.

The Wikipedia article also mentions couple of mnemonics to help remembering the key sequence.

Posted on October 30, 2013 at 09:04 by somedude · Permalink · Leave a comment
In: centos, linux, linux tips, redhat

fsck: Error determining size of the physical device: File too large

So, I got done patching this fairly important server, running RedHat 5.10. For reasons I will not go into, the server has about 30TB of direct attached storage. After the reboot, one of the filesystems, in this case it had 16TB of storage, failed to pass automatic fsck check on boot.

So, I booted it into single user mode to take a look:

[root@prdnb0001 dev]# fsck -n /dev/vgmd1200/optlv
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
Error determining size of the physical device: File too large

That did not look promising. I was little dumbfounded. I googled a few things, saw some not so helpful pages, but nothing 100% definite.

Nevertheless, I got a little bit of inspiration. By the way, my version of e4fsprogs was 1.41.12-3.el5. The filesystem was ext4 filesystem so I gave this a shot:

[root@prdnb0001 dev]# fsck.ext4 -n /dev/vgmd1200/optlv

…and filesystem check proceeded to run just fine. I did not have time to dig into why this worked and automatic fsck check on boot failed. Maybe someone can shed some light on this.

In any case, reading man pages always pays and once in a while you find out something cool such as:

SIGUSR1 This signal causes e2fsck to start displaying a completion bar or emitting progress information.

So by doing:

[root@prdnb0001 ~]# pkill -SIGUSR1 fsck.ext4

in another terminal window, I was able to get progress bar for the running check.

Posted on September 30, 2013 at 09:42 by somedude · Permalink · Leave a comment
In: centos, ext4, linux, linux tips, redhat, storage

Unfinished transactions in yum

I was patching this “pseudo server” the other day. Resources on the server were not monitored. Of course / was low on space and yum failed. Subsequent run of yum update threw the following message:

[root@web0001 sbin]# yum update
Loaded plugins: rhnplugin, security
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Install Process
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them.
The program yum-complete-transaction is found in the yum-utils package.

Not wanting to mess anything up at that moment, I installed yum-utils package. yum-utils contains yum-complete-transaction command. Using this tool, you can either finish incomplete transactions or you can clean up transaction files. In either case the directory of interest is /var/lib/yum.

All that was needed to be done to restart update process was to clean up transaction files:

[root@web0001 sbin]# /usr/sbin/yum-complete-transaction --cleanup-only
Loaded plugins: rhnplugin
This system is receiving updates from RHN Classic or RHN Satellite.
Cleaning up unfinished transaction journals
Cleaning up 2013-08-14.09:04.41

Posted on August 28, 2013 at 15:07 by somedude · Permalink · Leave a comment
In: centos, linux, linux tips, linux utilities, redhat