Tuesday, September 23, 2008

Red Hat support

I've had Red Hat subscriptions for years. Basically, I've paid for access to their updates, ie. the ability to type "yum update" on a RHEL5 box, or the up2date equivalent in previous RHEL releases. US$350 / year seemed a bit steep for that, but the product is good so I paid it. We have some SLES infrastructure too, and the RHEL / RHN updates always works (SLES 10 updates break constantly), so we've stuck with them.

Anyway, yesterday for the first time I had the need to contact Red Hat support with a technical question to which I couldn't figure out the answer:

Our RHEL Xen host machine has filled up its root mount and I'm having trouble figuring out why:

[root@vodka ~]# df -H
Filesystem Size Used Avail Use% Mounted on
/dev/md0 5.1G 4.8G 33M 100% /
tmpfs 2.1G 0 2.1G 0% /dev/shm
[root@vodka ~]#

It hasn't run out of inodes:

[root@vodka ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 1280000 80028 1199972 7% /
tmpfs 497495 1 497494 1% /dev/shm
[root@vodka ~]#

And du reflects what I think the situation should be:

[root@vodka ~]# du / --max-depth=0 -h
2.1G /
[root@vodka ~]#

How is this possible? Being a Xen host, the machine has very little installed or running on it - it's just there to host 3 virtual machines. The virtual machines have their own software RAID partition each as their virtual HDD's.

The virtual machines appear to be running unaffected by the problem at the moment, but the host isn't getting updates from RHN (due to the full root mount, I'd be guessing).


Now, I'm on the "Basic" RHEL support contract, meaning 2 business day response time by web only (no telephone support), because I've never really valued the support option. Anyway, in around 24 hours I got a nice response advising me to look at the following two knowledge base articles:

http://kbase.redhat.com/faq/FAQ_35_5209.shtm
http://kbase.redhat.com/faq/FAQ_80_10743.shtm

After looking, they were bang on. Xen has this save directory that fills my disk, and somehow the file gets deleted, but Xen holds a file descriptor open to it, just like the first of those two knowledge base articles says. This is confirmed behaviour on the Xen-Users mailing list:

http://lists.xensource.com/archives/html/xen-users/2007-07/msg00510.html

Very annoying. But if you follow the instructions in that mailing list post (except the file on RHEL5 is in /etc/sysconfig/xendomains, not /etc/default/xendomains) and tell it not to save the file, then restart your xend and xendomains, the space gets freed up again.

Anyway, I'm really happy with Red Hat support. They provided me an answer in half their SLA time which helped me solve the problem. You might think you'll never use their support - for years I wondered if it was any good and worth paying for. It is, and it's worth having the RHEL subscription for.

No comments: