A Sysadmins blog

Well today ran into a nice problem, the media changer on a Dell 124T had disappeared and the backup server had been neglected for a while. So Amanda was spewing out backup failed reports for a while now. A bit of investigation later found that the media changer had “gone”. More poking showed the tape drive still present hmm.

As this media changer and server live in another state (1200km away) and it was Friday night, the chances of getting the tape drive power cycled == 0. Luckily a rescan of the correct SCSI bus resulted in the media changer being found and usable again.

For reference, check out your SCSI devices by running:

cat /proc/scsi/scsi

In my case the host bus I wanted to re-scan was 1. So running this command tells the controller to rescan all channels, IDs & LUNs..

echo "- - - " > /sys/class/scsi_host/host1/scan

After this, I was able to interact with the tape changer device via the Amanda tools again.

Note: Always be-careful when running commands that hot add/remove SCSI devices. I have seen some servers crash from this and others lose access to all SCSI devices which required a reboot to fix (Mostly it is okay though).

At work we have a number of LSI megaraid controllers & dell perc cards. Today I ran accross a system with a “Other Error Count: XX” assigned to the both drives in a raid 1 array. It took some googling but it turns out these are not ciritcal drive media errors, but some other SMART errors.

For reference here is a list of the LSI megaraid error codes and the corresponding SMART failure count (Thanks to the dell linux mailing list).

• Predictive Failure Count == Number of SMART errors.
• Media Error Count == Number of SMART errors related to the drive media.
• Other Error Count == Number of SMART errors not related to the drive.

See wikipedia for the SMART error codes.

Lately I have been playing with ESX 3.5 and ESXi getting ready for a deployment at $work. Bellow is a tip to people want access using SSH keys to the root account.

As described in RTFM Education’s ESX guide it is possible to allow root user SSH/SCP/SFTP access to your ESX 3.5 server.

Disabling Auditing on ROOT (Not Recommended)
Note:
• Some applications do not support levitation to a higher plain – for example
WinSCP. Sure you could use WinSCP to gain access as an ordinary user,
but then you might lack permission to copy the files you need. If you try
to logon as root, WinSCP will give you access denied.
• If you wish to disable the restriction on ROOT not being allowed direct
access using SSH then carry out the following task. I wouldn’t recommend
doing this as you will loose enforcement of your audit trail.
1. nano –w /etc/ssh/sshd_config
2. Locate: PermitRootLogin no
3. Place a # in front of PermitRootLogin no like so: #PermitRootLogin no
4. Exit Nano & Save the file
5. Restart sshd with service sshd restart

You just need to modify the SSH daemon configuration. The described step will allow your root account open to logins with a password. In todays internet this is less than a great idea.

Personally, I like console logins as root to some servers using SSH keys. To enable root logins that allow SSH keys but disallow passwords edit your /etc/ssh/sshd_config as follows

1. Change the line readiing “PermitRootLogin no” to “PermitRootLogin without-password“

2. Add the line “PermitEmptyPasswords no” underneath.

3. Save and exit

4. Restart SSHD “service sshd restart“

Now you will need to place your ssh public key in the file system path “/root/.ssh/authorized_keys”

For further security you can restrict what hosts the key can login from. This is done by placing “from=hostname.domain.com” in front off the public key. It must all be on the one line.

Now you can use your SSH key to authenticate as the root user.

This guide assumes you have another user that has SSH access and can “su” to root on the ESX host. There is plenty of information in google on how to do that already. It also assumes you have a SSH keypair or can figure out how to generate them.

We have a few legacy servers at $work where they still use mod_jk and mod_jk2. Unfortunately for us one of these was setup by the client’s admins who did not like to configure tomcat context mappings within the virtual host that only needed them. This has the affect of sending many more URLs to tomcat/jboss than necessary.

One way to get around this when you cant change those mappings is to use the SetEnv module.

In the virtual host where you want to exclude certain paths from being mapped, add the following entries and reload the config.

        # Disable global mod_jk mapping that break stats.
        SetEnvIf Request_URL "/stats*" no-jk
        SetEnvIf Request_URL "/awstats*" no-jk

So we have a number of Red Hat based systems where clients use PHP on them. A few clients have requested we update the timezone information on them. Sure no problem.

However “pecl install timezonedb” fails with out of memory errors at 8mb, even though the php memory limit is 32mb or even 64mb on my own machine.

A dirty work around is to install it using

pecl bundle timezonedb
cd timezonedb
phpize
./configure
make
make install

You will need to ensure you have GCC, make, phpize (from php-devel package) installed though.

Currently I have not found a fix or any bugs filled in PECL bug tracker or the Red Hat bug tracker. This is reproducible on CentOS, which is to be expected.

I will update this post if I find a way to fix it or a bug ID in any of those trackers.

Update:

Make sure you have at least these packages installed.

autoconf
automake
libtool
bison
flex

Update 2:

pecl timezonedb will only work with php5 due to no date module existing in the php4 source.

This particular problem manifested because we have php4 and php5 installed on a single server, so clients can use either version as a CGI.

So you have a client who needs to download/upload office 2007 docs via an Apache HTTP server on a Linux server?

You will probably need to add these directives to have the download work correctly. They can be added to an individual virtual host or the main configuration file:

AddType application/vnd.ms-word.document.macroEnabled.12 .docm
AddType application/vnd.openxmlformats .docx .pptx .xlsx
#AddType application/vnd.openxmlformats-officedocument.presentationml.presentation .pptx
#AddType application/vnd.openxmlformats-officedocument.spreadsheetml.sheet .xlsx
AddType application/vnd.openxmlformats-officedocument.wordprocessingml.document .docx
AddType application/vnd.ms-word.template.macroEnabled.12 .dotm
AddType application/vnd.openxmlformats-officedocument.wordprocessingml.template .dotx
AddType application/vnd.ms-powerpoint.template.macroEnabled.12 .potm
AddType application/vnd.openxmlformats-officedocument.presentationml.template .potx
AddType application/vnd.ms-powerpoint.addin.macroEnabled.12 .ppam
AddType application/vnd.ms-powerpoint.slideshow.macroEnabled.12 .ppsm
AddType application/vnd.openxmlformats-officedocument.presentationml.slideshow .ppsx
AddType application/vnd.ms-powerpoint.presentation.macroEnabled.12 .pptm
AddType application/vnd.ms-excel.addin.macroEnabled.12 .xlam
AddType application/vnd.ms-excel.sheet.binary.macroEnabled.12 .xlsb
AddType application/vnd.ms-excel.sheet.macroEnabled.12 .xlsm
AddType application/vnd.ms-excel .xlt .xla
AddType application/vnd.openxmlformats-officedocument.spreadsheetml.template .xltx
AddType application/vnd.ms-excel.template.macroEnabled.12 .xltm
AddType application/vnd.ms-xpsdocument .xps
AddType application/application/vnd.ms-powerpoint .ppt .pot .pps .ppa
AddType application/msword .doc .dot

The commented lines are there incase the shorthand pptx and xlsx mimetype does not work.

Dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.

Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).

This has to be my most favourite bit of software right about now. You can download it from http://dag.wieers.com/home-made/dstat/ or get it via your distros package manager.

It really helps you find the bottle neck in a system without having 10 terminals open.

So one of the guys was on the phone to a client, something to do with a domain registration..

“Do they they turn off the Internet on weekends?”

Yes, clearly they do. No one uses the Internet on weekends, that is why we are closed on Saturday and Sundays.