Unknown Hz value! (75) Assume 100

I decided to run rkhunter to check for rootkits. Needless to say, I was disappointed to discover that I had the SHV5 Rootkit on my machine. Let me tell you, it’s not a good feeling to know you’ve been hacked. Now I know common wisdom dictates that rootkit-infected machines be formatted and reloaded from scratch but that is not something you do without a lot of planning and preparation. The first thing I wanted to do was get this bad guy off my machine and try to minimize the damage.

How it works

I did some quick googling and I discovered this page which was very helpful. However, I wanted to add a few more details that might help somebody out in the future. Basically what I discovered is that the rootkit installs an SSH server as a backdoor for the hacker to obtain access to your machine. In my case it was running on port 6522 but this is configurable by the hacker so your mileage my vary.

The hacker does not need to know any passwords to login to your system because the rootkit installs a public key file. The hacker simply needs to have the corresponding private key to authenticate to your server. Unfortunately all of this was invisible to me because the rootkit modifies several core utilities such as ‘ls’ and ‘netstat’ to hide it’s presence. The modified versions of these utilities work just like the real ones, except they don’t display anything that might be incriminating such as certain files and folders, or tell-tale processes.

Cleaning it up

The first thing you have to do before you can start cleaning up a rootkit infection is to get your original system files back. Luckily, the rootkit makes this rather easy by setting the immutable bit on the modified files. The purpose is to prevent anyone from deleting or overwriting the hacked files but it also gives us an easy way to tell what’s been modified. After you run rkhunter, there should be a log file in /var/log called rkhunter.log. Search through this log file to see what files have the immutable bit set. These are the ones we need to replace. I’ll use /bin/ls in my examples.

First, unset the immutable flag by using this command:

chattr -sia /bin/ls

Then, you can optionally make a copy of the hacked file for future use. This is helpful to compare the output from the hacked version to the output of the real version to see what’s different.

mv /bin/ls /bin/ls.hacked

Next, figure out which RPM your file belongs to by running this:

rpm -qif /bin/ls

This will tell you the source RPM. To get the binary RPM just replace ‘.src.rpm’ with ‘.i386.rpm’. For example, on my machine, the source RPM for /bin/ls is:

coreutils-5.97-19.el5.src.rpm

So the file I needed to download was:

coreutils-5.97-19.el5.i386.rpm

You can Google the name of this file or look on popular RPM sites such as rpmfind.net or rpm.pbone.net. Once you are sure you have the correct file, install it using the –force command.

rpm -i –force coreutils-5.97-19.el5.i386.rpm

This tells the installer to try to reinstall the package even though it’s already installed and to replace files as necessary. If you get any errors about files that it can’t  replace, it’s probably because those files have the immutable bit set as well. Clear the immutable bit on those files and install the package again until there are no errors.

Repeat this whole process for every file in rkhunter.log that has the immutable bit set. You will probably have to download several different packages.

Now that you have reliable system commands you can begin ripping out the rootkit. Use netstat to see what ports are open and look for anything suspicious.

netstat -ln –programs

This is where it’s handy to have the output from the hacked version of netstat so you can compare. In my case, I found a program called ttyload that was listening on port 6522. I killed the process and deleted the file. After that I confirmed that my server was no longer accessible on that port. I also used the information in rkhunter.log to locate and delete a number of other files and folders as well. Here’s a list of some of the more interesting ones.

Make sure you check your /etc/inittab and all the files in /etc/rc.d to see if the rootkit is loading anything at startup. In my case I had to remove the line that calls /usr/sbin/ttyload from /etc/inittab.

Final thoughts

Here are some final thoughts on what to do next.

1. Change the root password and the passwords for any other privileged accounts you may have.
2. Reboot the server and make sure things are still normal after the system comes back up.
3. Use iptables to block incoming connections on all ports other than the ones you need.
4. Strongly consider a full format and reload of your system.
5. Run rkhunter on a regular basis

#!/bin/sh
USERNAME=admin
PASSWORD=topsecret
BACKUPDIR=/var/local/mysql-backups

for i in $(mysql -u $USERNAME -p$PASSWORD -e "SHOW DATABASES;" --skip-column-names --batch)
do
   echo "Backing up database $i"
   mysqldump -u $USERNAME -p$PASSWORD --opt $i | gzip > $BACKUPDIR/$i.sql.gz
done

Of course, you will need to edit the first two lines to reflect your own username and password. When you run this script, it will create a bunch of files in the /var/local/mysql-backups directory, one for each database. The files are compressed to save space.  Of course, you’ll need to make sure the destination directory exists.

Personal Antivirus sounds like a wonderful program that might be good to have, right? Actually, it’s just one of an increasingly common type of malware called a rogue anti-virus or rogue anti-spyware program. It appears to be harmless or even beneficial on the surface, but it’s actual goal is to scare you into thinking you have some nasty virus that can only be removed by purchasing their removal software. It’s sort of like digital blackmail. Similar “products” include XP Antivirus, Antivirus 2008/2009 , Antivirus 360, etc…

How do I get rid of it?

Unlike most legitimate programs, there is no automatic un-installation method. That’s because the people behind these scams don’t want you to remove their software. They usually don’t show up in the list of programs in the Add/Remove Programs control panel applet. However, in many cases, manual removal is simple. Here are the steps I use. Keep in mind there are many different variants of this program so your mileage may vary.

Step 1: Terminate the application

Open Task Manager and look for a process called “pav.exe”. Highlight this process and click End Task. Once the process has been ended, the tray icon and application window should disappear.

Step 2: Prevent the program from starting up

Download a program called HijackThis. After you’ve installed it, run HijackThis and do a system scan. Look for any reference to the file pav.exe. Usually there will be at least one reference on a line that starts with “Run”. Put a check beside any that you find and click “Fix”

Step 3: Delete the program files

Delete the Personal Antivirus folder located in C:\Program Files.  You should also delete the shortcut on the desktop and the Personal Antivirus folder in the start menu.

That’s it. Reboot your computer just to make sure it doesn’t come back at startup. One optional step is to run a registry cleaner like CCleaner after you’re done. That should remove any leftover registry keys that reference the missing program file folder.

So do I really have a virus?

Just because Personal Antivirus was on your system and it said you have a virus, that does not mean you’re actually infected. However, I strongly recommend doing a full system scan with a legitimate anti-virus program and an anti-spyware program just to be safe. If you don’t already have one, here’s a list of some free alternatives.

Free Anti-Virus software (install only one at a time)
AVG
Avast
Avira AntiVir

Free Anti-Spyware software
Spybot Search & Destroy
AdAware

Blackberry Enterprise Server
If you’re lucky enough to have BES, setting up a new Blackberry is usually a piece of cake. All you have to do is add the user to the Blackberry Enterprise Manager, set an activation password and activate. I usually choose to do a wireless activation because it’s easier to do over the phone. Once you’ve created the password, tell the user to go to Options…Advanced Options…Enterprise Activation. Give them the password that you’ve setup for them and hit Activate. That’s it. Here’s an overview of how the process works.

Here’s a couple of things to watch out for:

1. Make sure they have a good signal. If their signal is spotty, it might take an excessively long time or it might not work at all. The amount of time it takes depends on how good their signal is and how much data they’re trying to sync. I’ve seen it take anywhere from 5-20 minutes.

2. Make sure they have the correct data plan.Phone carriers often charge more for a BES account than they do for a BIS account. If the user doesn’t have the higher-priced BES data plan, the carrier may not allow you to do an enterprise activation. Don’t make the same mistake I did. Call the carrier to confirm the data plan before you spend 2 hours trying to troubleshoot it on your own.

3. Not-so-helpful error messages. If the activation fails with “An error has occurred. Please contact your system administrator” there might be a problem with the password. Reset the password and have them try again. It takes awhile to get used to the Blackberry keyboard so choose a simple password that is easy to type on a small keyboard (e.g. all lower case, no numbers or symbols). This breaks all the normal rules of a good password, but it’s only a one-time activation password. After that, it’s never used again. You might also want to try wiping the handheld and starting fresh.

Blackberry Internet Service
So your company wouldn’t spring for the Blackberry Enterprise Software? Not to worry. Your carrier provides you with something called Blackberry Internet Service (BIS). It’s a web-based application that allows you to setup an email account and control how messages are delivered. It’s actually a service provided by RIM, but each carrier has a different URL for accesing it (for branding of course). Here’s a couple of examples

Verizon: https://bis.na.blackberry.com/html?brand=vzw
Sprint: https://bis.na.blackberry.com/html?brand=sprint
AT&T: http://bis.na.blackberry.com/html?brand=mycingular
Centennial: http://centennial.blackberry.com/

If you can’t find your carrier, go to Google and type “BIS” and your carrier’s name in the search box. It will probably be one of the first hits that comes up.

If you’ve never signed up, click “Create New Account” You will be prompted for your phone’s PIN number and ESN number. Both can be found on the Options…Status screen. Once you’ve created an account on the BIS site, you will be able to create a Blackberry email address for your device. This will be something like joe123@vzw.blackberry.net. It doesn’t really matter what the address is because once it’s set up, you simply forward your corporate email account to the Blackberry account. So it doesn’t matter if you’re using Exchange, Lotus Notes, Groupwise or some other random mail server. As long as you can forward mail, you’re in business. While you’re still logged in to the BIS website, make sure you edit the properties of your mail account and set the “reply-to” address to be your corporate email address. That way when you send a message from your Blackberry, the recipient will see a recognizable address rather than your generic Blackberry address.

That’s it. Note that BIS is also capable of downloading your mail directly from your corporate mail server using POP3 or IMAP. However, that can be tricky to get working and your mileage may vary. There is a wizard on the BIS website that attempts to auto-detect your mail server settings, but it doesn’t always work and you may have to specify the settings manually. I find it’s much simpler and easier to just forward messages to the Blackberry account. Have fun!

http://www.getdropbox.com/

http://www.cucku.com/