Problem

If you have relay access control properly configured, only authenticated users should be able to relay mail through your server. If you are still seeing messages being relayed, then it’s possible a user account has been compromised. This will allow you to see which users are relaying mail so you know which account has been compromised.

Solution

Set Transport Logging to Maximum. This way the SMTP service will log a 1708 Information event which tells you which user account authenticated and which login method they used. You can use the Event Viewer to view these event log entries, filter for event ID 1708 in the Application Log.

1.     Start Exchange System Manager.
2.     Expand Servers, right-click Your_ Server_Name, and then click Properties.
3.     Click the Diagnostics Logging tab, and then click MSExchangeTransport under Services.
4.     Under Categories, click the Authentication category.
5.     Under Logging Level, set the level to Maximum

Now the next time somebody tries to relay mail through your server, an event 1708 will be written to the Application log. The event will contain the username that was used to authenticate.

Applies To

Exchange Server 2003

Yesterday I noticed some odd behavior on one of the Linux servers that I maintain. For one thing, every time I would run ‘top’ or ‘ps’, I would see the following message

Unknown Hz value! (75) Assume 100

I decided to run rkhunter to check for rootkits. Needless to say, I was disappointed to discover that I had the SHV5 Rootkit on my machine. Let me tell you, it’s not a good feeling to know you’ve been hacked. Now I know common wisdom dictates that rootkit-infected machines be formatted and reloaded from scratch but that is not something you do without a lot of planning and preparation. The first thing I wanted to do was get this bad guy off my machine and try to minimize the damage.

How it works

I did some quick googling and I discovered this page which was very helpful. However, I wanted to add a few more details that might help somebody out in the future. Basically what I discovered is that the rootkit installs an SSH server as a backdoor for the hacker to obtain access to your machine. In my case it was running on port 6522 but this is configurable by the hacker so your mileage my vary.

The hacker does not need to know any passwords to login to your system because the rootkit installs a public key file. The hacker simply needs to have the corresponding private key to authenticate to your server. Unfortunately all of this was invisible to me because the rootkit modifies several core utilities such as ‘ls’ and ‘netstat’ to hide it’s presence. The modified versions of these utilities work just like the real ones, except they don’t display anything that might be incriminating such as certain files and folders, or tell-tale processes.

Cleaning it up

The first thing you have to do before you can start cleaning up a rootkit infection is to get your original system files back. Luckily, the rootkit makes this rather easy by setting the immutable bit on the modified files. The purpose is to prevent anyone from deleting or overwriting the hacked files but it also gives us an easy way to tell what’s been modified. After you run rkhunter, there should be a log file in /var/log called rkhunter.log. Search through this log file to see what files have the immutable bit set. These are the ones we need to replace. I’ll use /bin/ls in my examples.

First, unset the immutable flag by using this command:

chattr -sia /bin/ls

Then, you can optionally make a copy of the hacked file for future use. This is helpful to compare the output from the hacked version to the output of the real version to see what’s different.

mv /bin/ls /bin/ls.hacked

Next, figure out which RPM your file belongs to by running this:

rpm -qif /bin/ls

This will tell you the source RPM. To get the binary RPM just replace ‘.src.rpm’ with ‘.i386.rpm’. For example, on my machine, the source RPM for /bin/ls is:

coreutils-5.97-19.el5.src.rpm

So the file I needed to download was:

coreutils-5.97-19.el5.i386.rpm

You can Google the name of this file or look on popular RPM sites such as rpmfind.net or rpm.pbone.net. Once you are sure you have the correct file, install it using the –force command.

rpm -i –force coreutils-5.97-19.el5.i386.rpm

This tells the installer to try to reinstall the package even though it’s already installed and to replace files as necessary. If you get any errors about files that it can’t  replace, it’s probably because those files have the immutable bit set as well. Clear the immutable bit on those files and install the package again until there are no errors.

Repeat this whole process for every file in rkhunter.log that has the immutable bit set. You will probably have to download several different packages.

Now that you have reliable system commands you can begin ripping out the rootkit. Use netstat to see what ports are open and look for anything suspicious.

netstat -ln –programs

This is where it’s handy to have the output from the hacked version of netstat so you can compare. In my case, I found a program called ttyload that was listening on port 6522. I killed the process and deleted the file. After that I confirmed that my server was no longer accessible on that port. I also used the information in rkhunter.log to locate and delete a number of other files and folders as well. Here’s a list of some of the more interesting ones.

Make sure you check your /etc/inittab and all the files in /etc/rc.d to see if the rootkit is loading anything at startup. In my case I had to remove the line that calls /usr/sbin/ttyload from /etc/inittab.

Final thoughts

Here are some final thoughts on what to do next.

1. Change the root password and the passwords for any other privileged accounts you may have.
2. Reboot the server and make sure things are still normal after the system comes back up.
3. Use iptables to block incoming connections on all ports other than the ones you need.
4. Strongly consider a full format and reload of your system.
5. Run rkhunter on a regular basis

I wanted to dump all my MySQL databases using the mysqldump utility, but I wanted each database to be in a separate file. I couldn’t find a solution online, so I wrote my own script. Here it is:

#!/bin/sh
USERNAME=admin
PASSWORD=topsecret
BACKUPDIR=/var/local/mysql-backups

for i in $(mysql -u $USERNAME -p$PASSWORD -e "SHOW DATABASES;" --skip-column-names --batch)
do
   echo "Backing up database $i"
   mysqldump -u $USERNAME -p$PASSWORD --opt $i | gzip > $BACKUPDIR/$i.sql.gz
done

Of course, you will need to edit the first two lines to reflect your own username and password. When you run this script, it will create a bunch of files in the /var/local/mysql-backups directory, one for each database. The files are compressed to save space.  Of course, you’ll need to make sure the destination directory exists.

What is Personal Antivirus?

Personal Antivirus sounds like a wonderful program that might be good to have, right? Actually, it’s just one of an increasingly common type of malware called a rogue anti-virus or rogue anti-spyware program. It appears to be harmless or even beneficial on the surface, but it’s actual goal is to scare you into thinking you have some nasty virus that can only be removed by purchasing their removal software. It’s sort of like digital blackmail. Similar “products” include XP Antivirus, Antivirus 2008/2009 , Antivirus 360, etc…

How do I get rid of it?

Unlike most legitimate programs, there is no automatic un-installation method. That’s because the people behind these scams don’t want you to remove their software. They usually don’t show up in the list of programs in the Add/Remove Programs control panel applet. However, in many cases, manual removal is simple. Here are the steps I use. Keep in mind there are many different variants of this program so your mileage may vary.

Step 1: Terminate the application

Open Task Manager and look for a process called “pav.exe”. Highlight this process and click End Task. Once the process has been ended, the tray icon and application window should disappear.

Step 2: Prevent the program from starting up

Download a program called HijackThis. After you’ve installed it, run HijackThis and do a system scan. Look for any reference to the file pav.exe. Usually there will be at least one reference on a line that starts with “Run”. Put a check beside any that you find and click “Fix”

Step 3: Delete the program files

Delete the Personal Antivirus folder located in C:\Program Files.  You should also delete the shortcut on the desktop and the Personal Antivirus folder in the start menu.

That’s it. Reboot your computer just to make sure it doesn’t come back at startup. One optional step is to run a registry cleaner like CCleaner after you’re done. That should remove any leftover registry keys that reference the missing program file folder.

So do I really have a virus?

Just because Personal Antivirus was on your system and it said you have a virus, that does not mean you’re actually infected. However, I strongly recommend doing a full system scan with a legitimate anti-virus program and an anti-spyware program just to be safe. If you don’t already have one, here’s a list of some free alternatives.

Free Anti-Virus software (install only one at a time)
AVG
Avast
Avira AntiVir

Free Anti-Spyware software
Spybot Search & Destroy
AdAware

Blackberries. You gotta love em. They’re great when they work but sometimes it’s a challenge to get them setup properly. I must have taken at least three phone calls on Friday from clients who needed help getting their Blackberries setup with their corporate email accounts. Two of them were using BES (Blackberry Enterprise Server) and one was using BIS (Blackberry Internet Service).

Blackberry Enterprise Server
If you’re lucky enough to have BES, setting up a new Blackberry is usually a piece of cake. All you have to do is add the user to the Blackberry Enterprise Manager, set an activation password and activate. I usually choose to do a wireless activation because it’s easier to do over the phone. Once you’ve created the password, tell the user to go to Options…Advanced Options…Enterprise Activation. Give them the password that you’ve setup for them and hit Activate. That’s it. Here’s an overview of how the process works.

Here’s a couple of things to watch out for:

1. Make sure they have a good signal. If their signal is spotty, it might take an excessively long time or it might not work at all. The amount of time it takes depends on how good their signal is and how much data they’re trying to sync. I’ve seen it take anywhere from 5-20 minutes.

2. Make sure they have the correct data plan.Phone carriers often charge more for a BES account than they do for a BIS account. If the user doesn’t have the higher-priced BES data plan, the carrier may not allow you to do an enterprise activation. Don’t make the same mistake I did. Call the carrier to confirm the data plan before you spend 2 hours trying to troubleshoot it on your own.

3. Not-so-helpful error messages. If the activation fails with “An error has occurred. Please contact your system administrator” there might be a problem with the password. Reset the password and have them try again. It takes awhile to get used to the Blackberry keyboard so choose a simple password that is easy to type on a small keyboard (e.g. all lower case, no numbers or symbols). This breaks all the normal rules of a good password, but it’s only a one-time activation password. After that, it’s never used again. You might also want to try wiping the handheld and starting fresh.

Blackberry Internet Service
So your company wouldn’t spring for the Blackberry Enterprise Software? Not to worry. Your carrier provides you with something called Blackberry Internet Service (BIS). It’s a web-based application that allows you to setup an email account and control how messages are delivered. It’s actually a service provided by RIM, but each carrier has a different URL for accesing it (for branding of course). Here’s a couple of examples

Verizon: https://bis.na.blackberry.com/html?brand=vzw
Sprint: https://bis.na.blackberry.com/html?brand=sprint
AT&T: http://bis.na.blackberry.com/html?brand=mycingular
Centennial: http://centennial.blackberry.com/

If you can’t find your carrier, go to Google and type “BIS” and your carrier’s name in the search box. It will probably be one of the first hits that comes up.

If you’ve never signed up, click “Create New Account” You will be prompted for your phone’s PIN number and ESN number. Both can be found on the Options…Status screen. Once you’ve created an account on the BIS site, you will be able to create a Blackberry email address for your device. This will be something like joe123@vzw.blackberry.net. It doesn’t really matter what the address is because once it’s set up, you simply forward your corporate email account to the Blackberry account. So it doesn’t matter if you’re using Exchange, Lotus Notes, Groupwise or some other random mail server. As long as you can forward mail, you’re in business. While you’re still logged in to the BIS website, make sure you edit the properties of your mail account and set the “reply-to” address to be your corporate email address. That way when you send a message from your Blackberry, the recipient will see a recognizable address rather than your generic Blackberry address.

That’s it. Note that BIS is also capable of downloading your mail directly from your corporate mail server using POP3 or IMAP. However, that can be tricky to get working and your mileage may vary. There is a wizard on the BIS website that attempts to auto-detect your mail server settings, but it doesn’t always work and you may have to specify the settings manually. I find it’s much simpler and easier to just forward messages to the Blackberry account. Have fun!

Many things in your computer depend on having the correct date & time. Fortunately, there are network time servers all over the Internet that you can use to keep your computer’s clock in perfect sync. The trick is knowing which time server to use and how to configure the setting. The Network Time Protocol, or NTP, is the protocol your computer uses to communicate with other time servers across the network. You may have an NTP server on your local network but if not, you can use one of the many NTP servers on the Internet.

Step 1
First, open your date & time properties and make sure you are on the correct time zone. The NTP protocol will adjust your computer’s time, but not the time zone. You have to enter that manually. Also, if your location observes daylight savings time, make sure you check the box as appropriate.

Step 2
If you have Windows XP and are NOT part of a domain you will see an “Internet Time” tab. This is where you set the NTP server you want to sync with. The default time server, time.windows.com, is notoriously unreliable. Perhaps because every Windows PC built in the last 7 or 8 years is set to synchronize with that server. Instead, the time server I use is  north-america.pool.ntp.org. This is not really a time server at all. In fact it is a whole pool of time servers in North America that have agreed to let people use them. If you are outside North America, you can find a pool in your region by visiting http://support.ntp.org/bin/view/Servers/NTPPoolServers

Step 3
Now all you have to do is click the “Update Now” button to test it. You should see a message that the time has been successfully synchronized. If not, check that you entered the correct server name and try again. Also, make sure your clock is set to the correct date. If your time if off by more than 15 hours, it won’t work. Once you have it configured correctly, your clock will be synchronized automatically with a time server once a week.

Synchronizing Time on a Domain
If you don’t see an Internet Time tab, your computer might be part of an Active Directory domain. In that case, your computer will receive it’s time from a domain controller. In that case, all you need to do is make sure your domain controller has the correct time. To do this, log on to your domain controller as an administrative user and execute the following commands:

For Windows 2003 Server

w32tm /config /manualpeerlist:north-america.pool.ntp.org,0×8 /syncfromflags:MANUAL
w32tm /config /update
net stop w32time
net start w32time
w32tm /resync

For Windows 2000 Server

net time /setsntp:north-america.pool.ntp.org
net time /querysntp
net stop w32time
net start w32time
w32tm -s

Once your server has obtained the correct time, the rest of your client PC’s will follow suit.