Category Archives: Technology

How to determine who is relaying mail through your Exchange server

Problem

If you have relay access control properly configured, only authenticated users should be able to relay mail through your server. If you are still seeing messages being relayed, then it’s possible a user account has been compromised. This will allow you to see which users are relaying mail so you know which account has been compromised.

Solution

Set Transport Logging to Maximum. This way the SMTP service will log a 1708 Information event which tells you which user account authenticated and which login method they used. You can use the Event Viewer to view these event log entries, filter for event ID 1708 in the Application Log.

  1.     Start Exchange System Manager.
  2.     Expand Servers, right-click Your_ Server_Name, and then click Properties.
  3.     Click the Diagnostics Logging tab, and then click MSExchangeTransport under Services.
  4.     Under Categories, click the Authentication category.
  5.     Under Logging Level, set the level to Maximum

Now the next time somebody tries to relay mail through your server, an event 1708 will be written to the Application log. The event will contain the username that was used to authenticate.

Applies To

Exchange Server 2003

Dissecting and removing the SHV5 rootkit

Yesterday I noticed some odd behavior on one of the Linux servers that I maintain. For one thing, every time I would run ‘top’ or ‘ps’, I would see the following message

Unknown Hz value! (75) Assume 100

I decided to run rkhunter to check for rootkits. Needless to say, I was disappointed to discover that I had the SHV5 Rootkit on my machine. Let me tell you, it’s not a good feeling to know you’ve been hacked. Now I know common wisdom dictates that rootkit-infected machines be formatted and reloaded from scratch but that is not something you do without a lot of planning and preparation. The first thing I wanted to do was get this bad guy off my machine and try to minimize the damage.

How it works

I did some quick googling and I discovered this page which was very helpful. However, I wanted to add a few more details that might help somebody out in the future. Basically what I discovered is that the rootkit installs an SSH server as a backdoor for the hacker to obtain access to your machine. In my case it was running on port 6522 but this is configurable by the hacker so your mileage my vary.

The hacker does not need to know any passwords to login to your system because the rootkit installs a public key file. The hacker simply needs to have the corresponding private key to authenticate to your server. Unfortunately all of this was invisible to me because the rootkit modifies several core utilities such as ‘ls’ and ‘netstat’ to hide it’s presence. The modified versions of these utilities work just like the real ones, except they don’t display anything that might be incriminating such as certain files and folders, or tell-tale processes.

Cleaning it up

The first thing you have to do before you can start cleaning up a rootkit infection is to get your original system files back. Luckily, the rootkit makes this rather easy by setting the immutable bit on the modified files. The purpose is to prevent anyone from deleting or overwriting the hacked files but it also gives us an easy way to tell what’s been modified. After you run rkhunter, there should be a log file in /var/log called rkhunter.log. Search through this log file to see what files have the immutable bit set. These are the ones we need to replace. I’ll use /bin/ls in my examples.

First, unset the immutable flag by using this command:

chattr -sia /bin/ls

Then, you can optionally make a copy of the hacked file for future use. This is helpful to compare the output from the hacked version to the output of the real version to see what’s different.

mv /bin/ls /bin/ls.hacked

Next, figure out which RPM your file belongs to by running this:

rpm -qif /bin/ls

This will tell you the source RPM. To get the binary RPM just replace ‘.src.rpm’ with ‘.i386.rpm’. For example, on my machine, the source RPM for /bin/ls is:

coreutils-5.97-19.el5.src.rpm

So the file I needed to download was:

coreutils-5.97-19.el5.i386.rpm

You can Google the name of this file or look on popular RPM sites such as rpmfind.net or rpm.pbone.net. Once you are sure you have the correct file, install it using the –force command.

rpm -i –force coreutils-5.97-19.el5.i386.rpm

This tells the installer to try to reinstall the package even though it’s already installed and to replace files as necessary. If you get any errors about files that it can’t  replace, it’s probably because those files have the immutable bit set as well. Clear the immutable bit on those files and install the package again until there are no errors.

Repeat this whole process for every file in rkhunter.log that has the immutable bit set. You will probably have to download several different packages.

Now that you have reliable system commands you can begin ripping out the rootkit. Use netstat to see what ports are open and look for anything suspicious.

netstat -ln –programs

This is where it’s handy to have the output from the hacked version of netstat so you can compare. In my case, I found a program called ttyload that was listening on port 6522. I killed the process and deleted the file. After that I confirmed that my server was no longer accessible on that port. I also used the information in rkhunter.log to locate and delete a number of other files and folders as well. Here’s a list of some of the more interesting ones.

/usr/lib/libsh/hide a script that removes traces of the hacker’s activities from the system log files.
/usr/lib/libsh/.backup looks like a backup of all the system files that were modifed. DO NOT TRUST THIS.
/usr/lib/libsh/.sniff/shsniff a packet sniffer used to capture passwords off the network.
/lib/libsh.so/sshk the public key file for SSH authentication.
/lib/libsh.so/shdcf a configuration file that determines (among other things) the port to listen on.
/usr/sbin/ttyload a script that calls /bin/ttyload and /bin/ttymon. Used at startup.

Make sure you check your /etc/inittab and all the files in /etc/rc.d to see if the rootkit is loading anything at startup. In my case I had to remove the line that calls /usr/sbin/ttyload from /etc/inittab.

Final thoughts

Here are some final thoughts on what to do next.

  1. Change the root password and the passwords for any other privileged accounts you may have.
  2. Reboot the server and make sure things are still normal after the system comes back up.
  3. Use iptables to block incoming connections on all ports other than the ones you need.
  4. Strongly consider a full format and reload of your system.
  5. Run rkhunter on a regular basis

How to backup your MySQL databases to separate files

I wanted to dump all my MySQL databases using the mysqldump utility, but I wanted each database to be in a separate file. I couldn’t find a solution online, so I wrote my own script. Here it is:

#!/bin/sh
USERNAME=admin
PASSWORD=topsecret
BACKUPDIR=/var/local/mysql-backups

for i in $(mysql -u $USERNAME -p$PASSWORD -e "SHOW DATABASES;" --skip-column-names --batch)
do
   echo "Backing up database $i"
   mysqldump -u $USERNAME -p$PASSWORD --opt $i | gzip > $BACKUPDIR/$i.sql.gz
done

Of course, you will need to edit the first two lines to reflect your own username and password. When you run this script, it will create a bunch of files in the /var/local/mysql-backups directory, one for each database. The files are compressed to save space.  Of course, you’ll need to make sure the destination directory exists.

Remove Personal Antivirus in 3 easy steps

What is Personal Antivirus?

Personal Antivirus sounds like a wonderful program that might be good to have, right? Actually, it’s just one of an increasingly common type of malware called a rogue anti-virus or rogue anti-spyware program. It appears to be harmless or even beneficial on the surface, but it’s actual goal is to scare you into thinking you have some nasty virus that can only be removed by purchasing their removal software. It’s sort of like digital blackmail. Similar “products” include XP Antivirus, Antivirus 2008/2009 , Antivirus 360, etc…

How do I get rid of it?

Unlike most legitimate programs, there is no automatic un-installation method. That’s because the people behind these scams don’t want you to remove their software. They usually don’t show up in the list of programs in the Add/Remove Programs control panel applet. However, in many cases, manual removal is simple. Here are the steps I use. Keep in mind there are many different variants of this program so your mileage may vary.

Step 1: Terminate the application

Open Task Manager and look for a process called “pav.exe”. Highlight this process and click End Task. Once the process has been ended, the tray icon and application window should disappear.

Step 2: Prevent the program from starting up

Download a program called HijackThis. After you’ve installed it, run HijackThis and do a system scan. Look for any reference to the file pav.exe. Usually there will be at least one reference on a line that starts with “Run”. Put a check beside any that you find and click “Fix”

Step 3: Delete the program files

Delete the Personal Antivirus folder located in C:\Program Files.  You should also delete the shortcut on the desktop and the Personal Antivirus folder in the start menu.

That’s it. Reboot your computer just to make sure it doesn’t come back at startup. One optional step is to run a registry cleaner like CCleaner after you’re done. That should remove any leftover registry keys that reference the missing program file folder.

So do I really have a virus?

Just because Personal Antivirus was on your system and it said you have a virus, that does not mean you’re actually infected. However, I strongly recommend doing a full system scan with a legitimate anti-virus program and an anti-spyware program just to be safe. If you don’t already have one, here’s a list of some free alternatives.

Free Anti-Virus software (install only one at a time)
AVG
Avast
Avira AntiVir

Free Anti-Spyware software
Spybot Search & Destroy
AdAware

Blackberry purgatory

Blackberry BoldBlackberries. You gotta love em. They’re great when they work but sometimes it’s a challenge to get them setup properly. I must have taken at least three phone calls on Friday from clients who needed help getting their Blackberries setup with their corporate email accounts. Two of them were using BES (Blackberry Enterprise Server) and one was using BIS (Blackberry Internet Service).

Blackberry Enterprise Server
If you’re lucky enough to have BES, setting up a new Blackberry is usually a piece of cake. All you have to do is add the user to the Blackberry Enterprise Manager, set an activation password and activate. I usually choose to do a wireless activation because it’s easier to do over the phone. Once you’ve created the password, tell the user to go to Options…Advanced Options…Enterprise Activation. Give them the password that you’ve setup for them and hit Activate. That’s it. Here’s an overview of how the process works.

Here’s a couple of things to watch out for:

1. Make sure they have a good signal. If their signal is spotty, it might take an excessively long time or it might not work at all. The amount of time it takes depends on how good their signal is and how much data they’re trying to sync. I’ve seen it take anywhere from 5-20 minutes.

2. Make sure they have the correct data plan.Phone carriers often charge more for a BES account than they do for a BIS account. If the user doesn’t have the higher-priced BES data plan, the carrier may not allow you to do an enterprise activation. Don’t make the same mistake I did. Call the carrier to confirm the data plan before you spend 2 hours trying to troubleshoot it on your own.

3. Not-so-helpful error messages. If the activation fails with “An error has occurred. Please contact your system administrator” there might be a problem with the password. Reset the password and have them try again. It takes awhile to get used to the Blackberry keyboard so choose a simple password that is easy to type on a small keyboard (e.g. all lower case, no numbers or symbols). This breaks all the normal rules of a good password, but it’s only a one-time activation password. After that, it’s never used again. You might also want to try wiping the handheld and starting fresh.

Blackberry Internet Service
So your company wouldn’t spring for the Blackberry Enterprise Software? Not to worry. Your carrier provides you with something called Blackberry Internet Service (BIS). It’s a web-based application that allows you to setup an email account and control how messages are delivered. It’s actually a service provided by RIM, but each carrier has a different URL for accesing it (for branding of course). Here’s a couple of examples

Verizon: https://bis.na.blackberry.com/html?brand=vzw
Sprint: https://bis.na.blackberry.com/html?brand=sprint
AT&T: http://bis.na.blackberry.com/html?brand=mycingular
Centennial: http://centennial.blackberry.com/

If you can’t find your carrier, go to Google and type “BIS” and your carrier’s name in the search box. It will probably be one of the first hits that comes up.

If you’ve never signed up, click “Create New Account” You will be prompted for your phone’s PIN number and ESN number. Both can be found on the Options…Status screen. Once you’ve created an account on the BIS site, you will be able to create a Blackberry email address for your device. This will be something like joe123@vzw.blackberry.net. It doesn’t really matter what the address is because once it’s set up, you simply forward your corporate email account to the Blackberry account. So it doesn’t matter if you’re using Exchange, Lotus Notes, Groupwise or some other random mail server. As long as you can forward mail, you’re in business. While you’re still logged in to the BIS website, make sure you edit the properties of your mail account and set the “reply-to” address to be your corporate email address. That way when you send a message from your Blackberry, the recipient will see a recognizable address rather than your generic Blackberry address.

That’s it. Note that BIS is also capable of downloading your mail directly from your corporate mail server using POP3 or IMAP. However, that can be tricky to get working and your mileage may vary. There is a wizard on the BIS website that attempts to auto-detect your mail server settings, but it doesn’t always work and you may have to specify the settings manually. I find it’s much simpler and easier to just forward messages to the Blackberry account. Have fun!

Synchronize files between multiple computers without breaking a sweat

If you have more than one computer that you use on a regular basis, it seems like the file you need is always on the other computer. Sure, you can use a flash drive but if that’s still too much work, you need Dropbox! I use it all the time to sync files between my work computer, laptop and home computer. Check it out, it’s free!

http://www.getdropbox.com/

Free Online Backup…with a buddy.

One of the questions I get asked the most is “How do I backup my PC?” Of couse, there’s a million ways to backup data but few are easy, reliable, or cheap. This might be the answer to all three of those problems. All you need is a high-speed Internet connection, and a friend.

http://www.cucku.com/

Virtualization is coming to a desktop near you

Everybody’s heard about server virtualization and what it can do to make your server room easier to manage. Now desktop virtualization is poised to become the next holy grail. Check out this video of Citrix’s new Xen Based Client Hypervisor.

YouTube Preview Image