How a Simple Admin Error can Get Your Mail Server Hijacked

A few weeks ago I found myself spending more and more time working on my email server and mail customers than any other activity during some very long days at the office. Someone was using my mail server to send thousands of SPAM mails every day and that nothing I did to stop this happening made any difference. All we could do was to monitor the out mail queue and as soon as a batch of spam mail arrived, we would manually delete them!

You have no idea how tiring this exercise was, because when we were asleep thousands of SPAM mails would go out and by the time we woke up the DNS Blacklists had already got us marked as spammers and our normal customer’s mail started to bounce!

The Story

For my sins, I do the 2nd level support for our trusty old mail Ability Mail Server (by Code-Crafters). The thing for the most part runs on diesel. It almost never needs tending or maintenance and I’m pretty sure that if you opened the server case there would be dust and spider webs aplenty in there.

First sign of trouble came 6 weeks ago when a customer called me and said that for some reason her mails were being delayed by about 4 hours. But she was not too worried because the mail was still getting delivered in the end. So there was no problem. Right!
Next my wife complained that our accounts we not getting thru to one of our major customers. But then she always complains ... so there’s no problem. Right?

The following day I got LOTS of customer calls, mostly to do with delayed mail. When I checked the server all seemed ok. The only real issue was that there were about 40 odd emails in the Out Mail queue. The queue cleared itself over the next hour.

Real Trouble

Then the real trouble started. Customers who were sending mail to large corporations and banks were getting their mail refused and marked as SPAM.
I asked each customer I spoke to send me a copy of their bounced mails. This is the first step in all investigations of this nature. The most important rule is READ ALL the mail bounce messages VERY CAREFULLY.

Each of the mail failure messages they sent me had a subject similar to:
• Undeliverable Mail (Warning Only) – this one may still get sent
• Undeliverable Mail – this one will never get sent, the mail server has given up and will delete the main in question from the Out Mail queue.

Usually computers tell us what is wrong but most of the time we don’t read the message carefully enough and we spend extra hours looking for problems already explained in the message!

When I CAREFULLY read thru each bounce message these are some of the clues I found:
1) This is only a warning, please do not resend!
The following mail has not yet been delivered and has been in our queue for 139 minutes. We will continue trying to deliver the mail for a total of 300 minutes before it is failed.

2) Rejected with: 554 C300-1.imcf.co.za

3) Rejected with: 554 Your access to this mail system has been rejected due to the sending MTA's poor reputation
4) Rejected with: 554

The messages with the 554 and Poor Reputation really got me worried. This meant that we were believed by some mail systems to be SPAMMERS!

How was this possible?

Playing Russian Roulette with an IP Revolver

Based on the above messages I worked out that the problem was that SPAM sent from our mail server was getting our IP blacklisted and thus was stopping all our legit mail as well.

In my complete ignorance I asked my administrator to call some of them to try to get our mail IP white-listed. It only took me a few hours to work out how dumb this option was. The task of finding the right person in a bank’s IT department, convincing them you are legit and then getting them to fix the problem takes days, not hours.
In fact we were already on too many black lists to make removal by asking politely a practical route.

Looking thru our mail server setup I worked out that we could split our mail send and mail receive functions over separate IP addresses. So by specifying a different IP address for sending mail, we could get around the blacklisting and not affect any of the mail delivered for users on our server.

For the next few weeks every time we missed a bunch of SPAM and did not delete them in time, we would have change our sending IP to avoid the blacklists. This was obviously not a long term solution. At one stage, we were chewing thru an IP address every 36 hours.

What Next?

We now had a semi stable solution that allowed us to investigate any holes in our mail security. This we did – for hours and hours – each time believing that we had it licked. Only to wake up the next morning to find thousands of "enlarge you know what" messages in our out queue.

Stumped and very down, I went back and following my own advice and read and re-read the error message emails. And guess what? I found a pattern! What was happening was that only certain mailboxes were being used.

OK I thought, got ‘em now! As a temporary measure I disabled the mail boxes in question, patted myself of the back and went out for a cappuccino and a muffin to celebrate. That night I slept like a baby and woke the next morning confident that when I checked my out mail queue, it would be empty.

Well ... so much for thought. The queue was a full as ever. There were over 1000 mails waiting to go, my latest IP address blacklisted was already and my day was ruined.

After getting all the now familiar damage limitation steps completed, I again started to think about HOW was this happening?

Once again I went and looked thru all the mail items I had just deleted expecting to find the mail addresses I had deleted re-occurring. But that was not the case. There were new addresses being used and they were also mailboxes from domains on my mail sever.

My immediate thoughts were that someone had access to my server and was actually hacking in to get the mail box details. But no it turned out that there was a much simpler way.

On careful examination of the sending from mail addresses, I noticed (finally!) that each mailbox being used to send the SPAM was an admin or info type mailbox. They had familiar names like admin@domain1.com , sales@domain2.co.za and info@domain3.co.za . Then the penny finally dropped.

How It Happens

What happens when all these administrators set up mail boxes for their domains? Firstly they type in all the info for each mailbox and then they get the login info to their users in some way. And what do they do about making up mail passwords?. (Remember that most mail admin interfaces DONT yet force password policies on mailboxes).

I suspect that in many cases the administrators save hassles for themselves by using the USER NAME as the password. Thus for admin@domain1.com the password would be ... admin! And so on for all the rest of the standard mail box names like sales, info, support etc.
If you set out to be a mutant ninja spammer hacker, you will quickly find out that an endless supply of legitimate mailboxes comes very much in handy. So what you do is to get a programmer to write you a system that takes a list of domain names and one by one test each possible standard mail box on the domain using a standard password.

The system then makes a list of mailboxes that it gained access to in this manner. The spammer then sets his mail engine to use this mailbox name and password combination to send spam mails. As each hijacked mailbox gets blocked the spam engine just switches over to a fresh newly hijacked one and goes again.

The Lesson

Prevention is so simple. Make sure that any mailboxes that you administer or setup do not use passwords that echo the mailbox name. And even better use some form of strong password to be sure.

Good luck!

Syncrony Web Design and Software Development South Africa
Syncrony Web Design and Software Development South Africa