|September 18, 2005
It was simply a matter of
time. This site got hacked.
On the morning of December
20, 2004, I booted up my computer and visited DSL/Cable Webserver and I
got this message:
Maybe only the front page
got hacked? I checked another page and that was defaced too.
I check the forums....
I started to think, all in
all, I've been pretty lucky over the years. The site has has been
up for 4 years without a major hacking. Not too bad I figured.
Previously, a few commercial sites I work with had been hacked within 6
months. 4 years isn't too bad right? Still a pain in the butt
The first thing on my mind
was obviously: "Crap!".
the next thing was, "How
do I fix this thing - fast?". I did a search on Google and Google
groups for "NeverEverNoSanity" and there were ZERO results. Hmm...
nobody ever heard of this worm? That's just great.
I was rather touched to see
that I had received several dozen emails from visitors and friends to let
me know that my site had been hacked. That was cool and embarrassing
at the same time.
I sat in front of my server
and logged in. Whew!! I was afraid that the server was so compromised
that I wouldn't be able to log in at all. I noticed that the CPU
usage was running around 80-90% constantly. This is extremely unusual
since typical CPU usage runs around 3-5%. Either my server was still
being attacked or my server was still in the process of spreading the worm
and attacking other people. Either senario was not good.
I checked out the files on
my server and saw that all my files ending in .html and .php were overwritten
with the defaced message. What was really surprising was that files
OUTSIDE of the web publishing directory were affected as well. This
really made me mad since I had saved several web pages from over the years
and they were all now defaced! All the articles and pages that I
downloaded were gone. ARRGGHHHHH!!!!! There is a moral to this
I backup my website every
week so luckily I had a recent version of my site. I simply re-uploaded
all the content and everything seemed to be okay.
But I knew that since I didn't
fix the vulnerability in the site, I could get hacked again. But
I didn't know where the weakness was. I searched some more on Google
and saw 2-3 new entries about sites being attacked. Finally, some
news, but no real details on what was the real problem. So I started
to try to "fix" everything that I could think of.
I first ran the IIS lockdown
tool again to make sure that everything was closed on that end.
Then I searched on the web
and found that PHP had a recent vulnerability and that might be the problem.
So I download and installed the latest PHP engine. After restarting
the server, the CPU load was still extremely high.
The last thing I could think
of was our forum software, PHPBB. I was kind of doubtful that this
was the problem since I didn't think that a flaw in PHPBB could affect
all my files on the server. But hey, I was desperate and willing
to try anything.
You might be asking.
Why wasn't your PHPBB updated before the attack? Was I being extremely
negligent? Well, my answer is that no, I wasn't exactly being negligent,
I was being lazy. You see, I made a lot of customizations and modifications
to PHPBB. You couldn't see the mods as a forum user, but they were
for search engine indexing and admin stuff. If I were to update my
PHPBB to a newer version, I would have lost many of those mods. I
was being lazy in updating the software since I would have to reinstate
all those mods again, something I wasn't really looking forward to.
Back to our story, after
updating PHPBB to the latest version, the CPU usage dropped back down to
normal levels. Bingo. PHPBB was the culprit. Well, I
guess I was actually the culprit since PHPBB had issued a fix a few months
So aside from the personal
files I had stored on the server, I didn't lose too much data from the
At the end of the day, I
did a final search on Google and found thousands of entries for "NeverEverNoSanity".
A search on MSN found over 40,000 hits! I guess that worm really
Symantec released this page
with detailed information about the worm:
Turns out this particular worm
used Google to find websites using PHPBB and used the "PHPBB Viewtopic.PHP
PHP Script Injection Vulnerability".
By the next day all the major
anti-virus companies released virus definitions for their anti-virus software.
Google did their part as
well by blocking requests from the worm to spread. Although they
probably could have blocked the worm much earlier. They must have
noticed the strange pattern of searches. Oh well...
A few days later, there was
another variant of the worm that tried to exploit the same hole in PHPBB.
My site was targeted, but it didn't succeed. How did I know I got
hit? Well, I found it very suspicious that our forums suddenly got
500+ concurrent visitors. A quick trip to the PHPBB website confirmed
that other people were getting the same erroneous member count. Good
thing I was patched. (finally)
So what did I learn from this
incident? Here are some key points:
Well, getting hacked is never
any fun, but it keeps us on our toes and you can be sure that I am more
vigilant for having been exploited. No site is every hack-proof,
but you can take steps to make it as difficult as possible for the hackers.
Keep your software up to date.
This includes the operating system, all the software (webserver, mail,
ftp), all the scripting languages (perl, php), and scripting programs (phpbb,
Get on the mailing list for
each of the products you use. They will then let you know when a
critical update comes around.
Make consistent backups.
The reason I was able to get the site back up so quickly was because I
had a backup on hand. Back up to something permanent like a cdr or
dvdr, not to something that can be erased (hd, zip).
Don't store personal files on
your server. Assume everything on your server is public. If
you put stuff on the server, it could get damaged like my stuff was, or
it might be viewed on the internet. Either option is bad.
Use non standard ports whenever
possible, especially for FTP and Remote Desktop. Using non standard
ports wouldn't of helped my case, but for other types of attacks where
widely known port numbers are targeted, it would be good to use non standard