dacs.doc electric

Until recently, I thought I was safe and secure in my home computing environment1. Admittedly, my environment is a little out of the ordinary, having a 7x24 broadband connection and two servers running continuously (a web server and file/application server)--but I do practice "safe hex":

• No Microsoft applications connected to the Internet (Opera instead of IE, AKMail instead of Outlook, Apache web server instead of IIS, Linux instead of Windows on the server…)
• Antivirus scanning, with frequent updates
• Firewall running on the router directly behind the cable modem, with only port 80 intentionally exposed (forwarded to my web server)
• Intelligent handling of email attachments

A few weeks ago, we started experiencing difficulties with our Internet connection--typically, access to web sites and servers would be unavailable. The first reaction was to follow superstitious behavior--blame the cable system for DNS outages (not really), deep scan all computers for viruses (none found) and look for spyware (nothing). The only real pattern observed was that the outages typically followed some kind of heavy usage (often Morpheus or heavy AIM use by my son)--but these patterns weren’t really tight.

A bunch of experimentation over a few weeks showed that the problem went away if I simply rebooted my router. This solution wasn’t really satisfactory--while I could probably teach my kids how to telnet to the router and do a reboot, I didn’t think my wife would ever get the hang of it. To make matters worse, the frequency of this problem was increasing--enough to finally drive me to figure out exactly what was going on.

It turns out that one of my first thoughts was correct, viruses (or more specifically worms) appear to be at the root of the problem--however, the worms weren’t on my machine, they were on someone else’s. Before showing what I found, there’s a little background necessary to understand what was happening.

How NAT works

A router is a somewhat magical device that goes between your broadband connection (cable or DSL) and your home network. Its name comes from its main function--routing IP packets to the right destinations. Besides the routing, today’s routers have additional functionality, including acting as a firewall (filtering - blocking packets from the outside world that do (or don’t) match specified patterns). In most cases, the desired filter set is really simple--if I didn’t originate the transaction, don’t let the packet in! If you are running a server exposed to the world, things are a bit less secure--you really want to let in messages, which connect to your server (for example, a web server needs to see HTTP requests coming in on port 80).

Another basic function of the router is something known as Network Address Translation (NAT). A typical situation is where there are multiple computers on your network (each with a different IP address), all sharing the same Internet connection. The outside world doesn’t know anything about the addresses used inside your network--the whole collection is known by the IP address that your ISP assigns you. How does the response from a web server ever get back to the computer that originally wanted the page? The answer is NAT. Besides modifying the return address in the outbound packets; the router constructs a table, which contains a list of connections--this internal address just talked to that external one. When a response comes back, the source is looked up in the table and the address changed to pass the response back to the correct computer. The table is continuously being updated, with old entries being cleared based on rules including age. Sounds kind of simple, but it turns out that there’s a vulnerability that my system was falling prey to--the NAT table is a finite size (about 256 entries for my router) and there was no opportunity to modify the rules for expiring entries.

SYN-Floods

Denial of Service (DOS) attacks have been in the news periodically, as various well known web servers have succumbed (I believe, not too long ago, the New York Times web site was briefly taken offline from a DOS attack). DOS attacks are designed to muck up the communications for a server, in such a way that it becomes inaccessible. One common method for performing a DOS attack is known as SYN-flooding. When communications take place between a user and a web server, there are a few beginning steps to get things set up right: in simple terms, the user sends a message to the server saying "Hello--are you there?" (SYN) and the server responds with "Yup, talk to me" (ACK)--after which, they start interchanging information. The server knows who to respond to based on the address in the request.

In a SYN-flood, this communication is mangled--the SYN messages are modified to look like they come from random addresses (usually bogus). When the server tries to respond, there’s no one to respond to (due to the forged address). By design, the server keeps the connection open for a period of time, waiting for the response--this consumes resources on the server. Enough of these bogus messages will clog the server and effectively take it out of communication. To make matters worse, people have figured out to create worms, which will cause infected computers to launch DOS attacks on their behalf (creating the more harmful Distributed Denial of Service (DDOS) attack).

NIMDA Worm

NIMDA is a worm that emerged last fall and quickly spread around the world. Its original vector was through an infected executable email attachment, but had others built in. At the time, it was rather unique, using multiple vectors for spreading: email attachment, spreading through a LAN via file shares and compromising web servers2. Once a machine is infected, it tries to infect others through the paths above. The web server path is of considerable interest for this problem, since the infected computer will look around the network for IP connections with port 80 open--when one is found, a characteristic pattern of HTTP GET requests will follow (see box).

If the server found is on a Windows platform (and running IIS), then it will replace files and otherwise modify the system to propagate itself. Of interest is the way it looks for open ports—there’s a very strong bias for attacking machines on the same network. A NIMDA attack can’t modify a server running on a Linux platform, but can consume its resources.

What I think happened

Having uncovered all of the above information in my research, I re-examined what’s been going on, and found the following:

• Around the time that problems were experienced, the NAT table in the router was filling up (230+ entries, instead of the normal 20ish). This meant that only a few new connections would be possible – once the table filled, no more connections could happen.
• I examined the log files on my web server and found evidence of NIMDA attacks as far back as the logs went (Dec last year). Until recently, the attacks were at a low level (one every few days)--mostly from addresses outside of my broadband network. The current log (the 1st two weeks of March) showed 134 attacks, with over half coming from two addresses within the broadband domain.

Through these observations and some discussion in relevant groups, my best guess is that my outages were caused either by the frequency of the NIMDA attacks consuming router resources and possibly a low level SYN-flood attack as well (still not sure if NIMDA could really cause all this trouble).

My immediate solution was to close port 80 on the router, removing the web server from attack, which showed instant results--no more filling of the NAT table and no more outages. Unfortunately, this also means my web server is no longer visible.

The long-term solution is still being worked out--I’m upgrading my router to one that provides more flexible filtering and deeper IP packet inspection (claiming to be more secure against DOS attacks). I’m also looking into the possibility of blocking the NIMDA messages through content filtering (may not be workable). In addition, I’ve been in touch with the tech support for the cable system, providing them with a listing of the recent attacks—there’s a good chance that the folks causing most of the trouble will discover they are infected the hard way.

Footnotes

1 I’m not an expert in what I discuss in this article—most of this has been learned through web research over the last few days. Some of my conclusions might actually be incorrect (I’m still working my way through the solution space).

2 If you are on a Windows platform:

• go and check whether you have IIS running. If so and you didn’t intentionally install it, get rid of it! Most people have no need for a web server on their home machine
• but it gets installed anyways, with rather insecure settings--wide open for worm attacks.