The program lpr, which is the program used for printing files, is a classic example. lpr runs as root so it can copy files to a secure spooling directory so users can't see other user's files. lpr has a -s option that would make a symbolic link to the file to be printed rather then copy it. This was useful when disk space was slow and/or scarce. However, this option lead to an exploit. Here is the original description the original exploit.
The buffer overflow he exploited was something like this...
char line[512] ;
line[0] = 10 ;
gets(line) ; # should use fgets() here
The problem here is the gets() function. gets() does not stop reading data when it's buffer is full, like fgets() does. So, by continuing to feed the function data, or in Morris' case feeding data to fingerd, one can begin writing executable code into areas of the program memory. Then, when the program goes to run code from memory, it is actually running your code. Since programs like sendmail and fingerd are commonly run as root, security is breached.
So, you've been hacked. It happens. What do you do now?
One of the most common ways to stop hackers once they are in is to disconnect yourself from them. This may include pulling your connection to the Internet or modem banks. You may not want to do this right away however. Often, monitoring hackers for a while can provide multitudes of information, but eventually you will want them out.
Get in the habit of looking for such things on a regular basis. Also, don't trust system programs such as w, lastcomm, ps, etc. If a hacker is in, it is very likely these programs have been replaced with bogus versions to hide the hackers. Keep programs like these on a CD or floppy drive that cannot be written to or modified.
Next, patch the holes the hackers used to get in. This can involve installing patches from vendors or third parties, changing configurations to services or operating systems, or perhaps even discontinuing services even if just temporarily.
There are just about as many methods and processes to preventing break-ins as there are methods to break into a computer system.
This is a continual process of installing patches from vendors, upgrades from software suppliers, and operating system fixes. You should always keep up to date on this as best as you can. It is also not wise to second guess your software vendors or organizations like CERT and CIAC unless you really know what you are doing. Good rule-of-thumb is if CERT says you should fix something, fix it.
The basic concept here is, if nobody knows about your site, they won't try to break into it. Most of the time, this really isn't up to your control. Much of the Internet is in the .com domain, which usually wants to advertise its existence as much as possible. But it is true that hackers like to go after big-news targets like Yahoo and The CIA.
A firewall is a device that sits between your trusted computer system and untrusted computer systems such as the Internet. This definition is deliberately vague because of the wide variety of firewalls and implementations. Usually an actual firewall is either a router or some piece of hardware originally built for firewall purposes (see Cisco or a computer with multiple network interfaces configured for firewall purposes.
Some of the more common things firewalls can do are...
Title:The Cukoo's Egg
Tracking a Spy Through the Maze of Computer Espionage
Author: Clifford Stoll
Publisher: Mass Market Paperback
ISBN: 0-6717-2688-9
Title:Practical UNIX & Internet Security, 2nd Edition
Author:Simson Garfinkel & Gene Spafford
Publisher:oreilly
ISBM:1-56592-148-8