To find connections to unknown hosts you can rely on firewall logs or use netstat to actually look for them. To find rouge applications listening for connections on random ports you have to reply on netstat. The problem is finding the few "interesting" lines out of hundreds or thousands of recognized and hence uninteresting lines. The following netstat_filter.pl Perl1 script can take some of the pain out of this.
The netstat_filter.pl Perl script will capture the netstat output to a file and then filter out all the uninteresting lines as defined by a list of regular expressions (the reason for using perl, versus a VOS command macro). Anything not filtered out is by definition interesting. You can also define a set of regular expressions to explicitly identify interesting lines. Once interesting netstat lines are identified it will try to determine the processes attached to the corresponding sockets. Figure 1 shows the output of netstat, figure 2 the output from the script with 3 interesting sockets.
Active connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp 0 0 *:23 *:* LISTEN tcp 0 0 *:21 *:* LISTEN tcp 0 0 *:7 *:* LISTEN tcp 0 0 *:9 *:* LISTEN tcp 0 0 *:13 *:* LISTEN tcp 0 0 *:19 *:* LISTEN tcp 0 0 *:37 *:* LISTEN tcp 0 0 *:901 *:* LISTEN tcp 0 0 *:3000 *:* LISTEN tcp 0 0 *:3001 *:* LISTEN tcp 0 0 *:3002 *:* LISTEN tcp 0 0 *:3003 *:* LISTEN tcp 0 0 *:3004 *:* LISTEN tcp 0 0 *:3005 *:* LISTEN tcp 0 0 *:3006 *:* LISTEN tcp 0 0 *:3007 *:* LISTEN tcp 0 0 *:3008 *:* LISTEN tcp 0 0 *:3009 *:* LISTEN tcp 0 0 *:3010 *:* LISTEN tcp 0 0 *:3011 *:* LISTEN tcp 0 0 *:3012 *:* LISTEN tcp 0 0 *:3013 *:* LISTEN tcp 0 0 *:3014 *:* LISTEN tcp 0 0 *:3015 *:* LISTEN tcp 0 0 *:3016 *:* LISTEN tcp 0 0 *:3017 *:* LISTEN tcp 0 0 *:3018 *:* LISTEN tcp 0 0 *:3019 *:* LISTEN tcp 0 0 *:3020 *:* LISTEN tcp 0 0 *:3021 *:* LISTEN tcp 0 0 *:3022 *:* LISTEN tcp 0 0 *:3023 *:* LISTEN tcp 0 0 *:3024 *:* LISTEN tcp 0 0 *:3025 *:* LISTEN tcp 0 0 *:3026 *:* LISTEN tcp 0 0 *:3027 *:* LISTEN tcp 0 0 *:3028 *:* LISTEN tcp 0 0 *:3029 *:* LISTEN tcp 0 0 *:3030 *:* LISTEN tcp 0 0 *:3031 *:* LISTEN tcp 0 0 164.152.77.128:3001 164.152.77.203:59608 ESTABLISHED tcp 0 0 164.152.77.128:3002 164.152.77.203:60154 ESTABLISHED tcp 0 48 172.16.1.116:22 164.152.77.50:5446 ESTABLISHED tcp 0 0 164.152.77.128:3014 164.152.77.11:52813 ESTABLISHED tcp 0 0 172.16.1.116:445 *:* LISTEN tcp 0 0 172.16.1.116:139 *:* LISTEN tcp 0 0 127.0.0.1:445 *:* LISTEN tcp 0 0 127.0.0.1:139 *:* LISTEN tcp 0 0 *:10000 *:* LISTEN tcp 0 0 164.152.77.128:49154 164.152.77.203:3000 ESTABLISHED tcp 0 0 *:80 *:* LISTEN tcp 0 0 164.152.77.128:50211 164.152.77.34:3000 ESTABLISHED tcp 0 0 172.16.1.116:9999 *:* LISTEN tcp 0 0 *:2200 *:* LISTEN tcp 0 0 172.16.1.116:23 *:* LISTEN tcp 0 0 164.152.77.128:3004 164.152.77.34:60514 ESTABLISHED tcp 0 0 *:22 *:* LISTEN tcp 0 0 10.20.1.1:61196 10.20.1.9:48879 ESTABLISHED tcp 0 0 164.152.77.128:3003 164.152.77.34:60515 ESTABLISHED tcp 0 0 164.152.77.128:3002 164.152.77.34:60516 ESTABLISHED tcp 0 0 164.152.77.128:3000 164.152.77.34:60517 ESTABLISHED tcp 0 0 164.152.77.128:50455 164.152.77.11:3000 ESTABLISHED tcp 0 0 164.152.77.128:3001 164.152.77.34:60518 ESTABLISHED tcp 0 0 10.20.1.1:61197 10.20.1.3:48879 ESTABLISHED tcp 0 0 10.20.1.1:61198 10.20.1.3:48879 ESTABLISHED tcp 0 0 172.16.1.116:22 164.152.77.50:6458 ESTABLISHED tcp 0 0 10.20.1.1:61199 10.20.1.9:48879 ESTABLISHED tcp 0 0 10.20.1.1:61200 10.20.1.9:48879 ESTABLISHED tcp 0 0 10.20.1.1:61201 10.20.1.9:48879 ESTABLISHED tcp 0 0 10.20.1.1:61202 10.20.1.3:48879 ESTABLISHED udp 0 0 *:161 *:* udp 0 0 *:7 *:* udp 0 0 *:13 *:* udp 0 0 *:19 *:* udp 0 0 *:69 *:* udp 0 0 10.10.1.1:500 *:* udp 0 0 172.16.1.116:500 *:* udp 0 0 164.152.77.128:500 *:* udp 0 0 *:123 *:* udp 0 0 172.16.1.116:137 *:* udp 0 0 *:138 *:* udp 0 0 172.16.1.116:138 *:* udp 0 0 10.10.1.1:123 *:* udp 0 0 10.20.1.1:123 *:* udp 0 0 164.152.77.128:123 *:* Active LOCAL (UNIX) domain sockets Type RxMsgs TxMsgs State Local Name <-> Remote Name ------ 0 0 UNBOUND _aat8atz61KWebFN9 |
******************** Wed Dec 31 13:05:37 2008 ******************** 8dce3400 tcp 0 0 172.16.1.116:22 164.152.77.50:5446 ESTABLISHED stcp.m16_29548: Object is write locked by root.root (sshd) on module %phx_vos#m16 executing sshd.pm. 8db51c00 tcp 0 0 *:22 *:* LISTEN stcp.m16_14405: Object is write locked by root.root (sshd) on module %phx_vos#m16 executing sshd.pm. 8dcc4ec0 tcp 0 0 172.16.1.116:22 164.152.77.50:6458 ESTABLISHED stcp.m16_30344: Object is write locked by root.root (sshd) on module %phx_vos#m16 executing sshd.pm. |
Usage
perl netstat_filter.pl [-filter FILTER_FILE] [-out OUTPUT_FILE_BASE] [-sleep SLEEP_TIME] [-quiet] [-temp TEMP_FILE_BASE] [-once]
-filter filter_file
This is the path to a file containing the list of regular expressions defining the uninteresting (and interesting) netstat lines. The default is netstat_filter_list
-out base_output_file
This is the base path to the output file. The default is netstat_filter_out. A date stamp is appended to the name and automagically changes every day. For example, netstat_filter_out_2009-01-12, netstat_filter_out_2009-01-13, netstat_filter_out_2009-01-14
If you wish the output to go to the terminal window set the out file to STDOUT (case is important).
-sleep NUMBER
netstat_filter.pl will loop forever; this parameter tells it how many seconds to sleep after processing the current iteration. The actual time between iterations is this sleep time plus the time to process the last iteration.
-quiet
At the start of each iteration a header line is output (see figure 2) with a date time stamp. Hopefully, there will be nothing following the header line. If -quiet is set then the header line is only output if some interesting netstat lines are found. Also when the script is first run the regular expressions in the filter file are output (see figure 6), -quiet will also suppress that. So if you use -quiet and there is nothing interesting to report the output file will be empty.
-temp base_temp_file
netstat_filter requires a couple of temporary files, one to hold the netstat output and a second to hold the output from the dump_onetcb analyze_system command. These files are named base_temp_file.X and base_temp_file.Y. The default value is netstat_filter_temp.
-once
Execute the loop once and terminate. I envision this as a debugging argument,
Filter File
The filter file is a list of regular expressions and comments. Comments are indicated by a # character in column 1. Figure 3 shows the list I am currently using. The first set of lines filter out blank lines and header lines. The next set of lines filter out the AF_UNIX sockets and header lines. This is followed by 1 line that filters out CLOSED sockets. Next are lines that filter out connections from hosts on the locally attached subnets and from anything to (or from) the local host address. Finally I have lines defining all the uninteresting listening sockets.
# blank lines ^\s*$ # # netstat header lines Recv-Q Active connections # # AF UNIX domain sockets RxMsgs TxMsgs UNBOUND # # CLOSED sockets CLOSED$ # # local network connections 10.20.1.1:.* 10.20.1. 164.152.77.128:.*164.152.77. 172.16.1.116:.*172.16.1. # # anything local host 127.0.0.1: # # known listeners # # OSL :300.\s*\*:\* :301.\s*\*:\* :302.\s*\*:\* :3030\s*\*:\* :3031\s*\*:\* # # tiny services :7\s*\*:\* :9\s*\*:\* :13\s*\*:\* :19\s*\*:\* :21\s*\*:\* :37\s*\*:\* # # NTP :123.*\*:\* # # SAMBA socket :138\s*\*:\* :137\s*\*:\* :139\s*\*:\* :445\s*\*:\* # # IKED :500\s*\*:\* # # SSH # :22\s*\*:\* # # Specal SSH for testing :2200\s*\*:\* # # Telnet :23\s*\*:\* # # special telnet for testing :9999\s*\*:\* # # ndmpd :10000\s*\*:\* # # TFTPd :69\s*\*:\* # # SNMPd :161\s*\*:\* # # SWAT :901\s*\*:\* # # Apache :80\s*\*:\* |
You have to consider when you set up the filter what you are looking for. In my case I assume that any local connection, i.e. where the remote host is on a directly connected subnet is OK. Also, I have included the TCP Tiny services; you may not want to run these services in which case they should not be in this list.
In some cases it may be difficult to filter out uninteresting lines while keeping interesting lines. For example, let's say that you want to know of any telnet connections from any network except 10.20.1.0/24, even connections from other local subnets. The above filter list will not work because it would filter out telnet connections from clients on any of the local subnets. The alternative is to filter out local connections for all services except telnet; this will make the filter list much longer. Another alternative is to tell netstat_filter.pl that
         172.16.1.116:23.*
         164.152.77.128:23.*
are interesting (this assumes that only clients on the 10.20.1.0/24 subnet can connect to the local interface on that subnet).
You do this by placing a plus character in column 1 of the filter list. The plus is not part of the regular expression; it is a flag that tells netstat_filter.pl to mark matching lines as interesting. Regular expressions flagged as interesting should be at the beginning of the netstat_filter_list file, before expressions defining uninteresting lines.
# Interesting telnet connections +172.16.1.116:23\s*\d +164.152.77.128:23\s*\d # # blank lines ^\s*$ # # netstat header lines Recv-Q Active connections # # AF UNIX domain sockets RxMsgs TxMsgs UNBOUND # # CLOSED sockets CLOSED$ # # local network connections 10.20.1.1:.* 10.20.1. 164.152.77.128:.*164.152.77. 172.16.1.116:.*172.16.1. . . . |
Notes
You can run Perl directly from the VOS command line, it is not necessary to be in the GNU shell (bash).
If you run this as a started process you must set -privileged since it does use analyze_system to try to trace back the socket to a process
start_process 'perl netstat_filter.pl -quiet' -privileged ready 09:28:15 |
As the script reads the filter list it will print it to the output file (assuming -quiet is not specified on the command line). Lines flagged as interesting will be preceded with a 1 character, lines not flagged as interesting will be preceded with a 0 character (figure 6)
1 172.16.1.116:23.* 1 164.152.77.128:23.* 0 ^\s*$ 0 Recv-Q 0 Active connections 0 domain sockets 0 RxMsgs TxMsgs 0 UNBOUND |
The script calls the Perl system function at least once each iteration to execute netstat. If it finds something interesting it will call system twice more for each interesting socket to execute the analyze_system and who_locked commands. There is also a call to system to execute the set_implicit_locking command when the output file is first created. Each call to system creates 2 processes. If you are monitoring process creation and termination in the syserr_log this generates at least 4 lines every N seconds, where N is the value of the sleep argument (default 30).
15:55:58 Process 5510CD87, Noah_Davids.CAC (perl), created. 15:55:58 Process 5510CD88, Noah_Davids.CAC (perl), created. 15:55:59 Process 5510CD88, Noah_Davids.CAC (perl), terminated. 15:55:59 Process 5510CD87, Noah_Davids.CAC (perl), terminated. 15:56:29 Process 5510CD89, Noah_Davids.CAC (perl), created. 15:56:29 Process 5510CD8A, Noah_Davids.CAC (perl), created. 15:56:29 Process 5510CD8A, Noah_Davids.CAC (perl), terminated. 15:56:29 Process 5510CD89, Noah_Davids.CAC (perl), terminated. 15:56:59 Process 5510CD8B, Noah_Davids.CAC (perl), created. 15:56:59 Process 5510CD8C, Noah_Davids.CAC (perl), created. 15:56:59 Process 5510CD8C, Noah_Davids.CAC (perl), terminated. 15:56:59 Process 5510CD8B, Noah_Davids.CAC (perl), terminated. 15:56:59 Process 5510CD8D, Noah_Davids.CAC (perl), created. 15:56:59 Process 5510CD8E, Noah_Davids.CAC (perl), created. 15:56:59 Process 5510CD8E, Noah_Davids.CAC (perl), terminated. 15:56:59 Process 5510CD8D, Noah_Davids.CAC (perl), terminated. 15:56:59 Process 5510CD8F, Noah_Davids.CAC (perl), created. 15:56:59 Process 5510CD90, Noah_Davids.CAC (perl), created. 15:56:59 Process 5510CD90, Noah_Davids.CAC (perl), terminated. 15:56:59 Process 5510CD8F, Noah_Davids.CAC (perl), terminated. 15:57:29 Process 5510CD91, Noah_Davids.CAC (perl), created. 15:57:29 Process 5510CD92, Noah_Davids.CAC (perl), created. 15:57:30 Process 5510CD92, Noah_Davids.CAC (perl), terminated. 15:57:30 Process 5510CD91, Noah_Davids.CAC (perl), terminated. |
If you delete the current output file it maybe be recreated without implicit_locking set. A test is made at the start of the loop, if the file does not exist it is created and implicit_locking is set. If the file is deleted after this test and before the end of the loop something needs to be written the file will be created again. If that happens it will not have implicit_locking set. If you are then reading the file when the script tries to write to it, the script will fault with a locking error and terminate. I felt it was too expensive to call system after every write to set implicit_locking or even to always call system at the start of the loop. Don't delete the output file for the current date.
netstat_filter.pl
# netstat_filter.pl begins here
#
# Version 1.03 09-01-22
# Version 1.10 10-11-26 Added disclaimer
# noah_davids@stratus.com
#
# See http:noahdavids.org/self_published/netstat_filter.html
# for documentation
#
#
# This software is provided on an "AS IS" basis, WITHOUT ANY WARRANTY OR ANY
# SUPPORT OF ANY KIND. The AUTHOR SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES
# OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. This disclaimer
# applies, despite any verbal representations of any kind provided by the
# author or anyone else.
#
use strict;
use warnings;
use Getopt::Long;
my ($result, @filter_file, @output_file, @sleep_time,
@quiet, @temp_file, @once);
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst);
my ($date_stamp, $f, $r, $x, $matched, $now, $first, $output_to_file);
my ($FILTER, $NETSTAT, $OUT);
my (@filters, @interesting, $i);
my ($PCB, $ASOUTPUT);
my ($openparen, $closeparen, $device);
$result = GetOptions ('filter=s' => \@filter_file,
'out=s' => \@output_file,
'sleep=s' => \@sleep_time,
'quiet' => \@quiet,
'temp=s' => \@temp_file,
'once' => \@once);
if ($result != 1)
{
print "\n\nUsage:\n";
print "perl netstat_filter.pl [-filter FILTER_FILE] " .
"[-out OUTPUT_FILE_BASE] " .
"[-sleep SLEEP_TIME] [-quiet] [-temp TEMP_FILE_BASE] " .
"[-once]\n\n";
exit;
}
else
{
if (@filter_file == 0) {$filter_file [0] = "netstat_filter_list";}
if (@output_file == 0) {$output_file [0] = "netstat_filter_out";}
if (@sleep_time == 0) {$sleep_time [0] = 30;}
if (@quiet == 0) {$quiet [0] = " ";} else {$quiet [0] = "-quiet";}
if (@temp_file == 0) {$temp_file [0] = "netstat_filter_temp";}
if (@once == 0) {$once [0] = " ";} else {$once [0] = "-once";}
}
print "perl netstat_filter.pl -filter " . $filter_file [0] .
" -out " . $output_file [0] .
" -sleep " . $sleep_time [0] . " " .
$quiet [0] .
" -temp " . $temp_file [0] . " " .
$once [0] . "\n\n";
# set up temporary file names
$r = $temp_file [0];
$temp_file [0] = $r . "X";
$temp_file [1] = $r . "Y";
if ($output_file [0] =~ /STDOUT/) {$output_to_file = 0;}
else {$output_to_file = 1;}
if ($output_to_file)
{
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);
$year = $year+1900;
$mon = $mon+1;
$date_stamp = $year . "-" . $mon . "-" . $mday;
$f = $output_file [0] . "_" . $date_stamp;
# first open the file, this creates it, if it is there delete it and create it
open ($OUT, ">".$f) || die "Can't open outfile " . $f . " [-1]";
# file is open so close it
close $OUT;
# now call the system routine to execute the set_implicit_locking command
# to set implicit locking on the file
$r = system ("set_implicit_locking " . $f);
# reopen file file so we can actually write to it
open ($OUT, ">".$f) || die "Can't open outfile " . $f . " [0]";
}
# open the filter file so we can read the filer list
open ($FILTER, '<', $filter_file [0]) || die "Can't open filter file list: "
. $filter_file [0];
$i = -1;
while ($r = <$FILTER>)
{
if (substr ($r, 0, 1) ne '#') # skip comments
{
$i++;
if (substr ($r, 0,1) eq '+') # this means the line defines
{ # something interesting.
$interesting [$i] = 1; # set the interesting flag and trim
$r = substr ($r, 1); # off the leading plus character
}
else
{$interesting [$i] = 0;} # the line defines something
$filters [$i] = $r; # uninteresting. Either way save the
chop ($filters [$i]); # regular expression and chop off the
if (length ($quiet [0]) == 1) # ending CR that got added as the
{ # line was read. If -quiet was NOT
if ($output_to_file) # specified then print the line
# to either the out file or the
# terminal window
{print $OUT $interesting [$i] . " " . $filters [$i] . "\n";}
else {print $interesting [$i] . " " . $filters [$i] . "\n";}
} # (length ($quiet [0] > 1)
} # (substr ($r, 0, 1) ne '#') # skip comments
} # while ($r = <$FILTER>)
if ($output_to_file) {close $OUT;} # if writing to a file, close it.
close $FILTER; # file remains closed unless we are
# actually writing to it.
#
# this is the main loop.
while (1)
{
# reconstruct the output file name. This way we do not need to keep track
# of the day, when the day changes the file name automagically changes
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);
$year = $year+1900;
$mon = $mon+1;
$date_stamp = $year . "-" . $mon . "-" . $mday;
$f = $output_file [0] . "_" . $date_stamp;
# pretty much the same drill as above, if the file does not exist
# create the file, close it and set implicit locking.
# But this time the file is left closed.
if ($output_to_file)
{
if (!(-e $f)) # if output file does not exist, create it and set
{ # implicit locking
open ($OUT, ">".$f) || die "Can't open outfile " . $f . " [-1]";
close $OUT;
$r = system ("set_implicit_locking " . $f);
}
}
$now = localtime (); # get the current time stamp
if (length ($quiet [0]) == 1) # -quiet was NOT an argument so always
{ # output time stamp at start of loop
if ($output_to_file)
{
open ($OUT, ">>".$f) || die "Can't open outfile " . $f . " [1]";
print $OUT
"\n\n ******************** ". $now . " ********************\n";
close $OUT;
}
else
{print "\n\n ******************** ". $now . " ********************\n";}
}
# run netstat and send the output to the first temporary file
$r = system ("netstat -numeric -all_sockets -PCB_addr > " . $temp_file [0]);
# we only want to write the time stamp out once. The first flag is set to 1
# now and will be zeroed when the time stamp is written.
$first = 1;
# open the temp file with the netstat output and loop through all the lines
open ($NETSTAT, '<', $temp_file [0]) ||
die "Could not open temp file " . $temp_file [0];
while ($_ = <$NETSTAT>)
{
$matched = 0; # matched is set to 1 if the regular expression is found
$x = 0; # x is the index used to loop through all the filters
# loop until we reach the end or a match is found
while (($x <= $i) && !$matched)
{if (/$filters[$x++]/) {$matched = 1;}}
# if no match is found or we have matched a regular expression defining an
# interesting line. We need to subtract 1 from the array index because we
# have already incremented the index so the index is now 1 greater than the
# filter we just matched (if we matched a filter that is)
if ((!$matched) || ($matched && $interesting [$x - 1]))
{
if ($first) # first interesting netstat line found in this loop
{
if (length ($quiet [0]) > 1) # -quite was an argument so no time
{ # stamp was written at the start of
if ($output_to_file) # the loop so we need to do it now
{ # before first interesting netstat
open ($OUT, ">>".$f) || # line
die "Can't open outfile " . $f . " [2]";
print $OUT
"\n\n ***************** ". $now . " *****************";
close $OUT
}
else {print
"\n\n ***************** ". $now . " *****************";}
} # if (length ($quiet [0]) > 1)
$first = 0; # time stamp was written so set first to 0 so
} # if ($first) # we don't do it again in this iteration
# output a couple of blank lines followed by the interesting netstat line
if ($output_to_file)
{
open ($OUT, ">>".$f) ||
die "Can't open outfile " . $f . " [3]";
print $OUT "\n\n", $_;
close $OUT;
}
else {print "\n\n", $_;}
# the PCB addr is the first 8 characters in the line. Extract it out and
# use system to execute analyze_system to dump the TCB and related
# structures. Output goes to the second temp file.
# Note that this works even with UDP sockets - mostly. UDP sockets can be
# destroyed very quickly and no locking is done with this analyze_system
# request. The result is that request *may* fail and it can cause
# analyze_system to create a keep.
$PCB = substr ($_, 0, 8);
$r = system
("analyze_system -quit -request_line 'match dv; dump_onetcb " .
$PCB . "' > " . $temp_file [1]);
# The analyze_system output should be 3 lines, the first 2 are header lines
# the third line will contain the line with the device name. It is possible
# that the third line will not be there in which case defined ($r) is false.
# this means that a device was not found. This can happen if the socket is
# being cleaned up or is used by OSL.
open ($ASOUTPUT, '<', $temp_file [1]) ||
die "Could not open temp file " . $temp_file [1];
$r = <$ASOUTPUT>;
$r = <$ASOUTPUT>;
$r = <$ASOUTPUT>;
close $ASOUTPUT;
if (defined ($r)) # If the line there exrtact out the
{ # device name which is enclosed in ()s
$openparen = index ($r, '(') + 1;
$closeparen = index ($r, ')');
$device = substr ($r, $openparen, $closeparen - $openparen);
# once we have a device name execute the who_locked command.
if ($output_to_file)
{
$r = system ("who_locked \\#" . $device . " >> " . $f);
}
else
{
$r = system ("who_locked \\#" . $device);
}
} # if (defined ($r))
} # if ((!$matched) || ($matched && $interesting [$x - 1]))
} # while ($_ = <$NETSTAT>)
# if -once was an argument quit otherwise sleep and do the loop again
if (length ($once [0]) > 1) {exit;}
sleep ($sleep_time [0])
} # while (1)
#
# netstat_filter.pl ends here
Footnotes
1    Perl is part of the GNU C/C++ and GNU Tools product (S877).
Send comments and suggestions
This page was last modified on 10-11-26
to noad.davids@stratus.com