NOS-ABS Packet Filter
A "Receiver" Code Module for Monitoring Packets...
The Perl code following has been several months
in the development and testing phases and appears to be stable enough for
"release." It is named abscopy.plx. Its main function is to open
a socket connection to the Trace Server on JNOS and request a data stream
for a particular device, such as ax0 or hf. The output is directed into a
regular flat file, although it could be a pipe, named after the device. This
allows running as many "filters" as you have devices.
The actual filtering process keys on several identifier
words, such as BEACON, ID, MAIL, CQ or QST. Only data lines with
these keys will be further scanned for message or announcement information.
Once triggered, the code builds a data record associated with the sending
callsign. When an EOR is reached, the script writes the data to the
filehandle and returns to the packet scan to look for more key words.
(The actual database key is a composite built from the callsign and
the type of identifier which is unique.)
A Walk through the Code
This version of the code can be considered as a prototype still under QA test. However, it has been doing fine on my local vhf and hf networks with only a few "glitches" from certain types of atypical broadcasts. The refinement and testing phases continue. At the very least, this release serves as a declaration of the specifications from which code developers could work. This is a very basic module and could eventually be encorporated into a more general server. For now, in this phase though, it remains as standalone code for better error detection.
NOTE: I have made a few very slight modifications to the code. This involves the actual filter for beacons which previously attempted to pick up any line with the word beacon in it. It has been changed to ">beacon" to pick up just this class of broadcast...
- The Perl Source Code
Probably one of the more interesting uses of eval is its ability, in this case, to spawn a child timer process using the alarm function. This "solves" the last record problem. Without a "break-out" signal, the last record might remain hidden until a real packet comes along, EOR end-of-record, to trigger the close on the record. The alarm will force a close, thus ensuring that all data is written to the logger file without protracted delays.
abscopy.plx: the Receiver Module # # ======================================================================= #----- Globally applies to all modules... require 5.002; use strict; use Socket; #----- Signal/interrupt area... $SIG{ALRM} = \&timed_out; #===== Subroutine area... sub timed_out { #----- This shuts off the timer with message in $@... die "TO"; } sub open_sock { #----- Build the socket connection... #----- Have system assign you a port number, if not specified. #----- We have requested/specified a port number... 1236 my $iface = shift; # Such as hf or ax0 my $discode = shift; # The display code for trace my $remote = shift; # The target ip my $port = shift; # port is 1236 my $iaddr = ""; my $paddr = ""; my $proto = ""; my $cmd = ""; $port = getservbyname($port, 'tcp') if ($port =~ /\D/); #----- You need a port to build a socket connection! die "No port... stopping here..." unless $port; #----- You need a host machine... then create an internet address $iaddr = inet_aton($remote) or die "No such host: $remote"; $paddr = sockaddr_in($port, $iaddr); #----- Using tcpip here $proto = getprotobyname('tcp'); #----- Create socket the tcpip "SOCK" socket(SOCK, PF_INET, SOCK_STREAM, $proto) or die "Socket: $!"; #----- Make the "link" and connect to host for data stream... connect(SOCK, $paddr) or die "Connect: $!"; #----- Prepare server "instructions" for this session with these params #----- for this particular port... $cmd = "trace " . $iface . " " . $discode; #----- Prepare for immediate output on this "handle" select(SOCK); $| = 1; #----- NOTE: The telnet protocol needs "\r\n" at the end!!! #----- Instruct the server... send it! print SOCK "$cmd\r\n"; return *SOCK; } sub monitor { #----- Open at least one output file, could be more to write to... #----- If not a named pipe file, use tail -f dev- .log to see data #----- Then stay here and loop... my $mypath = shift; my $iface = shift; my $delay = shift; my $socket = shift; my $traceout = ""; my $lastline = ""; my $line = ""; my $record = ""; my $key = ""; my $callsign = ""; my $type = ""; my $rcline = ""; my $rest = ""; my $tdstamp = ""; my $lastrec = ""; my $textrec = ""; #----- Change if you wish... my $devprefix = "dev-"; #----- First open the output file... $traceout = $mypath . "/" . $devprefix . $iface . ".log"; open(FILEOUT, ">>$traceout") or die "Cannot open output file"; #----- Must do this if spawned from a server, else no output select(FILEOUT); $| = 1; #----- This is the "forever" monitoring loop... $lastline = ""; $line = ""; $lastrec = " "; UPPER: while ( 1 ) { $record = ""; $rcline = ""; $line = ""; while(defined($line = <$socket>)) {last;} next UPPER unless $line; #----- Condition the line, chop the "\r\n" EOL... $rcline = $line; $rcline =~ s!\r\n$!!; $callsign = ""; $type = ""; #----- Filter this line and assign the secondary part of the key #----- Modify as required... if anything make it shorter if ( $rcline =~ />ID|>BEACON|>CQ|>MAIL|>TEST|>QST/o ) { $rest = ""; if ( ($callsign, $rest) = split(/->/, $rcline, 2) ) { if ( $rest =~ /MAIL/o ) { $type = "MAIL"; } elsif ( $rest =~ /BEACON/o ) { $type = "BEACON"; } elsif ( $rest =~ /ID/o ) { $type = "ID"; } elsif ( $rest =~ /CQ/o ) { $type = "CQ"; } elsif ( $rest =~ /TEST/o ) { $type = "TEST"; } elsif ( $rest =~ /QST/o ) { $type = "QST"; } else { #----- Just in case something goes "haywire" $type = ""; } } } else { #----- If didn't find identifiers on the master list, then go back... next UPPER; } #----- Must find a type on the list... next UPPER unless $type; #----- Could have a callsign test as per above if needed... # next UPPER unless ... ; #----- This is an optional feature! It attempts to block rapidly sequential #----- CQ's, TEST's, or QST's, while always passing ID's, BEACON's, #----- or Mail's. #----- Either use this or "if" code block below, not both... # next UPPER if $lastline =~ /$line/; #----- This is the pass-thru list... always let through... if ( $type !~ /ID|BEACON|MAIL/o ) { #----- Return immediately repetitive CQ's, TEST's, QSTs... next UPPER if $lastline =~ /$line/; } #----- Build the composite key from "header" line, then build record... #----- A time stamp date can go here, could add julian number too, %j: $tdstamp = ""; $key = ""; chomp($tdstamp = `/bin/date +%H:%M:%S`); $key = $callsign . ":" . $type; $record = $key . " " . $tdstamp . " "; #----- Store last line for comparison $lastline = $line if $line; #----- Now read the record details if any exist... #----- Exit socket wait after the "delay" period RECLOOP: { $rcline = ""; eval { alarm($delay); while(defined($line = <$socket>)) {last;} alarm(0); }; if ( $@ =~ /TO/o ) { <$socket>; last RECLOOP; } $rcline = $line; $rcline =~ s!\r\n$!!; #----- Optional char filter for record... don't print unprintables... $rcline =~ tr[\0-\011\013-\037\177][]d; #----- When start of next data block arrives, stop record acculumlation #----- NOTE: This is specific to the JNOS Trace Server application!!! last RECLOOP if ( $rcline =~ /^$iface /o ); #----- NOTE: Not recording this data! But could optionally append... if ( $type =~ /MAIL/ && $rcline =~ /Active /o ) { $record = ""; last RECLOOP; } if ( $rcline ) { #----- New line subsitute is the pipe "|" char, could use something different $record .= $rcline . "|"; #----- Just store last line of text body... $textrec = $rcline . "|"; } redo RECLOOP; } #----- Now write record to screen/file... #----- Could split to different logger files too by type: ID, BEACON, etc #----- Don't print repetitive record texts ... comment out if not needed... if ( $lastrec ne $textrec ) { print FILEOUT "$record\n"; $lastrec = $textrec; } #----- Bottom of "forever" while loop } } # ------------------------------------------------------------------ # Main # ------------------------------------------------------------------ #----- These are default params for the Traceserver on JNOS machine... #----- May be passed in... my $iface = shift || "hf"; # The interface to trace my $discode = shift || 311; # The display code/mode my $remote = shift || "44.56.26.11"; # Enter as arg or this my $port = shift || 1236; # The trace port on JNOS #----- Initializations... my $mypath = "/root/uptoisp/db105"; # "Home" directory my $delay = 15; # Get "last rec" after delay (15-30) #----- Now build the socket connection and open it for read/write... my $socket = &open_sock($iface, $discode, $remote, $port); #----- Create an output log file then enter a continuous loop #----- and monitor packet activity until a quit/kill... &monitor($mypath, $iface, $delay, $socket); #----- Never gets here... exit; #----- End of abscopy.plx
Also, all instances of a new line, or new line carriage return, are swapped with the pipe "|" character. If someone gets "fancy" and starts using that character in broadcasts, then another more "obtuse" character may have to be chosen. (You might want to use something different anyway...)
The script is set to ignore immediately multiple broadcasts for CQ, TEST, and QST. Some stations will send out several CQ's back-to-back and there is a "trap" near the top of the filter loop to ignore these multiples. However, all BEACONS and IDs are passed though, unless the content of the broadcast is the same. This avoids the ACKs for relayed beacon/id broadcasts. The code checks the last line of any message and prints it to a file if it is different from the preceeding last line, more extensive testing could be done. All of the above "filters" may be commented out if found to be too restictive or inappropriate to your net operations. They were included here to "document" the options if needed.- How to Install and Run
You may locate this code anywhere you like. Just be sure to edit the mypath variable above so that it reflects the real path you are using. Be sure to chmod the code with chmod +x abscopy.plx so that it does become executable. And replace my JNOS IP with your IP into the application. To start it up, use:
- abscopy.plx ax0 &
... or ...
- ./abscopy.plx ax0 &
where you replace ax0 with your actual device if it differs... The code creates a logger file named dev-ax0.log and begins to write data to it as heard...
You may continue to start up as many copies of the code as you have devices to monitor. I start one for an hf device too:
- abscopy.plx hf &
... or ...
- ./abscopy.plx hf &
Again, the code creates a log file named dev-hf.log and begins to write data to that file.
And remember, this code connects to a socket on JNOS, 1236, which links to the Trace Server, so that server must be enabled for your JNOS system, or it won't be able to complete that link and this code won't work! So just copying this code and activating it is not enough. Be sure that your Trace Server is up and running and can take "calls" from clients, which is what this code is, a filtering client!- Some Typical Output and what to do with it
What does the output look like? It goes to a file, or possibly a pipe, and therefore you won't see any immediate output from the code on the screen as in a typical trace operation . The easiest way to prove that you have output is to "cat" the logger file. Here is a sampled excerpt from the dev-ax0.log file:
I have added some new line characters (\n) so that the logger file is easier to read here.
Sample Output: dev-ax0.log
Here is another sample from the dev-hf.log file:
And, again, for the long lines, I have added new lines here for readability. These files are really "raw" data files and need some further interpretation to be displayed in a readable format. Here is a very simple shell command line to view these records:
Sample Output: dev-hf.log
- cat dev-ax0.log | tr '|' '\n' | more
This will display the records as they arrived in a presentable format. Sometimes you need to see the "raw" data but in a more readable format. The above command line will show you the contents at a glance.
Here is a shell command line, which uses the follow switch for tail, that will display the contents as they arrive live at your station...
- tail -f dev-ax0.log | tr '|' '\n' &
(This will use double spacing between records. If you don't want that, then just use the tail part of the command.)
Having the data for a device stored persistently, and easily accessible, can be a great asset for acting on BEACON/ID alerts. There are numerous sorts and greps that can be applied to the "raw" file in Linux. For example, you could call out a sequence for any given station and sort by time. This would show any changes in messages carried by the BEACONS/IDs. You now have a complete record along a time line for all the day's beaconing events. The shell script code following may give you some ideas as to how to implement these methods...- A Slightly More Elaborate Displayer
If you need to see the output in an easily readable format, and in one that simulates a typical trace function, then the following somewhat "primative" shell script program will sort the named logger file by date, last heard, and then do a sort unique by callsign. Here is that script:
Displayer Script: display #! /bin/sh # display # 07-18-04 # 08-20-04 # KA1FSB/kbn # ---------------------------------------------------------------- # Format and display this file. This is a basic display script. # The input file is the local mheard.txt file! # # Now inlcudes a "call" to do_mysort to get a unique list... # Usage: display ax0 | more, or display hf | more # ---------------------------------------------------------------- #===== Subs... #----- Routine to sort a logger file... do_mysort () { #----- Inits & defaults... myfile="" bigBuf="" devtype="$1" if [ "$devtype" ] then myfile="dev-${devtype}.log" fi #----- Sort & write data to output file... #----- You may try different sorters here for fun! sort -k1,1 -k2,2r $myPath/${myfile} | \ sort -u -k1,1 | pr -t -n > $myPath/mheard.txt } #===== Main #----- Inits... line="" title="" body="" device="" off="\033[m" bld="\033[1m" myPath="/root/other/db105" #----- Check for args... if [ "$1" ] then device=$1 else #----- Your default device... device="hf" fi #----- Call this sort routine... do_mysort $device #----- Start at screen top clear #----- Spin thru the data file... while [ 1 ] do read line || break title=`echo "$line" | cut -f1-5 -d " "` body=`echo "$line" | cut -f6-500 -d " "` echo -e "${bld}${title}${off}" echo "$body" | tr '|' '\n' | pr -t -o6 done < $myPath/mheard.txt #----- End of display
You may have to "adjust" lines of this code to get it to work properly on your system as in all differing UNIX/Linux environments. For example, on my Slackware development platform, the cut commands are specified as above. However, on my SuSE platform, the first cut ends at 2 (-f1-2) and the next line begins at 3 (-f3-500). You will have to experiment to see how it runs on your machine. Also, the "pr -t -o6" may need to be changed to "pr -t -o10". You can format this display anyway you choose! Whatever conforms to your network, or personal format, style!- Caveats
As always, you use this code at your own risk. I have done my best to carry out extensive QA'ing, but there is no code that does not have some unforseen consequences, translate "bugs," lurking in the wings... even though I "compiled" it with the "NO-BUGS" switch :)
Also, this was developed on Linux OS 2.0.30 and in a Slackware package. If you run from another package, like SuSE for example, even though the application environment has been standardized a la Perl, there still may be subtle differences to watch out for in socket IO and file handling that are dependent on the OS. Just a "heads up" for general usage. "Beat the heck" out of it and keep any eye on it ... and modify as required.- Conclusions
I hope this short article has inspired you to try this NOS-ABS approach to opening up the BEACON/ID potential for *NOS! Use this code as a springboard for future developement on your system. Integrate it into your everyday operations with a view toward making your NOS station a true broadcasting node with emergency capability! Let me know if you have moved the "boulder" one micron ahead... Have fun and stay on-the-alert with NOS-ABS!
(Courtesy KBNorton Computer Services)