The Trace Server on JNOS

There are many good reasons to monitor your own transmissions, and transmissions in general. For example, sometimes, despite being a poor typist, I send too much data too fast without looking for that "RR", ready receive signal, in a monitoring window. Now, I "temper" my sending based on the traffic observed. This helps me to be a better operator, and ensures that my messages get through with a minimum of collisions and retries.

How to Work the Trace Server

What is the Trace Server?

The Trace Server is a non-concurrent, single queue server designed to display packets originating on JNOS interfaces. It is identical to the normal trace "functions" found in JNOS except that it can extend its capability over a network! For example, let's say you have a small Lan, with your radio server on one machine and a laptop or desktop at a "remote" location on another. You would like to be able to trace or monitor network activity from the remote machine you are working on. The Trace Server makes this possible by setting up a socket connection to the JNOS box. (The port number is fixed at 1236, but it is possible to change it...)

You don't necessarily have to be at a remote location. You could login to another session on Linux, telnet to the JNOS IP address on port 1236, and you would be monitoring packets in a dedicated screen until you break, or disconnect, the telnet connection. If you run "X", this would be a convenient way to make a movable, re-sizeable window which you could bring to the foreground when you want to resume your monitoring.

JNOS has the capability to toggle among sessions, in which you may have running any number of interface traces, but you cannot do this over a network connection, only on the host machine that you are logged in to. The Trace Server frees you from these restrictions, opening up your trace "window" from any location on your local net, or your host machine application environment...

Caveats...

This commentary applies to the JNOS version 1.11f. That's the version I am running here, so that is what I documented. As you know, software can vary widely from version to version. What you are reading now is a "snapshot" in time and could be quite different on another version, or another rendition of NOS, such as TNOS, for example... This attempt is meant to give you an overview and get you started working with the Trace Server. (I will update or make corrections as discovered, or as alerted to them.)

A Few Preliminaries...

From the autoexec.nos in the start up area for servers, you must include a "start" command for the trace server such as:

• start trace

or as a one-shot test, you could enter the same command from the admin console. This assumes that you have compiled JNOS with the traceserver enabled, or #define'd in the config.h file. And, the chances are that you may not have, since the trace server is a relatively undocumented feature in JNOS! So here is how to do that. Make sure you have a line under the heading "/* Network services */" in the config.h file that looks like this:

• #define TRACESERVER      1 /* remote interface trace server */

As you can guess by the comment line, the Trace Server is a bit different from the normal trace even though it may share the same code. This offers remote service over a network! It is also possible that your package came with this line already enabled, with the #define already in place. In that case, you do not need to re-compile anything! All you need to do is activate it by using the start command above... (However, if you changed or edited this line in the config.h file, you need to re-compile JNOS now.)

NOTE :The Trace Server is different from the usual trace that either already comes available in JNOS, or that you have defined in a similar fashion as above. You do not need to have the Trace Server "installed" to do tracing or monitoring from an admin screen or your autoexec.nos file, where you might issue a command such as, "trace ax0 311". But, if you want to do this over the network, then you need the Trace Server, and you need to tell the compiler to include it in the binary as "flagged" above in the config.h file!

As mentioned above, you may change the port number for the Trace Server. In the file socket.h, found in your JNOS compile area or directory, you will see a line that shows:

• #define IPPORT_TRACE 1236    /* Pulled out of the air -WG7J */

Change the 1236 to whatever you have decided on. Be very sure that your new port number is a free number and not being used by JNOS. You will have to check all the .h files for port numbers in use. (And again, if you change or edit this file, you need to re-compile JNOS now.) Generally, there is no convincing reason to change this number...

NOTE : Don't confuse this JNOS port number 1236 with any of the port numbers assigned on the Linux box in the /etc/services file. It is unique to the JNOS application when binding the socket connection to the JNOS IP address, and will not conflict with any services on Linux.

If you have done all of the above, the Trace Server will have started up OK, and will be waiting for a connection on its assigned port. However, there is no immediate evidence that the server is running! So when you do a "sockets" or "so" command from your admin session, you won't see anything yet in the server list. You will only see the Trace Server after you have actually made a connection to it!

Making the Connection

So, it is all in place and ready to go. You should test it out to make sure it is really working. Use your Alt-F keys to find another login screen, and login as root or anyone. Then enter:

• telnet "the_JNOS_host_IP" 1236

where "the_JNOS_host_IP" is the IP you assigned to JNOS when you set it up. Now you may issue your "so" command from an admin screen and you will see the remote host, its port number, and the server name, Trace Server, in the far right column. This verifies the activation of the server! Here is a sample excerpt from my socket list...

• 138 TCP         080f7e28  44.56.26.12:1027       080f3a88  Trace Server

At this point, you may enter all the commands that control trace. For example, you may enter the command "trace." This will show what trace commands had been given prior to your remote login. You may also now begin to issue new trace commands as though you were at the admin console. If you do not enter any commands, then nothing will happen. You must make a "new" request in order to begin viewing data, i.e., "trace ax0 311" ... and, it only accepts trace-related commands, all others are rejected.

You now control trace from this new "position." If you previously set trace commands in the autoexec.nos, for example, they are now over-written by you in this remote session. (Recall that the server is non-concurrent, so it can only serve one client at a time! If someone enters the Trace Server after you, then they will have the control and the output for the selected interface. Your data flow on that interface will be terminated... Last person in, owns it! (There is no direct sharing on this server as it has been implemented. However, sharing is possible by using special techniques, see below under "Sharing Data"...)

But that aside, let's say you want to set up a trace on the interface ax0. The command would be just the same as if you had entered it in the autoexec.nos:

• trace ax0 311

You may, of course, set all the interfaces from this session. Or, you may just set up one to view that single interface.

If you abruptly break the session, the "so" command will show that the Trace Server is not active, not on the socket list anymore. And this will automatically turn trace off for the interface(s) you had previously activated on this connection. However, you may also just turn trace off and still remain connected to the server:

• trace ax0 off

The trace for that interface is now turned off as indicated by a "trace" command from an admin session. But, you are still connected to the server as indicated by an "so" command.

Multiple Interface Tracings

Here is where things can get interesting. You may "open" more than one copy of the Trace Server! So, suppose you wanted to monitor another interface, ax1, for example. You would use the same telnet command to do this, and from a new login session enter:

• telnet "the_JNOS_host_IP" 1236

and, if you looked in the socket list, you would see the new host and port, which constitute the new socket on the remote host. You would want to "control" only the ax1 interface, so using the same command syntax as before, enter:

• trace ax1 311

which directs output from the ax1 interface. Recall, that you could shut off the ax0 interface from here too. But, playing by these rules, one interface per one session, you will now monitor just the ax1 packets in this "new" login and telnet session. (If you had an ax2 interface, you could set up a session for that one too, etc.)

What is the point of all this? Well, if you are serious about monitoring packets, breaking them down by interface can make the data flows much more readable. If you can jump to each window or session screen and track only one interface at a time, it is much easier to analyze the activity on the frequencies, easier to read BEACONs or IDs, etc. I only run a few interfaces here, but when things get busy, the packets fly by in an unreadable blur. When I assign each interface to a window or session, then I can really see what is going on! (This is especially true when I am troubleshooting one interface. I don't get "interference" from all the others.)

Tracing to an Output File: Sharing Data

Just as the usual trace function can write to a file, so too can the Trace Server! If you have a reasonable amount of space on your hard drive, you may re-direct the output a regular file. Make a telnet connection to the Trace Server as specified above, and then issue this command, or your variation:

• trace ax0 311 /root/ax0trace

Where you locate this file is your choice. You might choose instead a path such as, /tmp/ax0trace. It may be a good idea to name the file so that its content is clearly evident. I just choose ax0trace as an example. Make sure the interface, ax0, and the file name, ax0trace, are "in sync." (This is a pesky detail to watch out for, i.e., does the ax0 data go to the ax0trace file?)

Now here is where UNIX/Linux can really show its "stuff" and makes at least one compelling argument for running JNOS on Linux. At any given time, a file may have one writer but any number of readers. Combine this concept with that of "tailing" a file and you have setup a situation where you may now share this data publically. Tail has a very nice feature using the "-f" switch which indicates that you want to "follow" the output of the file in real time. Anyone who logs in to your system, locally or remotely, may issue this command:

• tail -f /root/ax0trace

and monitor packets from that interface file. This setup also acts like a "data diode" preventing casual viewers from tampering with the trace commands since none can be given through a "tail." And, of course, viewers may tail any number of ax[0-9]trace files in separate login sessions, provided you have set it up that way.

If you are running on a very busy node, or one that doesn't have too much hard drive space, you might think about running a "watchdog" script that checks on the size of these files and resets them to zero when they reach a pre-set maximum. You might miss one packet during the reset, and tail will "fuss" with a message that tells you the file has been "truncated." (Or you could do the same thing from cron, scheduling a script to run perhaps every few hours that would also clear the file.) If you can live with this minor glitch, you have built yourself a very reliable data handling system which can share packets with anyone who might be interested.

COMMENTARY : Now I hear you saying, "If someone knows about telnetting to the server, couldn't they 'preempt' the tail strategy and wreak havoc with the system?" And the answer is probably, yes. Whenever you want to share something, you run a security risk, as you know! The remote access for the Trace Server was, and is, intended for restricted, administrative use. So either be cautious or inventive, or both. :)

Data Analysis

If you enjoy programming in Java, Perl, or even C/C++, you can write code that will make a direct connection to the Trace Server! You will need to issue at least one command to instruct the server to trace on a selected interface. The server uses the telnet protocol which requires a "\r\n" sequence at the end of every line. So in Perl, for example, your command line to the server, after you have set up the socket connection, would be similar to this:

• print SOCK "trace ax0 311\r\n";

The server will respond with "ax0: input output (monitoring)" and when it does, you know you are ready to begin receiving data. (The "SOCK" in the print line above is the handle name for the socket. You may use whatever naming conventions you like.) Here is some sample Perl code for the "listening" and appending loop:

#----- Open file for continuous append open(AX0TRACE, ">>/root/ax0trace"); ... #----- Loop, listen and write if data ... while( defined($dataline = <SOCK>) ) { print AX0TRACE "$dataline"; }

Why would you want to do this? Well, you could replace the "write to a file" command line described above with your Perl/Java/C code that will do the same thing. It will read from the "SOCK" handle and write to any number of outputs, among them STDOUT, or AX0TRACE, or any number of file handles you choose. (If you are running a slow machine, you may have to limit the number of files written to... )

Using this technique, you may write to more than one type of file "simultaneously." You might want to design a filtering routine that would only record BEACONs or IDs, each in its own file. (You could even write to a fifo, named pipe file, but this gets a little trickier on the receiving end.) A common application for this type of "data mining" would be as a feed for a database system... There are many possibilities! If you are a programmer, and have been around Linux for a while, I am sure you can think of some interesting applications...

By way of an update, there is a Perl code module named abscopy.plx which has been developed as a packet filter and receiver based on the ideas presented in this article. (It has been designed to output data to a single file, but it could be output to a pipe file and shared among users on the same machine.)

Concluding Comments

In this very brief article, I have tried to document the Trace Server and suggest how you might use it on your system, highlighting both its limitiations and its potential. You should read all the information about trace in the user documentation that came with your JNOS package since this commentary is meant to serve as a supplement to that resource. (Get comfortable with trace before you jump in at the deep end with the Trace Server.)

Once you are confident about the trace commands and their applications, which are really quite straightforward, you can then move on to experimenting and "testing" the real power of the Trace Server and how you might want to apply it on your system. "Have fun," and let me know how you are doing... :)