SYSOP Documentation for GNUAres.  R. B. Morgan WB5AOH 6/13/2003

GNUAres is a replacement for ARESDATA.  It is a database for use on
amateur radio packet data networks, and is useful for handling situatons
requiring a networked database.  It is typically used for emergency response
to widespread calamities.  This program, and its documentation are free
software, as defined under the terms of the GPL, explained elsewhere.

NOTE: This program is still under development and testing.  Some of the
commands and procedures have not yet been written, and others are not fully
tested.  It is likely that some of this will also change slightly from what
is described below.  As of mid 2003 it is considered alpha level software.
Same caution exists for the documentation.

SYSOP info and configuration

1. This program is intended to run in an environment of a Linux system
whose kernel includes AX25 socket library and some other applications
including the AX25 node (OH2BNS?), and also has network access to Postgresql.
This program is required to be linked against the Postgresql library libpq,
whether or not Postgresql is installed on the target machine, so that
usually means either installing postgresql or its runtime libraries on
all machines that this program is installed on.

The program gnuaresd (to really become a real daemon later) runs as a
very unprivileged user.  A separate user needs to be created with very ordinary
rights, and a password assigned for a sysop to manage the system with,
primarily for the purposes of database tweaking, backup/restore, and so
forth.  This user will need to be defined to the postgresql database with
read/write privileges on database gnuaresdb.  See postgresql config file
pg_hba (?) which needs a permissions entry made in it.  This program should
never be run as root, and there certainly is no reason to ever do so.
This program needs to be started in system boot scripts in such a way
it will have only the rights of that user.

2. It is assumed that one or more AX25 radio transceivers are somehow
connected to each machine running this software, and that AX25 users can
connect to them and reach the AX25 node program either as local or as network
connects.  It is also possible to reach this program remotely via a telnet-like
tcp-ip connection (whether local or remote, all connects from the linux node
program to this one are done this way).  Each user of the AX25 node initiates
one instance of the node program when it connects to the node target callsign.
However, there is only one instance of this Gnuares program running on each
AX25 server, and it runs all the time, regardless of the number of users if
any.  The gnuaresd program makes one connection to the Postgresql daemon,
called postmaster, and in so doing initiates one instance of the postgresql
backend that will service gnuaresd for as long as gnuaresd is running, and
manages to avoid accidental disconnection due to abnormal network events.
Hopefully inetd, postmaster, ax25d, and gnuaresd, along with an instance of
the postgresql backend, will all be running from boot time until the system
is brought down for whatever reason a year or so later.  Instances of the
node program come and go as users come and go.  Gnuaresd also will try to
connect to aprsd (the APRS networking daemon) either locally, remotely on
the lan, or some aprsd available on the internet worldwide, for the purpose
of scanning, capturing and exchanging messages and events, and that connection
will also stay up for the duration, or as network events come and go.
In summary, gnuaresd initiates one connection to APRSD, one connection
to postgresql, and receives on demand individual connections from up to
as many as (currently 35) users via tcp-ip connections from instances of
the AX25 node programs, one per user.  In this way, gnuaresd is acting as
a sort of funnel for connections, needing only one instance of itself, and
also demanding only one instance of the postgresql backend, resulting in
considerable memory savings.  Enough memory must exist to support multiple
instances of the AX25 node, and to buffer up a lot of packets when there
are many users.

The AX25 node program seems to have a memory leak of 8K per instance
of a tcp-ip connect to gnuaresd.  The kernel appears to always reclaim
the leaked memory when the AX25 user disconnects from the node program
which then kills the node program off.  If for example, a user connects
to node, and connects and disconnects to ARES 3 times while connected
through node, 24K of memory will leak out, but should be reclaimed when
the user finally disconnects from node.  At the present time, this bug
isn't seen as detrimental to operations.

This program is intended to work on a lan on which there exist one or
more AX25 packet radio equipped servers, and one server on which the
database itself resides.  The database could reside on the same machine
as one of the AX25 nodes, or elsewhere on the lan.  There will be one
instance of gnuaresd running in each AX25 server, and there will be one
instance of the postgresql backend running on the database server for
each AX25 server that is assumed to be continuously connected to the
database.  In this way it is possible to support possibly a hundred or
more simultaneous users, spread out over many channels to make the
channel loading somewhat tolerable.  A typical challenge at such a server
site would be to manage the adjacent channel interference and blocking
that would occur with several AX25 transceivers running in close proximity.
One solution is to implement more than one site, and maintain some kind
of system of reliable network trunks that can place some of the local RF
connectivity at some moderate distance from others like itself.  Keep in mind
that the intended purpose of this system is to function in an emergency, so
something other than normal commercial network service is required for
supporting such an expanded lan.  Such an expanded lan is under construction
where this program is being alphatested, and will include several different
network paths between sites such as higher speed AX25 on UHF network trunks,
wireless ethernet, commercial ethernet, in addition to the normal 1200b
AX25 networking.

It is also possible to install a standalone system containing AX25 radio
ports, gnuaresd, aprsd, postgresql, httpd and so forth on one machine and
run one or more AX25 radio ports, and optionally connect it to the internet,
or optionally connect it to some other lan located at an EOC that could be
used for the agencies to interrogate its database.  Obviously such a system
can be fairly easily expanded over time by adding AX25 packet servers.

It is also possible to install a packet BBS such as FBB on the same system,
and it should be possible to configure FBB to do some file handling on
behalf of gnuaresd with the file processing extensions of that program.
For the time being, it is also likely that the PBBS might be used to
handle the workload of binary file transfers for the users of the database.

ARES users should seriously consider the advantages of placing a copy of
a forwarding PBBS (even a simple one) into the field with users, in
addition to using them at central server sites.  If a user has a BBS in
the field, and it is configured to autoforward messages to the central site
at something like 1-2 min intervals, the user only has to compose a message
or file at the console window of the BBS.  When saved, it will be sent
automatically without the user having to take the time to interact with
the TNC, networks, and the servers.  The communications will take place in
the background while the user can enter more data on the next message.
Eventually the design of gnuaresd will include more BBS interoperability
as time permits and specific applications turn up during field testing.

3. File gnuaresd.conf is the configuration file required, and there
is of course one of these per machine.  It is assumed to reside in /etc.
It is read in its entirity when gnuaresd starts up, and is re-read when it
receives a SIGHUP (when that code gets written later), for some of the
parameters that are considered alterable while it is running.  Not all
of the parameters make sense to be changed during operation, such as for
instance the location of the database.  For that, gnuaresd must be brought
down and restarted after making those changes.  Comments in the .conf file
make clear which category is which, and for now, most of the documentation
for gnuaresd.conf is internal to that file itself.

4. In addition to the C source files, the .conf file and the documentation,
the distribution package includes a skeleton database restore file that
is loaded as a batch submittal via stdin redirection to psql, which is the
general purpose sysop interface to postgresql.  After postgresql is installed
and tested and found to run, and configured as necessary, it is necessary
to run the posgresql utility:
createdb databasename
where databasename is the name of the database found in gnuaresd.conf and
is usually: gnuaresdb
That will create an empty but connectable database.  Then it is necessary
to load it with the table descriptions and SQL logic it will need to run.
That is done as follows:
psql < aresdb.inp
where aresdb.inp is a flat ascii file furnished with the source, that
will initialize the database itself and install all of the tables and
rules.
After this is done, it is necessary to cause gnuaresd (and also aprsd if it
will be used) to run now, and thereafter when booted, and gnuaresd should
connect to the database.  Running ps aux should show one instance of
gnuaresd (and a lot of various instances of aprsd if used), and one instance
of the postgresql backend along with postmaster.  As users come on line,
ps aux should show instances of node for each user.  If the database is
located on some other server, ps aux has to be run on that server to see
the instances of postmaster and the postgres backends for each machine
running gnuaresd.  If you don't account for all of them, something isn't
running or connected where it ought to be.  When all looks well, it ought
to be time to try some AX25 user connects to the database.

5. A few words about planning a database system.  About the only limits
on postgresql are diskspace and memory.  In particular, be sure to provide
a big enough data partition to hold potentially thousands of rows on
several tables, if you anticipate serving a large metropolitan area.
Postgresql normally uses /var/lib/pgsql/data/base/ as a path to the
data.  It is possible to reconfigure the database location, and also
possible to add an alternate location, to any partition and path you
choose.  Of course, enough room has to be planned in advance of the time
that the disk drive is partitioned.  If you are planning a high reliability
system, and have a hardware budget, you ought to consider something that
is both large and fault tolerant, such as a raid.  Then you have to configure
postgresql to find that block of storage space.  There is guidance elsewhere
associated with both postgresql and with various hi-rel disk hardware that
you can find and use.

To periodically back up the data in postgresql, it is necessary to do either
of the following:
a. bring down instances of gnuaresd to ensure there are no transactions in
progress, and then run the postgresql utility:
pg_dump gnuaresdb
to create a current ascii flatfile save of the database
(it is reloaded from stdin through psql if a restore is necessary), or
b. Perform whatever kind of a disk snapshot the raid drive requires to
generate a snapshot image of the data.
And in either case, save the files for emergency reload as and where necessary,
and in case a, to restart the things that were shut off temporarily.
c. There is a commercial product that can be used to snapshot a running
postgresql, and is available (someplace, Stu Green knows).
d.  Somewhere in the future, there will probably be a sysop command that
will cause the database to pause and write a backup file at such time as
no transactions are in progress, which might interrupt the database
data processing for something like a few seconds at most.

6. This program was developed and tested on systems using kernel 2.4.4 which
was loaded from a SuSE 7.2 distribution, which also included postgresql 7.04?

The testing was done with Austin (Texas) Travis County ARES, which works
regularly with the city/county EOC which is one of the probably 10 best
organizations of its kind anywhere around, and regularly exercises the
packet data handling operations of ARES to provide emergency communications
during its periodic drills.  Quite a bit of the communications is done
in conjunction with a program names ARCHES, which has placed amateur radio
and packet equipment and portable computers into the emergency response
departments of local hospitals that operate ER's and are likely to receive
many casualties during a mass casualty disaster.  Many functions of this
database, including the shortcomings of the earlier ARESDATA program, were
built into gnuares specifically as a result of the outcome of the training
and drills that the ARCHES program has triggered.  Some of the design of
the potential size of the system, such as number of users, number of rows
in the database, various standing databases and scenarios, and so forth
were also based on personal observations of the 9/11 attack disaster
whih among other things seemed to produce a missing persons list of several
thousand.  Had there been survivors, this gives a clue to the size of
the hospital admissions lists and the time frame and number of sites that
they might be entered over.

7. Internally, this program makes extensive use of poll(2) to juggle dozens
of connects.  Simple arrays in memory are used to store various connect-time
parameters of each connect, including callsign-ssid, socket number, and
so forth.  Other data describing each connect is actually committed to the
database itself, such as site ID, a pointer to last conference message
received, nets and databases joined, number of times connected since first
noted using the system, and so forth.  Each user is serviced one at a time
as encountered by packets received by poll(2) that trigger work to do in
the program.  If a user requests voluminous output, that output is sent
to (something not yet written) that can parcel it out to that user at some
controlled rate while other users get some chance to use the database, and
also prevents all of the i/o buffers and radio bandwidth to be overwhelmed.
Otherwise, each user's work is completely performed before poll(2) is
interrogated again to look for more input from other users that will trigger
more database work.  As far as the database is concerned, database transactions
are initiated and completed in their entirety one outstanding access at a time.
Poll(2) demands that as users come and go asynchronously, and inactive slots
develop in the arrays that poll(2) is juggling, that the arrays be re-packed
to present an unbroken array of valid connect sockets prior to the next time
poll is called, and that necessitates some additional juggling of the contents
of the arrays, and a requirement for another array to keep track of the arrays.

8. All of the transactions are logged to a logfile.  There is also a separate
logfile for APRS traffic.  A new file for each is created at mignight of
each day.  Some files could be empty if there is no daily activity.
The path/file names of both logfiles are found in gnuaresd.conf.  When
configuring these paths, create directories in /var for the paths specified
in the .conf file, and make sure they are writeable by the unpriveleged
user that the daemon gnuaresd is running as.  For the present, the contents
of the logfiles is simply the user input, but more than this needs to be
added.  For one thing, logs typically show what is transmitted in the way
of traffic, and this initial system isn't there yet.

9. Configuration checklist: (roughly in order.  Start with diskspace).
a. disk space partitioning for database storage, make sure that the
   partition intended for the database is large enough.  If you like, you
   could install, partition and format an auxiliary drive on an existing
   system, and point postgresql at that new partition directory.
b. kernel options (include ax25 for instance), recompile and install/test
c. system program installs, such as postgresql, libax25, node, httpd, gcc,
   other amenities such as FBB, APRSD (maybe want ax25-sockets version)
d. general system and network config (address, hosts, routing, mail, etc)
e. AX25 general config for all attached radio transceivers, etc.,
   and enter changes to node.conf to point ARES at gnuaresd connect port.
f. APRSD config if used.
g. FBB configuration if used.
h. create users
i. Postgresql config, and don't forget to mention gnuaresd user in permissions
   as a trusted user, either at localhost, or at wherever host(s) connect to
   it.
j. Gnuaresd.conf. Create logfile path directories also.
k. Load gnuaresdb with skeleton database.
l. Startup scripts.  During boot, after networking is loaded, bring up things
   like httpd and postmaster (postgresql connect daemon), and after standard
   amenities are run, first bring up the layer 2 parts of AX25, such as
   things like kissattach, bpq and so forth, then layer 3 such as netrom,
   then such things as ax25d.  After ax25d is running, aprsd and gnuaresd
   can be started (start aprsd first since gnuaresd will want to find it).
   Sometime after ax25d is running, fbb can be started if used.

10. Specific configuration info:

For /etc/ax25/node.conf it is necessary to add an ALIAS line to support
tcpip connections to gnuaresd, when the user types the command word ARES.
The syntax is a little tricky where the quotes and escape sequences are
concerned.  If necessary you can change such things as the alias name,
the target host and port and so forth to match the config and location
of gnuaresd.  It is:

Alias  ARES 'telnet localhost 4161 \"NewCon CRLF \%U \%S \%P \%N \%T \%h \%I\"'

In the above example, localhost is the host that gnuaresd is running on, and
4161 is the port number that gnuaresd is listening on for user connects.
Note that node translates ax25 users to/from tcp-ip, and the connection to
gnuaresd is running something like a telnet session.  (You could telnet to
gnuaresd for testing.  If you do you have to send as your first line the
expanded version of the arguments that follow, so that the things like
callsign, port, node, etc are filled in.)  All the stuff following the
port number is escape codes that node intreprets and fills in with the
callsign and other ax25 info.  The word NewCon is a hardwired command to
gnuaresd, that works once and only once at the front end of a tcpip connection
to tell the database the identity of the user connecting.  See the manpage
for node.conf for more info on the alias statement and the escape sequences.

If gnuaresd is installed on some other host than the packet server running
node, then of course you would change localhost to the address of that host.
On a small lan you would probably be better off just using the address itself
in case there is a nameserver problem, but on a small lan it would really make
more sense to install a copy of gnuaresd on the same machine(s) as the packet
transceiver server(s) themselves, and point gnuaresd.conf at the common
database server.

If you had a larger system that for some reason needed more than one database
server (which at this time would be independent since there is no database
sharing or auto failover designed yet), then it would make sense to create
two instances of the alias, such as ARES1 and ARES2 or something similar
that would be recognizable to the users, and point each at separate hosts
of gnuaresd.  If it was necessary to configure a packet server for more than
one instance of gnuaresd, then at this time it would be necessary to compile
them separately under different runfile names, and change the internal coding
to point at different copies of the .conf file, and also specify different
tcp-tp connect port numbers, and set up the alias lines to point at the
different ports so compiled (or configured).  At some point, gnuaresd should
have a commandline option to specify a non-default .conf file, and the listen
port should be in the .conf file as a parameter, and at that time the need to
compile separate instances of gnuaresd will disappear.  If you do need to
run more than one instance of gnuaresd on the same host, it would be well
to consider and rethink the number of AX25 connects available.  The arbitrary
number of 35 (?) was arrived at by noting that I *think* that the standard
kernel ax25 networking code allows something like 50 ax25 sockets to exist
on any one host.  Since some sockets will be used for other things, it seemed
reasonable to use around 2/3 of the available resources for gnuaresd.  If
more than one copy of gnuaresd runs, it might be necessary to lower the 35
to maybe about half that.  Or maybe track down the ax25 kernel limit and
change it (and recompile/reinstall the kernel).  Or rethink the design of
the site.  The larger problem will be making all of that adjacent channel
RF behave at any one site.