Newly created for EW v7.0 is export_ack, export_scnl_ack, and import_ack for acknowledgement handshaking.
The modules import_generic, export_generic, and export_scn represent the Earthworm long-distance communication modules that provide a rapid, secure message exchange facility between two or more Earthworm or other processing systems. Any link capable of supporting the TCP/IP protocol suite can be used. The philosophy underlying these modules is that they implement a static, long-term linkage between pairs of cooperating installations. It is presumed that this linkage has been agreed upon by both institutions, and that both installations establish matching configurations that provide reasonably good assurance that only the designated machines are participating in the linkage.
Export_generic accepts a list of message logos (installation id, module id, and message type) to export. Any messages with matching logos will be queued in a memory based FIFO, and sent to the receiving "import_generic", or similar program, via a TCP connection. If the available bandwidth of the link is lower than the aggregate rate of messages being queued for shipping, the FIFO will eventually overflow. In that case, an error is logged, and the queue becomes circular. That is, if the queue is configured to be N messages long, it will contain the most recent N messages. The intent is to avoid shipping uselessly old messages such as when two systems exchange phase picks or trace data for 'real-time' event detection.
Import_generic receives such messages, and releases them into its local system under the logos with which they were sent. Other modules in that Earthworm may then selectively examine or ignore such messages by specifying which installation id's they are interested in.
Both import_generic and export_generic offer 'stub' routines through which all messages pass. The intent is to provide an easy place to insert custom code to process messages before they are sent out (in export_generic), and after they are received, but before they are released into the local system (in import_generic). Examples of such filtering may be to send only relevant picks to a neighboring network, or to change formats to accommodate another system. The intent is that the insertion of such code will result in a new module with a name of the form import_xxx or export_xxx.
One such specialized version of export has been created: Export_scn is a modified version of export_generic that specializes in sending trace data. Its configuration file differs in that the user can specify the station, component, and network names of the traces to be exported.
A further modification of export_scn is export_scn_pri. This version
offers the additional
capability to specify a priority for each station, component, and
network name combination
to be exported. Export_scn_pri always transfers the highest priority
data before lower
priority data. This feature is intended to improve system throughput
events or periods of network degradation.
To simplify the adoption of export_scn_pri, onfiguration (".d") files are interchangable between it and export_scn. Export_scn_pri will read priorities from the command lines specifying the station, component and network names, or will use a default priority (lowest) if not specified. Export_scn will ignore any priorities that are specified.
Export_scn_pri uses a prioritized queue instead of a circular queue. If bandwidth limitations arise, lower priority messages are dropped or rejected before higher priority messages.
Two-way heartbeats are used to assure that the link is actually capable of transmitting data. The heartbeats consist of agreed-upon text strings sent at an agreed-upon rate. If a proper heartbeat is not received in time, it is presumed that the link is broken. Both Import and Export then log the error, close their connections, and attempt to re-establish the link.
It has been found that socket-based programs which offer (rather than accept) a connection are sometimes unable to re-establish a link after a break has been detected. To address this, and other 'massive failure' modes, a feature has been added to export which causes a restart of the module in such a manner that its previous network environment is not re-inherited. This is done by inserting the token "restartMe" in the error descriptor file of the sender (e.g. 'export_generic.desc'). Then if the statusManager module detects a lack of heartbeat form export, a request is made to start_stop to kill and restart the module.
Neither export_generic nor import_generic change the logos of the
messages being transferred. So it is important to make sure that logos
being imported have unique logos: installation id, module id, and
message type. If the imported logos match any of the existing logos,
you will have serious problems with earthworm! The message transport
system will get very confused by these matching logos. Keep this in
mind when you have more than one earthworm system with the same
installation id that share data through export_xxx and import_xxx.
Since the installation id's and message id's (e.g. TYPE_TRACEBUF) will
match, the module id's on these two machines must be different for the
modules that are creating the messages.
Export Generic makes available on a certain TCP/IP port at a certain IP address an export of data, and Import Generic makes a connection to that exporting machine/ IP address/ port to retrieve that data. In some cases this is difficult to configure because of firewalls, where the exporting machine is behind a firewall and can't receive incoming connections. In a situation like that, you can use export_gen_actv and import pasv. Export Generic Active exports data (as described above for Export Generic), but goes one step further by trying to drop off that data onto a specific remote machine and port. That remote machine should be running Import Passive, which sits and waits at a specific port and IP address for an exporting computer to connect start dropping off data. This moves the firewall issues to the importing machine from the exporting machine. In this case, the importing machine will require the ability to receive remote connections at a specific port; this may necessitate opening the IP address and port in the firewall/router.
NEW: export_ack, export_scnl_ack:
This new version of export expects to receive an acknowledgment packet for every packet it writes to the socket. Creation of this new version was desired because we observed during the export of continuous trace data that during a communication failure, export was able to successfully write several messages to the (broken?) socket, but import never received them. When the socket was re-established, those messages were lost forever.
Acknowledgments are received asynchronously by the
socket-reading thread. A circular buffer, SendQueue, tracks the
status of msgs after they been pulled from the MessageStacker
queue. If the socket is broken/reconnected, export*ack re-sends
any msgs in the SendQueue which haven't been acknowledged. In
normal operation, if export*ack laps the SendQueue and finds a
message that has been sent but not ACKed, it will sleep 10ms and
then check status again, looping until an ACK is received before
preparing to send the next message.
The SendQueue length is configurable, from 1 to 254 messages
(default=100). Using a shorter SendQueue can cause a
decrease in the rate at which export will send messages. A slower rate
might be desireable so that an import system is not flooded by
data after a prolonged communication break. In testing with
export and import on the same machine, these are export's
approximate maximum data rates:
Using the default SendQueueLength, export*ack's data rate was not
observably different than export_generic's data rate.
export*ack requires using the new import_ack to communicate
properly. Since the receipt of ACKs is asynchronous, we may now have
the opposite problem to that we had before (duplicate data
instead of lost data). It is now possible that import_ack
actually received a packet, but export_ack didn't get the ACK before
the socket broke. In this is case, export_ack
will re-send the packet and import_ack will see the same packet more than once. The system running import_ack must be prepared for duplicate data.
This new version of import is required as the data exchange partner
for the new export*ack modules. However, it can communicate with
either old or new-protocol export modules, and can therefore be
used anywhere in place of import_generic.
From the first message received per socket connection,
import_ack discovers what flavor of export it's talking to. It
will do either
old non-ACK or new ACK protocol as appropriate on the fly.
Import_ack has been modified to use same configuration commands as
export for setting up the server connection and socket heartbeat
strings and intervals. It still recognizes the original import_generic
Since the receipt of ACKs by export_ack is asynchronous, we may
now have the opposite problem to that we had before (duplicate
data instead of lost data). It is now possible that import_ack
received a packet, but the socket broke before export_ack
received the ACK. In this is case, export_ack will re-send the
packet and import_ack will see the same packet more than once.
Therefore, the system running import_ack as a partner to export_ack
must be prepared to handle duplicate data.