Import-Export Overview

Newly created for EW v7.0 is export_ack, export_scnl_ack, and import_ack for acknowledgement handshaking.

The modules import_generic, export_generic, and export_scn represent the Earthworm long-distance communication modules that provide a rapid, secure message exchange facility between two or more Earthworm or other processing systems. Any link capable of supporting the TCP/IP protocol suite can be used. The philosophy underlying these modules is that they implement a static, long-term linkage between pairs of cooperating installations. It is presumed that this linkage has been agreed upon by both institutions, and that both installations establish matching configurations that provide reasonably good assurance that only the designated machines are participating in the linkage.

Export_generic accepts a list of message logos (installation id, module id, and message type) to export. Any messages with matching logos will be queued in a memory based FIFO, and sent to the receiving "import_generic", or similar program, via a TCP connection. If the available bandwidth of the link is lower than the aggregate rate of messages being queued for shipping, the FIFO will eventually overflow. In that case, an error is logged, and the queue becomes circular. That is, if the queue is configured to be N messages long, it will contain the most recent N messages. The intent is to avoid shipping uselessly old messages such as when two systems exchange phase picks or trace data for 'real-time' event detection.

Import_generic receives such messages, and releases them into its local system under the logos with which they were sent. Other modules in that Earthworm may then selectively examine or ignore such messages by specifying which installation id's they are interested in.

Both import_generic and export_generic offer 'stub' routines through which all messages pass. The intent is to provide an easy place to insert custom code to process messages before they are sent out (in export_generic), and after they are received, but before they are released into the local system (in import_generic). Examples of such filtering may be to send only relevant picks to a neighboring network, or to change formats to accommodate another system. The intent is that the insertion of such code will result in a new module with a name of the form import_xxx or export_xxx.

One such specialized version of export has been created: Export_scn is a modified version of export_generic that specializes in sending trace data. Its configuration file differs in that the user can specify the station, component, and network names of the traces to be exported.

A further modification of export_scn is export_scn_pri. This version offers the additional capability to specify a priority for each station, component, and network name combination to be exported. Export_scn_pri always transfers the highest priority data before lower priority data. This feature is intended to improve system throughput during high-volume events or periods of network degradation.
To simplify the adoption of export_scn_pri, onfiguration (".d") files are interchangable between it and export_scn. Export_scn_pri will read priorities from the command lines specifying the station, component and network names, or will use a default priority (lowest) if not specified. Export_scn will ignore any priorities that are specified.
Export_scn_pri uses a prioritized queue instead of a circular queue. If bandwidth limitations arise, lower priority messages are dropped or rejected before higher priority messages.

Two-way heartbeats are used to assure that the link is actually capable of transmitting data. The heartbeats consist of agreed-upon text strings sent at an agreed-upon rate. If a proper heartbeat is not received in time, it is presumed that the link is broken. Both Import and Export then log the error, close their connections, and attempt to re-establish the link.

It has been found that socket-based programs which offer (rather than accept) a connection are sometimes unable to re-establish a link after a break has been detected. To address this, and other 'massive failure' modes, a feature has been added to export which causes a restart of the module in such a manner that its previous network environment is not re-inherited. This is done by inserting the token "restartMe" in the error descriptor file of the sender (e.g. 'export_generic.desc'). Then if the statusManager module detects a lack of heartbeat form export, a request is made to start_stop to kill and restart the module.

Neither export_generic nor import_generic change the logos of the messages being transferred. So it is important to make sure that logos being imported have unique logos: installation id, module id, and message type. If the imported logos match any of the existing logos, you will have serious problems with earthworm! The message transport system will get very confused by these matching logos. Keep this in mind when you have more than one earthworm system with the same installation id that share data through export_xxx and import_xxx. Since the installation id's and message id's (e.g. TYPE_TRACEBUF) will match, the module id's on these two machines must be different for the modules that are creating the messages.

export_gen_actv, import_pasv

Export Generic makes available on a certain TCP/IP port at a certain IP address an export of data, and Import Generic makes a connection to that exporting machine/ IP address/ port to retrieve that data. In some cases this is difficult to configure because of firewalls, where the exporting machine is behind a firewall and can't receive incoming connections. In a situation like that, you can use export_gen_actv and import pasv. Export Generic Active exports data (as described above for Export Generic), but goes one step further by trying to drop off that data onto a specific remote machine and port. That remote machine should be running Import Passive, which sits and waits at a specific port and IP address for an exporting computer to connect start dropping off data. This moves the firewall issues to the importing machine from the exporting machine. In this case, the importing machine will require the ability to receive remote connections at a specific port; this may necessitate opening the IP address and port in the firewall/router.

NEW: export_ack, export_scnl_ack:
  This new version of export expects to receive an acknowledgment packet  for every packet it writes to the socket. Creation of this new version  was desired because we observed during the export of continuous trace data that during a communication failure, export was able to successfully write several messages to the (broken?) socket, but import never received them. When the socket was re-established, those messages were lost forever.

  Acknowledgments are received asynchronously by the socket-reading thread.  A circular buffer, SendQueue, tracks the status of msgs after they been  pulled from the MessageStacker queue. If the socket is broken/reconnected,  export*ack re-sends any msgs in the SendQueue which haven't been  acknowledged. In normal operation, if export*ack laps the SendQueue and  finds a message that has been sent but not ACKed, it will sleep 10ms and  then check status again, looping until an ACK is received before preparing  to send the next message.

  The SendQueue length is configurable, from 1 to 254 messages (default=100).   Using a shorter SendQueue can cause a decrease in the rate at which export will send messages. A slower rate might be desireable so  that an import system is not flooded by data after a prolonged  communication break. In testing with export and import on the same  machine, these are export's approximate maximum data rates:

Using the default SendQueueLength, export*ack's data rate was not observably different than export_generic's data rate. 

  export*ack requires using the new import_ack to communicate properly. Since the receipt of ACKs is asynchronous, we may now have the opposite  problem to that we had before (duplicate data instead of lost data). It  is now possible that import_ack actually received a packet, but export_ack didn't get the ACK before the socket broke. In this is case, export_ack
will re-send the packet and import_ack will see the same packet more than  once. The system running import_ack must be prepared for duplicate data.

import_ack:

This new version of import is required as the data exchange partner for the new export*ack modules. However, it can communicate with either  old or new-protocol export modules, and can therefore be used anywhere in place of import_generic.

From the first message received per socket connection, import_ack  discovers what flavor of export it's talking to. It will do either
old non-ACK or new ACK protocol as appropriate on the fly.

Import_ack has been modified to use same configuration commands as export for setting up the server connection and socket heartbeat strings and intervals. It still recognizes the original import_generic commands. 

Since the receipt of ACKs by export_ack is asynchronous, we may now  have the opposite problem to that we had before (duplicate data instead  of lost data). It is now possible that import_ack received a packet,  but the socket broke before export_ack received the ACK. In this is case,  export_ack will re-send the packet and import_ack will see the same packet more than once. Therefore, the system running import_ack as a partner to export_ack must be prepared to handle duplicate data.