Earthworm Release Notes V6.3
(Sept 26, 2005)
Release Notes: Earthworm Version "working"
Cleaned out after release of v6.2 on 4/15/03
Modified Exlusively by Paul Friberg (PAF) after 4/15/2004
to build interim version 6.3 release. Between 4/15/2003
and 4/15/2004 the notes were updated as part of regular
bug fixes and mods so any notes during that time are
attributable to the original author. After 4/15/2004
any notes are from PAF with respect to incorporating
new code and bug fixes between v6.2 and v6.3.
IMPORTANT NOTE ABOUT SCN:
*************************
This will be the LAST SCN version of earthworm and
the v7.0 will contain Location Codes for all programs.
NEW MODULES:
***********
startstop_service:
(added to CVS by Paul Friberg 6/29/2005, authored by Mark Morrison)
From the README.txt found in the src/system_control/startstop_sevice:
startstop_service is identical to the old startstop_nt, except that
it runs as a Windows service. This means that the parameters are all the
same - read from startstop_nt.d - but that startstop isn't just executed
from a command window or via the scheduler. Note that this version is
taken from startstop_nt, so I haven't compiled or tested any of this under
Solaris or other systems. I believe it's generally felt, however,
that these modifications don't apply to Solaris.
q3302ew:
(added to CVS by Paul Friberg, authored by Hal Schehner of ISTI.com).
This is a EW module which pulls data from a Quanterra 330
digitizer and outputs trace_buf packets onto a ring, much like q2ew
does for older quanterras.
srparxchewsend - Symmetric Research Digitizer module in
data_sources for use with their PC-based A/D boards. For more info
see them at www.symres.com. This compiles under Linux and Windows,
but not yet under Solaris.
slink2ew - Chad Trabant contributed this application to pull data from
seedlink into Earthworm. The version provided with EW v6.3 is
version 1.2.2 last modified by Chad on 2005.07.14
Here's the slink2ew note from the v7.0 release of EW for future reference:
A seedlink client for acquiring data from seedlink servers (e.g. IRIS)
and converting to tracebuf2 (or optionally tracebuf). Much more robust
than liss2ew. Written by Chad Trabant of IRIS; modified for tracebuf2
and debugged by Chad Trabant and Jim Luetgert.
JHL 4/19/05
startstop_unix:
(created by Alexandre Nercessian), imported into Earthworm by PAF.
This version compiles and runs under Linux. It does not have correct
reporting of run time for a module, but otherwise works. It uses and
looks for a startstop_unix.d file (not a startstop_sol.d file as in the
previous version with ew6.2-lnx). PAF 7/21/2005
grf2ew - Generic Recording Format to earthworm. This is for DAQ systems
digitizers. This compiles under Linux and should under Windows too. I
only installed this in the release and did not test it. PAF 9/10/2005
MODIFICATIONS/BUG FIXES TO EXISTING MODULES:
startstop_nt:
Had a restart option added in by Murray McGowan of GSL (9/6/2005). This allows
a user to restart any module from startstop by typing "restart nnn" followed
by the module name or the process id.
startstop_unix/startstop_solaris:
Added in the same restart command as above. It's a nice feature to have. (9/7/2005)
libsrc/solaris/transport.c:
Added: #include
Function tport_bufthr() now returns NULL. WMK 3/22/04
libsrc/winnt/transport.c:
Added: #include
libsrc/util/kom.c
Changed: "static struct k_buf" to "struct k_buf" WMK 3/22/04
libsrc/solaris/sleep_ew.c:
Added: #include WMK 3/22/04
gaplist:
Added a "Total Gap" column to the output tables. "Total Gap" is the
total, in seconds, of the length of all gaps for a particular channel.
"Total Gap" gets reset to 0 when the day rolls over. Also, added a
column named "Dead Time", which is the elapsed time since a channel
died. If "Dead Time" is blank, the channel is currently alive.
"Dead Time" does NOT reset to 0 when the day rolls over. WMK 1/8/04
gaplist: Slight table reformat. WMK 1/20/04
********************************************
libsrc/winnt/sendmail.c
Increased BUFFSIZE from 200 to 250, to accomodate more email recipients.
This is a kludgy fix. It might better to allocate the buffer on the fly,
to accommodate any number of email messages. Also, note that I did not
increase BUFFSIZE in libsrc/solaris/sendmail.c WMK 6/11/03
evtK2ora:
Fixed endTime bug in makeEwSnippet(). Thanks Jim L.
Alex 5/9/03
shakemapfeed:
The last two arguments to ewdb_api_GetSMDataWithChannelInfo were
reversed. That meant that the number of returned SM messages would be
incorrect IF that number exceeded MaxDataPerEvent. The result would be an
out-of-bounds memory access and corrupted SM data. PNL 6/9/03
shakemapfeed:
Changed the sorting used in Filter_SM_data. Now SM messages are sorted by SCNL
instead of by idchan. That means that the output XML will have components of
one station listed consecutively, a big help for shakemap. PNL 10/30/03
shakemapfeed:
Changed the length of the long station name in the mapping file from
20 to 50 characters. LDD 2/5/04
skakemapfeed: (EWDB API - Strong Motion) ewdb_internal_GetAllSMMessages()
(Note: This function is called by ewdb_api_GetSMDataWithChannelInfo())
Fixed a bug in PostGetAllSMMessages(). The idEvent field may be null
for some records, and Oracle does not appear to be writing a 0 value to
the memory area for the null field, so that record assumes the
idEvent of the previous record located at that point
in the internal buffer. The result is data corruption, such that you get
messages that are unassociated with an Event, but they appear to be associated
with an event.
DK 02/12/04
shakemapfeed: Added new optional command, SMQueryMethod, to control how SM data
is requested from the DBMS. Default behavior is original 2-stage query (get
all SM data associated with this eventid, then get all UNassociated SM data
in a space/time box). If SMQueryMethod is non-zero, shakemapfeed will request
all SM data in a space/time box. This will return SM data associated with this
event, SM data associated with other events, and UNassociated SM data.
Regardless of query method, post-query filtering will be performed on eventid,
external eventid, and external author. SM data associated with a different
eventid but with the same author will be rejected. Data with a different
author will be allowed - this is most likely data imported from another
network and its eventids are meaningless in our DBMS. LDD 02/13/2004
shakemapfeed: Changed the auto-feed scheduling logic to allow for variable
time intervals between feeds to shakemap. Feeds are scheduled at
fixed times after the arrival of the archive message by using the new
command, ScheduleFeed (one time per ScheduleFeed command, up to 20 allowed).
Auto-feeds will continue at the last ScheduleFeed interval until UpdateDuration
minutes after the event origin time. ScheduleFeed replaces the original
DelayFirstFeed and UpdateInterval commands. These original commands are still
accepted and will set up the auto-feed schedule in the original manner in the
absence of ScheduleFeed commands. LDD 3/4/2004
k2ew:
The channel bitmap bug reported below has been fixed. K2ew now reads the
correct bitmap to determine which channels are streaming. PNL 6/9/03
liss2ew, ew2liss, dumpseed:
There was a bug on Microsoft based systems that caused the user defined
types BYTE, WORD, and LONG to be ignored. This was a problem specifically
with the type WORD, which was changed from a signed 16-bit value to an
unsigned 16-bit value, causing many problems, most noteably with the
signed seed varibles for sample rate and sample rate multiplier. All
liss based modules have had this problem corrected. JMP 06/17/2003
getfile_ew
There was a feature in getfile_ew that could cause more cpu to be
used than necessary by checking time since last heartbeat more
often than necessary. Added sleep_ew(1000) within heartbeat
loop. MMW 06/29/2003
getfileII
Ported getfileII to FreeBSD. Two new files are in the source directory:
makefile.bsd and socket_bsd.c. No other changes to files. WMK 11/25/03
sudsputaway:
Added functions to properly initialize SUDS structures to
Banfill's default values. CJB 10/3/2003
sudshead.h:
Updated to make more compatible with suds.h in current
release of PCSUDS. CJB 10/3/2003
file2ew:
Fixed bug in raw2ew.c which caused file2ew to exit when the
SuffixType command contained only 2 arguments. The third
argument (installation ID) was supposed to be optional, and
now is! LDD 10/3/2003
ew2file:
Fixed bug in timing of heartbeat file generation.
Had been using HeartBeatInt instead of HeartFileInt to control
the production loop. Caused continuous heartbeat file generation
when HeartFileInt==0. LDD 11/24/2003
sniffwave:
Changed to sniff headers of both TYPE_TRACEBUF and TYPE_TRACE_COMP_UA msgs.
Will only print data values for TYPE_TRACEBUF messages. Changed to accept
the any of these strings as wildcards in SCN: "wild", "WILD", "*".
LDD 10/31/2003
vX.X/src/data_exchange/Makefile:
Added directives for STANDALONE_MODULES (right now getfileII and sendfileII)
so that their executables, which are created in their source directories,
are also copied to the vX.X/bin directory. LDD 11/27/2003
libsrc/util/rw_strongmotionII.c:
Changed rd_strongmotionII() to interpret both "-" and "?" as NULL strings
when reading the QID: line qid and qauthor fields. Previously only
accepted "?" as a NULL string. Change was required by CISN data which
uses "-" as a NULL string in those fields. LDD 2/13/2004
sendfileII:
Ported to BSD Unix. Added a new makefile, named makefile.bsd.
To compile, type "make -f makefile.bsd". Otherwise, everything is the
same as on Solaris. WMK 2/17/04.
EWDB - Station Information:
Modified the two stored procedures that handle the processing of SCNL
from station list tools (stalist_XXX2ora) and automatic data (trace feed,
automatic pick feed, strong motion, etc.).
Modified the procedures to properly process the Location Code of the
SCNL. Prior to this, Location codes weren't properly being handled.
Because this change is to SQL procedures and not C code, you must
refresh the DB stored procedures by updating with the SQL scripts from
"working". (run ewdb_load_external_utils.sql from schema/sql_scripts).
DK 02/17/04
CHANGES TO CONFIGURATION FILES and DESCRIPTOR FILES:
****************************************************
shakemapfeed.d: Added new optional command, SMQueryMethod, to control how SM
data is requested from the DBMS. Default behavior (SMQueryMethod 0) is original
2-stage query (get all SM data associated with this eventid, then get all
UNassociated SM data in a space/time box). If SMQueryMethod is non-zero,
shakemapfeed will request all SM data in a space/time box. This will return
SM data associated with this event, SM data associated with other events,
and UNassociated SM data.
Regardless of query method, post-query filtering will be performed on eventid,
external eventid, and external author. SM data associated with a different
eventid but with the same author will be rejected. Data with a different
author will be allowed - this is most likely data imported from another
network and its eventids are meaningless in our DBMS. LDD 02/13/2004
shakemapfeed.d: New command "ScheduleFeed "
sets the shakemap feeding schedule. Up to 20 ScheduleFeed commands are
allowed, setting one time per command. Auto-feeds will continue at the
last ScheduleFeed interval until UpdateDuration minutes after the event
origin time. ScheduleFeed replaces the original DelayFirstFeed and UpdateInterval
commands. These original commands are still accepted and will set up
the auto-feed schedule in the original manner in the absence of ScheduleFeed
commands. LDD 3/4/2004
KNOWN BUGS:
***********
liss2ew:
liss2ew has been observed intermittantly producing malformed
TRACE_BUF messages. Currently the conditions for causing this problem
are unknown. Due to this, liss2ew should be treated as suspect. Use at
your own risk. JMP 6-18-2003
k2ew:
k2ew uses the k2hdr.rwParms.misc.channel_bitmap parameter to decide
which channels it will see as streams from the K2. This is actually the
parameter which the K2 uses to decide which channels to record in an event
file (see K2's command). The K2's is the parameter
that shows which channels the K2 is streaming out. This is the parameter
that k2ew should be using to decide what data it will see. The big problem
is that none of the header files from Kinemetrics (nkwhdrs.h) seem to
show this parameter anywhere. This issue is only a problem for those folks
who want to record more channels on the K2 than they want to stream continuously.
Terry Dye (Univ of Utah) discovered this problem when trying to stream only
3 channels of a 12 channel K2. LDD 4/11/2002 FIXED PNL, 6/9/03
wave_serverV:
it occasionally sends the following error to statmgr:
UTC_Thu Sep 06 03:30:14 2001 WSV1/wave_serverV_nano ReadBlockData
failed for tank [z:\nano57.tnk], with offset[999908] and record
size[64]! errno[0] Mail sent.
The nominal tank size is 1 megabyte, and the actual tank size is 999908.
It looks like waveserver is trying to read off the end of the tank.
WMK 9/6/01
wave_serverV:
appears not to reply to requests for a single sample of data. I noticed
when testing wave_viewer, that if the start time and end time of a request were
equal (in which case there should be one sample of data) then wave_serverV did
not reply to the request (ASCII request) at all. No Data, No Flags, No Reply,
No Nothing. It needs to issue a reply to every request.
DavidK 09/25/01
Automatic restarts of adsend (using the "restartMe" line in the descriptor
file) can cause an NT system to hang. Therefore, you should never
use the autorestart feature with adsend, but you should bring down
the entire Earthworm system if adsend needs to be restarted.
LDD 5/31/2000 Comments added to adsend.desc, but leave this warning here!
libsrc/utils/site.c: The strings used for station, channel and network are
required to be fixed length with trailing spaces added to short names. If
the strings given to site_index do not have these trailing blanks, SCN names
will not match. This is not documented anywhere. PNL 10/15/00
socket_ew: (libsrc/util/socket_ew_common.c libsrc/solaris/socket_ew.c
libsrc/winnt/socket_ew.c include/socket_ew.h)
Fixed a bug in the connect_ew function().
When run in non-blocking mode (clients connecting to
a server - using a timeout value), there was a bug in
the connection code(discovered on Solaris) that caused
the function to return a timeout-error when there was
any kind of error during connection.
The bug was discovered when connecting to a non-existent socket.
When trying to connect to a non-existent wave_server on a machine,
the underlying socket library was returning a Connection-Refused error,
but the socket_ew library was passing back a timeout error.
-
The change only applies when a socket-error occurs while the function is
waiting for the connect to happen. The instance where you will most-likely
see a difference, is when you try connecting to a non-existent socket.
Previously the function would return TIMEOUT, now it will return
connection REFUSED.
-
Added a new function: socketSetError_ew(),
and defined a new socket_ew return code: CONNREFUSED_EW.
DK 2003/02/04
sm_ew2ora:
There is a bug in sm_ew2ora that involves having multipe time intervals for
components and channels. If a strong motion message containing information
for a channel that the DB has never seen before is loaded into the DB, and
then later another message for the same channel with an earlier timestamp
is loaded, the load of the second message will fail, due to problems with
overlapping time intervals, and the call that sm_ew2ora uses to create
those time intervals. This problem only affects stations that were
not previously loaded via one of the station loader programs stalist*2ora,
and only when receiving SM data that is timestamped with a time that is
prior to the original time for that channel. The bug lies in the logic
of ewdb_api_PutSMMessage(), and not in the underlying code.
Davidk 05/25/01
A change was made to ewdb_api_PutSMMessage() that dramatically affects
sm_ew2ora. Please see the note about that function. Davidk 2001/07/26
ewdb_api_CreateWaveform() (ewdb_internal_CreateSnippet.c
ewdb_internal_CreateWaveformDesc.c)
Added a call to release the cursors used by the internal functions, when they
fail. Fixes a bug which resulted in a DB cursor leak when a call to stuff
a snippet into the DB failed.
DK 2003/02/04
logit.c:
1) Added a new function logit_core()
int logit_core( char *flag, char *format, va_list ap);
This function is the same as logit(), except that it takes a va_list argument
for the variable length parameter instead of '...', and it has an int
return value.
This function can be called by other functions that receive a '...' variable
argument list, where as logit() cannot. logit_core() is to logit()
what vsprintf() is to sprintf().
-
2) Moved all of the functionality in logit() to logit_core(), and modify
logit to call logit_core().
-
3) Added a new function get_prog_name2(), as an intended replacement
of get_prog_name(). get_prog_name2() includes an additional
parameter (the buffer length of the output buffer).
-
4) Modified logit_init() to use get_prog_name2() instead of get_prog_name()
-
All four(4) of these changes should be backwards compatible with all existing
earthworm code, and the modified logit.c has already been tested with several
earthworm modules.
-
ewdb_api:
Added new function ewdb_api_GetEventSummaryInfo() to retrieve summary
information for an event(including the preferred szSource and szSourceEventID
of the event). 011904 DK
KNOWS DEFICIENCIES:
*******************
ringtocoax:
In Windows NT, the time resolution of sleep_ew() is about 16 msec (one clock
tick). On Solaris, the resolution is about 10 msec. This is a problem for
ringtocoax, since packet delays need to be set to a few milliseconds.
statmgr: A space is needed between "tsec:" and the value.
If it isn't there, things fail. Artifcat of the kom routines. (Alex)
threads functions: The KillThread function on WindowsNT and Solaris
terminate the thread without ensuring that no mutexes are held. If a thread
holds a mutex when it dies, no other thread can get that mutex. PNL 1/12/2000
carlsubtrig:
The system time must be set to GMT and ew_nt.cmd must have
TZ=GMT for carlsubtrig to work. Comments in ew_nt.cmd done 5/25/00. Barbara
localmag:
needs GMT set on the system
ew2seisvole:
on NT, exits with horrible crash when system is stopped.
NUMBER OF RINGS LIMITED ON SOLARIS:
Under Solaris 2.6 (and probably other versions as well), the maximum number
of shared memory segments is six. This means that on an out-of-the-box machine
you can only configure six rings. If you try to configure more than that, you
will see a cryptic message from tport_create about too many open files. The
fix to this problem is to add the following lines to the /etc/system
file, and then reboot the system.
set shmsys:shminfo_shmmax = 4294967295
set shmsys:shminfo_shmmin = 1
set shmsys:shminfo_shmmni = 100
set shmsys:shminfo_shmseg = 20
set semsys:seminfo_semmns = 200
set semsys:seminfo_semmni = 70
This allows for 20 rings.
Lucky Vidmar (7/6/2000)
startstop_solaris:
Fixed bug in call to logit_init().
The program was printing error messages:
Invalid arguments passed in.
Call to get_prog_name failed.
WARNING: Call logit_init before logit.
startstop_solaris:
There MAY be a problem with the signal that
startstop sends to modules during the shutdown sequence. The shutdown
sequence is started (after typing "quit" to startstop or running "pau")
by startstop placing a terminate message on all transport rings. Modules
should see this message and start their own shutdown. After a configurable
delay, startstop checks to see that all modules have exitted. Any that are
still running are sent a signal to terminate them. Currently that signal
is SIG_TERM. But since wave_serverV has a handler for SIG_TERM, wave_serverV
sees that as essentially the same as a terminate message. So if wave_serverV
is having problems completing its shutdown, SIG_TERM won't do anything. The
result is that startstop may give up and exit, leaving wave_serverV running.
If that happens, the operator will have to terminate wave_serverV by doing
"kill -9 ". That may leave shared memory and semaphores
stranded in the kernel: run the command "ipcs -a" to see. If necessary,
the stranded shared memory and semaphores may be cleaned up with the
ipcrc command; must be run as root; see the man page.
This problem only exists on Solaris/Unix, not on WindowsNT.
PNL, 10/4/2000
libsrc/utils/kom.c: fix comment in k_open()
The comment above k_open() says that only one file can
be open at a time. Yet the Kbuf array has slots for MAXBUF (currently 4) open
files. Does this work, or is the comment to be taken at it's word?
PNL 10/15/00
libsrc/utils/logit.c: logit_init() requires a module_id number, which it uses
to construct the log file name. This is not helpful, since the module_id
number is not meaningful to people. Worse, it requires that the config file be
read and earthworm.d lookups be completed before logit calls can be made. Thus
errors in the config file can only be reported to stderr or stdout instead of
being saved in a file. PNL 11/29/00
libsrc/util/k2evt2ew.c: This library supports a maximum EVT data size
of 800000 samples per channel per EVT file. This value is hardcoded
as MAXTRACELTH in include/k2evt2ew.h. k2evt2ew() will return a warning
if the EVT file exceeds this size, and process as much as the EVT file
as the hardcoded limit allows.
DK 2003/01/18
TRACEBUF messages.
The definition of `endtime' of the TRACEBUF message is not documented.
Some programmers are taking it as the "expected start time of the next
TRACEBUF packet (if the sample interval is uniform.)" The more accepted
practice is that `endtime' is the time of the last sample of the current
TRACEBUF packet; that is, one sample interval less than the expected
start time of the next TRACEBUF messsage. Using this last definition, if a
TRACEBUF packet has exactly one sample, then its starttime and endtime are
the identical. Clearly this distinction needs to be documented. The file
waveform_format (in the /home/earthworm/DOC directory) gives no specifics
about start or end times. PNL 1/24/01