README.changes

Changes to reftek2ew


#########################################
#########################################
#########################################
Converting reftek2ew to prevent bogus tracebufs 
from entering the earthworm datastream
Davidk 06/04/2004, LynnD 6/21/2004

Codework was done primarily to prvent reftek2ew from
emitting tracebuf messages with bad timestamps, 
into the EW data stream.

The code now performs the following check on each
packet before emitting it as a tracebuf:
  a)  starttime < endtime  (no backwards time in the packet)
  b)  starttime > endtime_of_last_packet (no 
        backwards time between packets)
  c)  Number of sample in each packet > 0  (no empty packets)
  d)  starttime < system_clock + time_slop 
        (ensure that the timestamp is reasonably valid
         and is not 20 years in the future).

The changes also included the following:

  a) Added support for issuing error messages via
     the earthworm error system (statmgr).

  b) Added global access (within the program) 
     to config file params

  c) Improved performance of SCNL lookups for each
     packet, by sorting the channel list and using
     bsearch(), instead of a full list traverse.

  d) Converted send.c and scn.c to use the EW logit()
     mechanism in lieu of fprintf(stderr)


The following three files were minimally modified
to handle globalized params and new params:
init.c main.c params.c

The following files were heavily modified to accomodate
the changes listed above:
import_rtp.h
scn.c
send.c



Added earthworm status codes for use in sending error
messages to EW statmgr.

Added program name (prog) to parameters struct.
Added TimeJumpTolerance parameter to parameters struct.
TimeJumpTolerance defaults to 24 hours.

Made the parameters struct globally available, so that
any function can access the params.

Added get_dt_index() function to improve performance
on SCNL lookup for each packet.

Added reftek2ew_send_error() function to issue errors
to statmgr via EW error system.

Added GetLastSampleTime() / SetLastSampleTime() functions
to get and set the last sample time for each channel.
Used to ensure that each channels data only flows forward
in time, and does not make errant clock jumps.

scn.c
added functions SetLastSampleTime() GetLastSampleTime()
    get_dt_index(), add_dt_entry_to_list(), compare_lookup_table_entries()
Modified function load_scnlp().  Broke it down into smaller parts.
The entry array, is now in sorted order, and is accessed via bsearch().
The matching entry record for each dt packet is retrieved via
add_dt_entry_to_list.
Then the scnlp header is retrieved via the index.  
Changed fprintf(stderr) calls to logit("et")

send.c
Added reftek2ew_send_error()
Added code to  send_wav() to validate tracebuf timestamps
before sending them into the EW data stream.
#########################################
#########################################
#########################################

#########################################
#########################################
#########################################
Additional packet filtering based on samplerate and
compression errors.
The goal was to fix the screwed up logic that calculated the
sample rate and endtime for each packet.  The previous logic
did this based upon the contents of the current packet and the 
previous packet, and tended to blow up when the datalogger clock jumped, 
which in turn tended to cause destructive packets with incorrect data to 
be emitted from reftek2ew.
A constraint was to not take the time and effort neccessary to wade 
into the reftek data protocol, and figure out how to extract the official
sample rate.
The result is a hacked up method for calculating the
nominal sample rate of a channel, based upon the sample rate
of the first few stable packets received per channel, and a
list of acceptable sample rates (from the config file).
It is envisioned that this fix, should keep reftek2ew running
in a non-offensive manner, until RefTek (or someone else) modifies
reftek2ew so that it obtains the official sample rate from the 
datalogger, and properly handles errors and time tears.
Davidk 06/08/2004 (with lots of design assistance from LDD).

Codework was done primarily to prvent reftek2ew from
emitting tracebuf messages with decompression errors,
and bad sample rates.

reftek.scnl
Added additional column SampRate, that is the nominal sample rate
for the channel.  This parameter is optional.

import_rtp.h reftek2ew.d params.c
Added config file params:
DropPacketsWithDecompressionErrors
                       - filter out packets with decompression errors
SendTimeTearPackets    - Send packets that appear immediately after a time tear.
AcceptableSampleRates  - List of acceptable samplerates for channels emitted by reftek source.
                         iNumAcceptableSampleRates - number of sample rates listed.


import_rtp.h
Added functions:
EWIsValidPacket        - function that performs validation on tracebufs before
                         they are sent out into EW land.
SampleRateIsValid      - function that validates the sample rate vs. the nominal for the channel.
SetNomSampleRate       - function that stores the nominal sample rate for the channel,
                         given that it stored already and that the calculated rate is 
                         acceptable.           
GetNomSampleRate       - function that returns the nominal sample rate for the channel.
SampleRateIsAcceptable - function that verifies that the sample rate is in the acceptable list.

import_rtp.h params.c
Added hardcoded parameters:
FilterOnSampleRate     - filter out packets with questionable sample rates.
GuessNominalSampleRate - program should try to guess the sample rates for
                         channels not defined in the config file, by looking
                         at the sample rate of incoming packets.

scn.c
Added functions:
SampleRateIsValid      - function that validates the sample rate vs. the nominal 
                         for the channel. Allows for a small amount of slop defined 
                         by (ALLOWABLE_SAMPLE_RATE_SLOP).
SetNomSampleRate       - function that stores the nominal sample rate for the channel,
                         given that it is not stored already and that the calculated 
                         rate is acceptable.   
GetNomSampleRate       - function that returns the nominal sample rate for the channel.
SampleRateIsAcceptable - function that verifies that the sample rate is in the acceptable
                         list. Allows for a small amount of slop defined by 
                         (ALLOWABLE_SAMPLE_RATE_SLOP).
Modified code to parse SCNL list file, to extract optional "Sample Rate" column.


send.c
Modified send_wav() to optionally drop packets with decompression errors.
Moved all EW "bad tracebuf" filtering out of send_wav() and into a new function:
EWIsValidPacket().
Modified send_wav() to calculate the packet endtime and sampletime from the
nominal sample rate that is stored in scn.c for the channel, instead of calculating
the params from the reftek get_samprate() function.

Added function: 
EWIsValidPacket        - function that performs validation on tracebufs before
                         they are sent out into EW land.  Added support for sending
                         of packets immediately after a time discontinuity, if 
                         SendTimeTearPackets is set to 1 in config file.

Added filtering based on samplerate functionality.

#########################################
#########################################
#########################################


