CONCIERGE

CONTENTS

PURPOSE

OUTPUTS

1. ora_trace_req - a program that turns EW trigger messages into snippet requests that it records in the database.
   (see ora_trace_req documentation for more info)
   
   
2. ora_trace_fetch - a program that reads snippet requests from the database and attempts to retrieve waveform data from wave_servers and archive the retrieved data into the database.
   (see ora_trace_fetch documentation for more info)
   
   
3. EWDB_WaveformRequest API - This is a collection of EWDB_API library code used by ora_trace_req, ora_trace_fetch, and potentialy other programs to read/write/update snippet requests in the database. Application developers wanting to request snippets via the Concierge system can utilize this API.
   (see EWDB Waveform Request API documentation for more info)
   
   

SUB-TOPICS

1. How you specify snippets to request (ora_trace_req)
   Literally, snippet requests are specified by calling an EWDB API function that records a request in the database. For most folks that aren't looking to write their own specialty programs every time they want to do something, the default solution is to use ora_trace_req. ora_trace_req is an Earthworm module that converts Earthworm trigger messages(from a ring) to snippet requests, and stores those snippet requests in the database. This leaves you with requirement of converting your "desired snippets" list into one or more Earthworm trigger messages. This can be done with an editor, or you can teach automated programs to write trigger messages. (See Earthworm Trigger Message documentation for more info.) If you generate a trigger message by hand, then you can insert it into an Earthworm Ring using the "file2ring" utility. Carltrig and HypoInverse/arc2trigII already generate trigger messages.



2. How the snippet requests get fulfilled (ora_trace_fetch)
   Snippet requests are fulfilled by an Earthworm module (ora_trace_fetch) that periodically retrieves the requests from an EW Database. ora_trace_fetch periodically queries the given database for a list of requests. For each request, it queries wave servers for data that matches the request, and then stores the retrieved data in the database. Lastly it updates/deletes the request based upon the results of it's data retrieval.



3. When the snippet requests get fulfilled(Scheduling/Retrieval)
   The fulfillment of the snippet requests is a 3-layer ordeal:
   • When the snippet requests are scheduled to be processed
     The snippet requests are scheduled to be processed based upon a scheduling algorithm that resides within the snippet-retrieval program(ora_trace_fetch).
     ora_trace_fetch contains a file schedule.c, that controls when snippets are scheduled. From a user point of view, the scheduling algorithm used should be specifiable in the ora_trace_fetch config file.
   
   
   
   • When the snippet requests are actually processed
     The retrieval program may not actually process the requests at the time they are scheduled. This may be because of a backlog in requests, where ora_trace_fetch cannot process the requests fast enough to stay on schedule, or this may be because ora_trace_fetch is running in a special mode where it is processing all requests regardless of when they may be scheduled.
   
   
   
   • When the requests are successfully processed
     When the requests are actually processed, is described above. However, there is no guarantee that they will be processed successfully. The requests may ask for data that a wave_server does not yet have, or data that a wave_server no longer has. There may not be a wave_server that has data for the requested channel, or there may be a network problem that causes the data to temporarily be unavailable. Maybe this is way too obvious, but in order for the request to be processed successfully, the data has to be avaiable from a wave_server that ora_trace_fetch can reach at the time ora_trace_fetch processes the request.
   
   See the detailed section on Scheduling/Retrieval for more information on scheduling and retrieval times.
   

ORA_TRACE_FETCH

DETAILS
ora_trace_fetch is an Earthworm module, that attempts to fulfill snippet requests. It reads snippet requests from the database, attempts to retrieve waveforms from wave servers, and then archives the waveforms into the database.
ora_trace_fetch utilizes Concierge (EWDB Waveform Request) API code to read snippet requests from an EW Database. It then utilizes Earthworm wave_server client (ws_clientIII) routines to attempt to retrieve data from one or more wave servers(wave_serverV). It then records the results of its attempt via EWDB Concierge API code and EWDB Waveform API code.

• If ora_trace_fetch is able to obtain all requested data from a wave_server, then it writes the retrieved data to the database via the EWDB Waveform API library code, and deletes the Snippet Request via the Concierge API library code.
• If ora_trace_fetch is unable to obtain any of the requested data from a wave_server, then it reschedules the request for a later attempt, and updates the Snippet Request via the Concierge API library code, indicating that it failed.
• If ora_trace_fetch is able to obtain part, but not all of the request, then it writes the retrieved data to the database via the EWDB Waveform API library code, reschedules the remaining portion of the request for a later attempt, and updates the Snippet Request via the Concierge API library code, indicating that it was not completely successful.

There is some trickery about how often ora_trace_fetch reads the table of requests, and when/how often it attempts to fulfill each request.

PARTIAL SUCCESS
The beauty of ora_trace_fetch and Concierge when compared to the previous XXX_trace_saves for the DBMS, is that the requested snippets can be processed multiple times, and snippets can be built incrementally. When ora_trace_fetch experiences a partial success in the fulfillment of a request, it writes the waveform data that it WAS able to obtain to the database as a Snippet, and then sets a link from the request to that Snippet. It then adjusts the request, so that the request now contains only that portion of the original request that was not obtained previously.

Example ora_trace_fetch retrieves a snippet request from the database, asking for data for AAA,EHZ,XX for the time range (0 - 100).
ora_trace_fetch attempts to retrieve the requested data. It obtains data from 0 - 20. It creates a snippet(123) in the EW DB for AAA,EHZ,XX (0 - 20). It then updates the request so that it is: (AAA,EHZ,XX : 20 - 100, w/Snippet 123). ora_trace_fetch reschedules the request for processing at a later time.

At a later time, ora_trace_fetch retrieves the snippet request from the database again. Now the request asks for data for AAA,EHZ,XX for the time range (20 - 100), and indicates that there is an existing snippet(123).
ora_trace_fetch obtains data from 20 - 45 for the channel from a wave_server. ora_trace_fetch retrieves the existing snippet(123) from the database, and updates it so that it contains dat for (AAA,EHZ,XX (0 - 45)). ora_trace_fetch also updates the request so that it is: (AAA,EHZ,XX : 45 - 100, w/Snippet 123). ora_trace_fetch reschedules the request for processing at a later time.

At a later time, ora_trace_fetch retrieves the snippet request from the database again. Now the request asks for data for AAA,EHZ,XX for the time range (45 - 100), and indicates that there is an existing snippet(123).
ora_trace_fetch is unable to obtain new data for this request(maybe the network was down).
ora_trace_fetch reschedules the request for processing at a later time.

At a later time, ora_trace_fetch retrieves the snippet request from the database again. Now the request asks for data for AAA,EHZ,XX for the time range (45 - 100), and indicates that there is an existing snippet(123).
ora_trace_fetch obtains data from 45 - 100 for the channel from a wave_server. ora_trace_fetch retrieves the existing snippet(123) from the database, and updates it so that it contains data for (AAA,EHZ,XX (0 - 100)). ora_trace_fetch deletes the request because it has been completely fulfilled.
(END OF STORY)

Overriding the Scheduling Algorithm
This is no longer applicable as all scheduling is done with ora_trace_fetch. Check the config file and find the algorithm that best suits your needs. If none work for you, write your own!

RETRIEVAL MODES

For more detailed information on Modes, see the Program Modes section in the Programmer Notes.

In addition to the various modes of operation, the behavior of ora_trace_fetch is altered by the reception of a trigger message. Whenever ora_trace_fetch receives a TYPE_TRIGLIST2K message, it will immediately check the request list again to see if there are any requests to be processed. This behavior modification only has an affect when the program is running in STANDARD_DATA_MODE, because the other two modes already do this during each cycle of the main loop.

DETERMINING RETRIEVAL TIMES

There is currently (1) scheduling algorithm provided by ora_trace_fetch. It is described below.

NOTE:Despite the impressions given in the following examples, all scheduling algorithms, do not schedule tNextAttempt. The schedule tDeltaNextAttempt, that is they calculate the time between now and when the next attempt should take place, and they pass that value to the EWDB Waveform Request API, which calculates the current time and adds the tDeltaNextAttempt to the current time to produce the actual tNextAttempt. This prevents algorithms from scheduling processing in the pass.

SCHEDULE EXPONENTIAL

SCHEDULE EXPONENTIAL is the default scheduling mechanism, and as of this writing, the only one that exists. SCHEDULE EXPONENTIAL utilizes an exponentially increasing time scale for attempting processing of snippet requests.

By default, SCHEDULE EXPONENTIAL is setup to attempt retrieval 11 times over a 7 day period. The timing of the retrieval attempts is approximately as follows:
Approximate time of attempts (after generation of request):
0(m)inutes, 10m, 30m, 1(h)our, 2.5h, 5h, 10h, 21h, 2(d)ays, 3.5d, 7d

There is a caveat(atleast one). If the request results in a partial success, SCHEDULE EXPONENTIAL will record the partial success and then furlough the request attempt for a few minutes(10), and then try again, without having recorded that the attempt was made.

The Details
When a snippet request is first handled by SCHEDULE EXPONENTIAL, it is assigned a set of scheduling parameters that are used by it's scheduling algorithm to calculate an exponentially increasing time series.
The current algorith has 5 variables.

• iNumAttempts: The total number of times that data retrieval will be attempted for this request.
• tAttemptInterval: The base time-attempt interval. This value is multiplied by an exponentially increasing value, in order to calculate the next attempt time.
• dAttemptMultiple: This is the base value that is exponentialized, in order to create an exponentialy increasing attempt time.
• tCurrentTime: This is the current time (used when updating the SnippetRequest and calculating the next attempt time.
• iNumRemainingAttempts: This is the number of attempts (out of iNumAttempts) that are remaining in the request's life.

  tNextAttempt = tCurrentTime + tAttemptInterval * 

                               (dAttemptMultiple ** 

                                (iNumAttempts - iNumRemainingAttempts - 1)

                               )

• 1) The snippetrequest is first handled by ora_trace_fetch and SCHEDULE EXPONENTIAL, and the iNumAttempts is set to the value configured for the algorithm within the ora_trace_fetch config file. The iNumAttempts and iNumRemainingAttempts are thereafter stored with the request in the database. tCurrentTime is always the current time, derived from the system clock. The remaining parameters, tAttemptInterval and dAttemptMultiple are provided by the ora_trace_fetch config file or are defaults.



• 2) ora_trace_fetch notices the request, and attempts to fulfill it. Assuming it is not completely successful, iNumAttempts and and iNumRemaining attemps are set in the database and the request is updated. ora_trace_fetch supplies the current time (tCurrentTime). iNumRemainingAttempts is decremented since an attempt has been performed. SCHEDULE EXPONENTIAL calculates the next attempt time as:
  

    tNextAttempt = tCurrentTime + tAttemptInterval * 
  
                                 (dAttemptMultiple ** 
  
                                  (iNumAttempts - iNumRemainingAttempts - 1)
  
                                 )



• 3-[X-1]) Except for the final attempt, every attempt after the second one, is just a repeat of the 2nd. With the exception that the tCurrentTime and iNumRemaining attempts will change for each successive attempt.



• X) After the final scheduled attempt, when the snippetrequest is updated, SCHEDULE EXPONENTIAL deletes the request from the DB because it has reached its number of allowed attempts.

• 1) ora_trace_fetch notices the request within 5 seconds. It attempts to retrieve the data from one or more wave_servers(this takes 10 seconds). At time 1000000015, it completes the attemtpt. Because it was not completely successful, SCHEDULE EXPONENTIAL sets the iNumAttempts and iNumRemainingAttempts for the request to 11. SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 11 to 10. SCHEDULE EXPONENTIAL computes the next time as:
  

  tNextAttempt = 1000000015 + 600 * 
  
                               (2 ** 
  
                                (11 - 10 - 1)
  
                               ) =  1000000615
  

  tNextAttempt is updated to 1000000615. Delta T from the original request is 615. (10 minutes)



• 2) at 1000000615, it completes the attempt. Because it was not completely successful, SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 10 to 9. SCHEDULE EXPONENTIAL computes the next time as:
  tNextAttempt = 1000000625 + 600 * (2 ** (11 - 9 - 1)) = 1000001825
  tNextAttempt is updated to 1000001825. Delta T from the original request is 1825. (30 minutes)



• 3) at 1000001825, it completes the attempt. Because it was not completely successful, SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 9 to 8. SCHEDULE EXPONENTIAL computes the next time as:
  tNextAttempt = 1000001835 + 600 * (2 ** (10 - 8 - 1)) = 1000004235
  tNextAttempt is updated to 1000004235.Delta T from the original request is 4235. (1 hour)



• 4) at 1000004235, it completes the attempt. Because it was not completely successful, SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 8 to 7. SCHEDULE EXPONENTIAL computes the next time as:
  tNextAttempt = 1000004245 + 600 * (2 ** (10 - 7 - 1)) = 1000009045
  tNextAttempt is updated to 1000009045. Delta T from the original request is 9045. (2.5 hour)



• 5) at 1000009045, it completes the attempt. Because it was not completely successful, SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 7 to 6. SCHEDULE EXPONENTIAL computes the next time as:
  tNextAttempt = 1000009055 + 600 * (2 ** (10 - 6 - 1)) = 1000018655
  tNextAttempt is updated to 1000018655. Delta T from the original request is 18655. (5 hours)



• 6) at 1000018655, it completes the attempt. Because it was not completely successful, SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 6 to 5. SCHEDULE EXPONENTIAL computes the next time as:
  tNextAttempt = 1000018665 + 600 * (2 ** (10 - 5 - 1)) = 1000037865
  tNextAttempt is updated to 1000037865. Delta T from the original request is 37865. (10 hours)



• 7) at 1000037865, it completes the attempt. Because it was not completely successful, SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 5 to 4. SCHEDULE EXPONENTIAL computes the next time as:
  tNextAttempt = 1000037875 + 600 * (2 ** (10 - 4 - 1)) = 1000076275
  tNextAttempt is updated to 1000076275. Delta T from the original request is 76275. (21 hours)



• 8) at 1000076275, it completes the attempt. Because it was not completely successful, SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 4 to 3. SCHEDULE EXPONENTIAL computes the next time as:
  tNextAttempt = 1000076285 + 600 * (2 ** (10 - 3 - 1)) = 1000153085
  tNextAttempt is updated to 1000153085. Delta T from the original request is 153085. (2 days)



• 9) at 1000153085, it completes the attempt. Because it was not completely successful, SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 3 to 2. SCHEDULE EXPONENTIAL computes the next time as:
  tNextAttempt = 1000153095 + 600 * (2 ** (10 - 2 - 1)) = 1000306695
  tNextAttempt is updated to 1000306695. Delta T from the original request is 306695. (3.5 days)



• 10) at 1000306695, it completes the attempt. Because it was not completely successful, SCHEDULE EXPONENTIAL updates the request reflecting that it attempted and was not completely successful, decrementing iNumRemainingAttempts from 2 to 1. SCHEDULE EXPONENTIAL computes the next time as:
  tNextAttempt = 1000306705 + 600 * (2 ** (10 - 2 - 1)) = 1000613905
  tNextAttempt is updated to 1000613905. Delta T from the original request is 613905. (7 days)



• 11) at 1000613905, ora_trace_fetch attempts to process the request. It attempts to retrieve the data from 1 or more wave_servers(this takes 10 seconds). At time 1000613915, SCHEDULE EXPONENTIAL, prepared to update the request notices that the request has been attempted its maximum number of times(11 of 11). Instead of updating the request decrementing iNumRemainingAttempts from 1 to 0, SCHEDULE EXPONENTIAL deletes the request. Note: The request was deleted at time 1000613915, approx 7 days after its creation.

INSTALLING CONCIERGE

CONFIGURING CONCIERGE

KNOWN ISSUES

BLAME