What is the BGS method for creating Quasi-Definitive Data ?
Note: We can use this method because we have great faith in the stability of the variometer instruments. It would be much more difficult if the baselines were trending more rapidly. As a rough rule of thumb use the following procedure on a year’s worth of historic data to check whether this method is suitable:
- Calculate an extrapolated baseline for a period of time where you already have a definitive baseline, starting the ‘extrapolation’ at the time of the first absolute observation. Record the difference between the extrapolated baseline and the true (definitive) baseline at the time of the next absolute observation.
- Repeat this calculation for a number of absolute observations throughout the year
If these differences are less than 2.5nT, then your baselines should be stable enough to use this method.
The BGS system produces QD data on a next day basis. The variometer data (both one-minute and one-second) and the daily baseline values are stored in separate files. Data from these files are combined to produce baseline corrected data. This allows changes to the baseline to be easily updated.
There are two parts to the production of quasi-definitive data, which incorporates both manual and automated operations.
Daily data processing and quality control procedures for variometer data
- During the day and on a next day basis plots are viewed by the duty processor. These are magnetograms, F difference (or closing error) plots, individual component comparison plots (where an observatory has more than one system installed, such as in the UK). These plots are automatically produced or can be regenerated manually.
- Any errors in the variometer data are either removed, or, in the case of the UK where backup systems are running, replaced with an appropriate set of data. For real time products, the data processing software will carry out the latter automatically following changes to configuration files.
- Every working day morning any required adjustments to the variometer data for the previous day (or three days following the weekend) is completed.
Processing of absolute observation measurements and production of baseline values
- Absolute observations are recorded and processed using a Java program, which will also read in the variometer data and output the spot baseline values (that is the absolute measurement minus the variometer reading at the time of the measurement). This is carried out as soon as possible after absolute measurements are made.
- A separate FORTRAN plotting program is used to view the spot baselines against the current continuous daily baseline values. The plot includes the daily mean F difference or closing error as well as the daily mean temperature in the variometer chamber. When new observations are available this plot is assessed to make sure the current baseline is still well within the QD definition.
- Once a month, or sooner if required (see above), the recent spot baseline values and other absolute observation data are analysed using an Excel spreadsheet. The continuous baselines are fitted using a series of piecewise polynomial fits, which are subjectively chosen following the removal of outliers. Outliers are removed based on a set of rules, which takes into account the collimation errors as well as any other obvious factors.
- Adjustments are made to the piecewise polynomials as required and the baseline updated. All previous versions of the baseline file are saved using a simple version control system.
- Daily baseline values are created for the complete year. i.e. including a projection into the future based on the baselines on the day of computation. Account can be taken of any current trend (if any) however where this is the case it is even more important to repeat the process with new observations as soon as they are available, thus reducing the number of days of predicted baseline values.
Combining the two parts enables:
- Real time and next day processing to be carried out automatically. The processes read in the most up-to-date baselines (part 2) and the corrected one-minute (or one-second) variometer data (part 1) to produce real time and next day data that should meet the QD definition that has been established by INTERMAGNET.
- The QD data to be prepared and delivered to INTERMAGNET manually on a next (working) day basis following the completion of all necessary checks as described.
Back to FAQ Index