Duration: 40 minutes
Validation is an essential processing task, with two primary objectives: · To eliminate incorrect values from a file;
· To enhance the consistency of the data.
Many companies use a centralised corporate database to store their corporate and scientific data. If your company uses such a system, it places increased pressure on you to ensure that your data are valid and consistent before uploading into your Database Management System (DBMS). Companies are also becoming increasingly aware of the value of their data, which is encouraging good validation processes. At a simpler level, some Micromine functions may produce undefined results if they encounter invalid data.
After this lesson you’ll be able to: · Validate fields in a data file;
· Validate drillhole data comprising collar, downhole survey, and interval files.
Before you continue...
Make sure you’ve completed Lesson 2 – Working with data files.
Validating Data in a Project
Validation
Micromine supports the validation of nearly all types of codified data (with the obvious exception of fields containing random comments) and can also perform rigorous validation of drillhole data.
There are five validation functions in Micromine: · Validate in the Tools menu of the File Editor
(File Editor: Edit | Tools | Validate)
· Validate under the Fields item in the File menu
(Main menu: File | Fields | Validate) · Validate in the Drillhole menu
(Main menu: Drillhole | Validate | Drillhole Database) · Validate in the Drillhole menu
(Main menu: Drillhole | Validate | Drillhole) · Validate in the Drillhole menu
Notes: The first two validation options are designed to validate the contents of fields
in a file, whereas the last three are designed to validate drillhole data.
Drillhole database validation is covered in Part 2 – Displaying and Manipulating Data.
Validating a Data File
You have two choices for validating fields in a file: Either Edit | Tools |
Validate or File | Fields | Validate. With Edit | Tools | Validate, you
can only validate one field at a time, but it’s easy to use because you can validate at the same time as you’re editing the file.
File | Fields | Validate is a more powerful function because it can
simultaneously validate up to ten fields and you can set validation ranges on numeric fields. You’ll use this function in this lesson.
The Validate function checks the specified fields in the target file for correct values. If it finds an incorrect value it will write an error message to a report file, along with the name of the field in which the error was found and the number of the record.
A check file is used to validate the contents of fields where there’s a fixed set of possible values, which most commonly applies to character fields. The Validate function determines whether the values are correct by comparing them against the check file. If a value exists in the target file but doesn’t exist in the check file, it’s considered an invalid entry. You must have previously created the check file.
Although it’s possible to use a check file to validate numeric fields (e.g. 1 = MINED, 0 = UNMINED), most numeric validation is done by range checking. The Validate function determines which values are valid by testing if they fall within the ranges that you specify. Range checking helps to eliminate typographical errors and other mistakes that often occur in data entry.
Creating validation check files
There are many ways to create a check file. If your company uses a corporate database you should be able to create one by importing the relevant lookup table from the database. Otherwise you can use File |
Fields | Extract Unique to list all codes – including errors – that exist
in the main data table. Clearly you should correct the errors in the extracted data before using it as a check file.
Exercise 5.1: Validate fields in a file
Validation is often used to check the consistency of lithology codes used to describe sample data. In this exercise, you’ll simultaneously perform character validation using a check file and numeric validation using range checking, to validate records in the NVG_LITH_WITH_ERR file. (This file
Notes: contains deliberate errors.) The numeric range checking will identify values that fall outside the range of 0 (zero) to 75.
To validate the file:
1. From the main menu, select File | Fields | Validate.
The Fields Validate dialog is split into two halves: The upper half allows you to set up the input, check, and report files, whereas you define the various validation rules in the lower half.
2. Fill out the upper half of the Fields Validate dialog as listed:
Prompt Setting
Input File NVG_LITH_WITH_ERR
Type DATA
Data type DRILL HOLE
Check file NVG_VALID_LITH
Type DATA
Report file INVALID_LITH
3. Right-click the Check file NVG_VALID_LITH and inspect its contents. The listed codes are all confirmed as being valid; therefore any codes that occur in NVG_LITH_WITH_ERR but are absent from the check file will be recorded as invalid.
Data type: GENERAL vs. DRILL HOLE
Both data types validate in exactly the same way; only the way in which errors are reported changes. If you use the GENERAL data type, errors are reported as being on LINE
x
of the input file. If you use DRILL HOLE, errors are reported as being on LINEx
, HOLEy
, FROMa
TOb
.Notes:
When you select Case Sensitive, values in the file you’re validating must match the case of your check file or they’ll be recorded as invalid. For example, ANDS would be seen as being different to ands or Ands.
5. Click the Run button on the dialog to run the process, and dismiss the Errors were detected message box.
6. The Fields Validate window will disappear, leaving the Report Viewer docked at the bottom of the window.
7. Double-click a record in the Report File. This will automatically take you to the matching record in the Input File, which you can correct if necessary.
Notes:
Validating Drillhole Data
The drillhole validation function (Drillhole | Validate | Drillhole) detects a large number of errors and inconsistencies in collar, downhole survey, interval, and event files.
It detects:
FROM < previous TO Duplicate collar entry
FROM >= TO Dips or Azimuths change by more than x FROM or TO missing Surveys beyond total depth
Collar missing or incorrect Missing hole in interval file Record beyond total depth Compulsory field blank Hole excluded by collar filter Total depth missing Duplicate hole Rate of Deviation Non-consecutive surveys Sample Interval Lengths
Micromine performs this validation by checking the relationship between the files and the relevant fields, as illustrated below. For example, it checks for missing holes by listing the hole ID’s it finds in each file and then checking to see if any are missing from any of the other files.
As you can see, drillhole data undergoes comprehensive testing. You should always validate drillhole data to ensure trouble free operation in later processes.
You must perform a separate validation run for each interval or event file that you’re checking, using the same collar and downhole survey file in each case. For example, if you have interval files for assays, lithologies, and oxidation, you must perform three separate runs of Drillhole | Validate | Drillhole, using a different file each time.
Hole ID From/To or Depth Hole ID Survey Depth Azimuth/Inclination Hole ID Coordinates Total Depth
Collar
File
Survey
File
Interval
Notes: Alternatively, you can validate all files in one pass using Drillhole |
Validate | Drillhole Database. Creating and validating a drillhole
database is explained in Part 2 – Displaying and Manipulating Data.
Exercise 5.2: Validate drillhole data To validate the drillhole data, do the following:
1. Select Drillhole | Validate | Drillhole from the menu.
2. Ensure the Collar File tab is active and make the following entries:
Prompt Setting
File NVG_COLLAR
Type DATA
Hole field [Automatic]
Easting, Northing, Z field [Automatic] Total depth field [Automatic]
3. Switch to the Survey File tab and make the following entries:
Prompt Setting
File NVG_SURVEY
Type DATA
All other fields... [Automatic]
4. Enter the following on the Interval File tab:
Prompt Setting
File NVG_ASSAY_WITH_ERR
Type DATA
All other fields... [Automatic]
5. There are no event files to validate, so skip the Event File tab. 6. Finally, set the following options on the Report tab:
Prompt Setting
File DH_VALID
Notes: Check for missing holes Set Check inclinations Cleared
Check azimuths Cleared
Check maximum deviation Set Check sample interval lengths Cleared Valid inclination change Disabled Valid azimuth change Disabled
Max deviation 0.05
Max interval Length Disabled
Grade Field Disabled
7. Click OK. You’ll receive a message saying, “There were X errors found”. These errors are written to the Report File; when you click OK to dismiss the message box the Report Viewer will appear at the bottom of the window.
What’s max deviation?
Max deviation is a smart validation tool that takes into account the distance between successive downhole surveys as well as their varying orientations. It will also handle sub-vertical holes with azimuths that vary over the full 0 – 360° range, but which in reality may only deviate by one or two degrees. The traditional dip/azimuth options poorly handle both of these situations.
Max deviation is expressed in terms of degrees per linear unit, and a good way to set an initial value is to consider the total amount of deviation you would allow over a 100 m drillhole. If that amount is, say, 5 degrees, then the max deviation would be 5 / 100 or 0.05.
8. Inspect the report file and note the errors. The file begins with just over 30 downhole survey errors caused by consecutive surveys deviating by more than 0.05° per metre. (These errors could in fact consist of valid data, so it’s entirely up to you to determine whether or not corrections are required. We’ll assume they’re valid for this exercise.)
9. Scroll to the bottom of the file and double-click the last record. 10. You’ll be taken to the matching record in the NVG_ASSAY_WITH_ERR
file, where you can see the offending interval. If this were a real project you’d insert the missing data, but for the remainder of the training we’ll use an alternative file, NVG_ASSAY, that contains no errors.
11. Close the Report Viewer and NVG_ASSAY_WITH_ERR file in preparation for the next exercise.
Notes:
How did Micromine know the right field names?
You’ve probably noticed that Micromine knew the names of the fields in the drillhole files. You can set the field names that Micromine allocates to dialog prompts by selecting Tools | Options | Forms from the main menu. Here you can enter the names you use for the fields at your site. However, it’s still good practice to click each of the Fields buttons to check that the fields have been correctly allocated!
Notes: Lesson 5 Summary
These are the key points of this lesson:
· To ensure consistent results you must validate data before using it. · Validation is a prerequisite for data that will be stored in a central
repository.
· Validation processes can be automated on sites where code usage is consistent.
· Drillhole data should be validated before using it in other drillhole processes.
To validate the contents of a file:
Select File | Fields | Validate, and
Select the File to be validated, and the Check file, then Define the validation rules.
To validate drillhole data:
Select Drillhole | Validate | Drillhole, and
Select the appropriate Collar, downhole Survey, Interval, and Event files, then
Select the desired reporting options. Good Practice
· If you’re processing many files containing similar codes, you can create validation files and re-use them.
· You should always run drillhole validation after adding any new drilling data or after importing it or reading it from other systems. Most drillhole display problems are caused by data errors that Validate will easily identify.
· If you use multiple interval files (such as separate assay and lithology files), or multiple event files, you must do a separate validation run for each interval file using the same collar and downhole survey file for each run.
· Use Drillhole | Database | Validate to simultaneously validate all files in a drillhole database.
Help Topics
For information on: See:
Field validation Files > Fields > Validate Drillhole validation Drillholes > Validate
Notes: