Back to the Table of Contents

6.7 Quality Assurance, Correlation, and Map Reliability

Quality and reliability of the ecosystem maps and associated databases are extremely important in TEM. Maps are costly and time-consuming to produce, and users expect them to be well produced and reliable. Time taken during mapping, sampling, data entry, and compilation will result in a map product and database that will benefit the client and many users in future years. Principles and procedures to ensure high quality mapping are presented in this section.

6.7.1 Quality assurance and correlation

This report provides standards to ensure that all ecosystem mapping in the province follows the same methods and is compatible from project to project, using the same ecosystem unit characterization and symbology. Similarly, the Field Manual for Describing Terrestrial Ecosystems (BC Ministry of Forests and BC Ministry of Environment, 1998a) provides data collection standards for sampling site, soils, vegetation, trees and wildlife habitat. For assistance in applying mapping and sampling standards, the BC Ministry of Forests' Regional and Research Branch Ecologists and BC Environment's Provincial Correlators with the Resources Inventory Branch should also be consulted.

The goals of the TEM correlation process are to: 1) improve TEM mapping by giving the mappers technical support before, during, and after the field season, providing them with comments on interim products, and offering them constructive criticism; 2) furnish a critical evaluation and quality assurance function to ensure that the RIC standards for TEM are promoted and enforced; and 3) provide a feedback mechanism for the continued evaluation, improvement, and evolution of TEM.

For each project, a project ecologist should be identified to ensure that project members are mapping consistently and meeting quality standards. Periodic reviews are required to evaluate the sampling plan, pre-typing, field data collection, final mapping process, map legends, and reports describing the ecosystem units that are mapped. The project ecologist should use the expertise of Regional and Provincial Ecologists, terrain specialists of the BC Ministry of Forests and BC Environment, and project clients, to ensure timely and on-going review. Reviews conducted throughout the mapping process are more likely to result in quality products for the least cost. Where interpretive maps and databases are being produced, they should also be reviewed by appropriate specialists.

If mapping is done under contract, the agreement should outline the review requirements for each deliverable of TEM. Those requirements can be summarized as follows:
  1. Sampling plan. The sampling plan, including working legend (based on existing information), should be reviewed for the timing of field sampling; consideration of access (hike-in/road/helicopter) and relative costs; strategy for sampling ecosystems, structural stages, parent materials; etc.
  2. Preliminary mapping (pre-typing). A preliminary set of photos should be reviewed before fieldwork, to assess ecosystem unit and/or bioterrain delineations and provide feedback to the mappers. This step includes an assessment of alpine and parkland biogeoclimatic boundaries (if these units occur in the area) as they are important to subsequent delineations.
  3. Field sampling. Correlation during field sampling allows for assessment of field sampling procedures (e.g., plots in appropriate locations, forms completed correctly and accurately); and determination of whether site series are identified and mapped and whether biogeoclimatic units are mapped reasonably at the larger scale.
  4. New site series. If new site series are recognized, they will require approval by the Regional Ecologist. In order to review this stage, the ecologist should be supplied with all appropriate field data, vegetation and environment tables by sites series, and brief notes justifying any proposed new site series.
  5. Biogeoclimatic boundaries. The linework should be reviewed before final mapping, as these lines are critical to subsequent ecosystem mapping and sampling.
  6. Draft ecosystem map. A review of an early draft of the ecosystem map and databases may identify errors or inconsistencies that are more easily corrected than at the end of the project. At this time, all aspects of the mapping should be reviewed. This means determining that ecosystem unit symbology, designations, proportions, and total of proportions are correct; use of "non-vegetated" or "anthropogenic" types is appropriate; each polygon is closed and properly matched to photo borders; site modifiers are used where appropriate; and linework, map legend, GIS spatial data, databases, and attributes follow standards. Final terrain attributes will also be reviewed, and when bioterrain is the initial base mapping for ecosystem, it should be evaluated prior to completion of ecosystem mapping.
  7. Final mapping. This is the last opportunity to review the TEM project. A final review of photo interpretation, mapping, and databases, (as in item 6), should be conducted before acceptance. This should be a less rigorous review than that of the draft map, to confirm that recommendations have been incorporated.
  8. Final report. If produced, the report should be reviewed to ensure it is accurate and consistent with the map, standards for labelling, etc. All projects with a wildlife component require an expanded legend.
To ensure a quick and efficient review, all necessary project materials should be provided to the reviewer.

6.7.2 Map reliability

An assessment of map reliability is an evaluation of how accurately the map and legend represent the landscape. Both qualitative and quantitative assessments of map reliability are useful.

Qualitative assessment

The map and the process of preparing the map are qualitatively assessed during the correlation procedure outlined in Section 6.7.1. In addition, there may be specific field-based qualitative reliability assessments after mapping. These trips would entail comparing map delineations and labels with what is observed on the ground. The differences between a qualitative assessment and the quantitative assessment outlined below are in aspects such as selection of sample points and compilation of results. The qualitative approach may still be systematic and focused, but does not rely, for example, on selecting random polygons or plotting transects within polygons. Rather, its emphasis is on assessing as many polygons as possible within the time frame available, using whatever access is possible (e.g., road transects, visual observations, helicopter hovers). The compiled results are used to assess whether the map is reliable for the intended interpretations and can be used to focus a subsequent quantitative assessment.

Quantitative assessment

A sample of map polygons should be assessed in the field by the mapping project supervisor or correlator. The objective of this assessment is to provide some statistics on the accuracy of the polygon designations and to "audit" the mapping project. The process of assessing map reliability quantitatively is discussed in Steers and Hajak (1979), Wang (1982), Forbes et al. (1982), Valentine (1986), and Gopal and Woodcock (1994). Although a standard procedure has not yet been developed, the following principles should be followed in any assessment:
  1. Identify aspects of the project that require a quantitative assessment. It may be a general assessment of all ecosystem unit components, or a focus on specific problems identified from the qualitative assessment. For example, the focus may be site modifier designations, the assessment of ecosystem unit proportions, the placement of polygon boundaries, or the mapping of one or two site series.
  2. Identify specific attributes to be assessed at each ground sample to be audited (e.g., site series, site modifier, structural stage, or ecosystem unit proportion) and determine the allowable variation from the measured value. This will usually involve setting an assessment scale where "almost" right answers get intermediate values. For example: if absolutely wrong, a value of 1 might be applied; if understandable, but wrong, a value of 2; if a reasonable result, a value of 3; if a good answer, a value of 4; and if absolutely right, a value of 5.
  3. Randomly select a sample of polygons. This can be done for the total map or for a stratified sample of polygons addressing the "problem" being assessed. The maximum size of the subset of polygons depends on the size of the population, the issues being addressed, and the resources available. However, it should not be less than 30 polygons. Depending on the focus of the assessment, an efficient field procedure might involve randomly selecting a set of polygons and deciding a priori to sample the surrounding two to three polygons at the same time.
  4. Plot line transects. Within each polygon, a transect that will cross all significant landscape features should be plotted. Mark 10-20 sample sites at a fixed interval along the transect. Again, an effective field procedure would be to plot the transect through the cluster of polygons to allow for efficient travel.
  5. Sample along each transect. Each sample site should be assessed using the ground-truthing attributes selected above.
  6. Analyze the data and generate statistics. The data for each polygon should be assessed and the observed values compiled into a value that can be compared to the mapped value. The polygon can then be evaluated and scored, using the chosen assessment scale.
Other useful procedures for assessment are to assess other polygons (of the same population) by helicopter hover or photo interpretation. The results of these assessments can also be used in the analysis of the problem or project.

As mapping is somewhat of an art, it is often difficult to determine whether a mapped attribute is clearly right or wrong. Therefore, a "fuzzy analysis" of the data is recommended. Gopal and Woodcock (1994) outline several options. For example, the so-called "RIGHT" function can be used if the objective is to determine the nature and distribution of errors. Other functions can determine magnitude or significance, source, or the nature of errors.

Determining a pass/fail grade requires setting certain minimum values. For example, using the RIGHT function, a minimum passing value might be set at 65% for each map category assessed and 75% overall. The proportion of polygons meeting the quantitative criteria will vary according to survey intensity level. At the lowest intensity level, perhaps only 60% of the polygons will pass, whereas at the highest level, most polygons should.

[Back to TOC] [Previous] [Next]