5.0 Problems with Archaeological Predictive Model Applications
Should managers select a correlative (inductive) model, which is easier to design, takes less time to develop, and is initially more accurate, or should they plan to use an explanatory (deductive) model, which is more complex and difficult to develop and may not be as accurate a predictor? (Judge/Martin 1988:577)
Cultural resource managers and archaeologists are less concerned with the overall predictive success rate of a model than with the likelihood of an inaccurate prediction. There are two types of predictive errors: a prediction can be made that a location (or area) contains a site when in fact it does not, and a prediction can be made that a location does not contain a site when in fact it does. The first type of error may lead to increased costs or to inefficient use of resources and is a wasteful error. Errors of the second type lead to the destruction of cultural resources and are gross errors.
In a hypothesis-testing framework there are always two potential errors: rejection of the null hypothesis when it is in fact true (wasteful error) or acceptance the null hypothesis when it is false (gross error). An ideal predictive model minimizes both types of errors: it makes accurate predictions. In practice, however, models do make inaccurate predictions. Generally, it is much more costly in cultural resource management to make a gross error than a wasteful one, and the likelihood of making a gross error is inversely related to the likelihood of making a wasteful error (Altschul 1988:62).
The choice between two models, then, has less to do with overall success than with minimizing errors, especially gross errors. "In general, a more powerful predictive model is one that for a specific proportion of gross errors to total predictions also minimized the area predicted to contain cultural resources" (Altschul 1988:63).
Predictive models are probability statements; they are not "facts," and cannot substitute for facts in any application requiring the use of hard data about specific areas containing archaeological resources.
The problem is that some archaeologists have told some planners that our predictive models can be used as hard data, when in fact, it is our hard data on site location and significance that must be figured into the planner's cost-effective ratio. To substitute a scientific hypothesis (our predictive model) for scientific fact (actual site location) as a criterion for a planning decision is to court disaster. There is only one way for us to get the hard data for use in such decisions: by an intensive ground-reconnaissance of the entire area to be affected by a proposed project (Kohler 1988:24).
The specific failings of past modelling efforts have included:
failure to address management needs, lack of specificity, poor use of existing data, ineffective or biased sampling designs, inappropriate statistical analysis techniques, failure to collect inventory data suitable for the development of a predictive model, development of models using non-replicable techniques, lack of comparability of and inappropriate use of environmental variables, lack of phasing to allow for model testing and refinement, and failure to use such technical aids as remote sensing and geographic information systems (GIS) to streamline model development (Sebastian/Judge 1988:10).
Resource managers have found that predictive modelling is being employed in a wide variety of ways and that there is little . mutually agreed-upon theory, method, or policy to guide the use of this technique (Judge/Martin 1988:571). Issues of primary concern are:
In order for a cultural resource manager to use information derived from models, even for the most general planning purposes, he or she must know that the model works within specified levels of confidence and precision (Sebastian/Judge 1988:6).
Too frequently, a good fit between model predictions and observations is taken as a confirmation of the model. It appears that the dominant goal of location models in archaeology is to get the facts and the model to fit. However, no model can account for all the facts, and several contradictory models can predict the same set of facts. Too often neither the model nor the facts are properly theorized, and therefore alternative explanations are not rigorously considered (Keene 1988:242).
Land managers should take care against the improper use of intuitively derived models in influencing inventory efforts.
Archaeologists who work frequently in an area often develop a "feel" for where sites should be found. occasionally, these intuitions have been used as a basis for limiting inventory to certain areas without testing others (Kincaid 1988: 561). A danger in this approach is that if sites are sought only where they are thought to exist, the prediction becomes a self-fulfilling prophecy. Potential results can include destruction of significant resources or introduction of a strong bias into the data base.
Intuitions should not be dismissed, but they should not be equated with scientifically verified information. Intuitions must be formalized, expressed in terms that can be measured and applied in the inventory process, and subjected to a rigorous testing program. In this way they can be of vital importance in effective model development (Kincaid 1988:562).
Models of site location based on existing data can lead to predictions with very high accuracy rates. After all, if people have only looked for sites in certain types of places, then it is inevitable that site locations will be highly correlated with specific environmental attributes.
The problem is that many archaeologists stop there and never formalize their answer. Thus, no matter how brilliant their insight or how many sites they find, no one can objectively evaluate how well their model works (Altschul 1988:65).
The development of model components and the definition of their interrelationships should be the areas in which archaeologists make their greatest contribution to the predictive modelling process. This, however, has not been the case. Instead, there has been a tendency among archaeologists producing predictive models to concentrate on the sophisticated multivariate mathematical techniques and to give only casual attention to the predictive variables. In most cases, methodological discussions focus on the inner workings of the statistical procedures with only passing references to the reasons why specific variables are theoretically related to site location. Indeed it appears that investigators are assuming that the relationship(s) between the environment and site location cannot be specified, other than that there is one, and that if only enough environmental variables are put to the equations something useful will come out (Altschul 1988:84).
The task of location modelling is to isolate those aspects of the environment that do influence settlement behaviour and place them into perspective with non-environmental factors that also influence settlement behaviour (Kohler 1988:25).
This is a crucial observation for the task of location modelling. Many valid criticisms can be made of naive environmental determinism for its suggestions of large-scale, simplistic correlations between environmental and cultural features. Correlation is not explanation and does not tell us anything about causality.
Because correlative models are designed to tell us where sites are located (relative to various environmental variables) and not why they are located as they are with respect to those variables, even when they work exceedingly well, it is not known why they work. To the manager who only needs to know where archaeological sites are this may not immediately appear to be a major limitation. But if it is not known why a model works in one particular study area, it cannot be explained whether it is expected to work in the next valley or watershed or in a similar but distant environment. Thus correlative models are not truly predictive, but consist of projections of an observed pattern from a sample to the whole universe.
Another limitation arises because correlative models require measurable, mappable data. For this reason, they depend heavily on environmental factors to provide their independent variables, and because of this they are most successful when applied to societies whose movements, group size, and activities are highly regulated by aspects of their environment - generally hunters and gatherers (Sebastian/Judge 1988:5).
Because correlative models are relatively straightforward to develop and because simple environmental variables are relatively easy to measure, these models are viewed as cost-effective and objective. And in the short run they often provide the kinds of information needed (Sebastian/judge 1988:8).
Although there may be arguments about how to test for correlation or how to measure the strength of a correlation or assign confidence limits to it, once those are resolved the only question that remains is whether a correlation exists or not (Sebastian/Judge 1988:10).
A major problem with associational models is generalization. They are usually not derived from probabilistic sample surveys and thus may contain biases that will be magnified if the model ,is generalized (ie. extended to areas that have not been surveyed) (Altschul 1988:66).
There is no question that controlled probabilistic sampling is an efficient way to learn various things about a population without having to examine every member of that population. "Except in the few cases where there is essentially no environmental variation within the area to be sampled, one of the things we obviously want to find out is what kinds and numbers of sites might be located in which types of environments" (Ambler 1984:140).
In principle, a perspective is gained concerning what kinds and densities of sites are found in what kinds of locations, and from this sample projected to the subject area as a whole. However, "no matter how rigorous our sampling design, how thorough our sample, or how complex our statistics, we have not learned anything about the area as a whole, we have only learned about our sample; all else is projection, within certain limits of probability. We can predict likely locations for sites, but cannot predict locations where sites are absent" (Ambler 1984:141).
If one cannot rely on the results of probability sampling due to gross errors in application, then predictive models built into these data are equally unreliable. Here we have the focus of the current controversy over predictive modelling. It is not the modelling and it is not the sampling that makes archaeologists uncomfortable, it is the substitution for verification. Previous work has demonstrated that prehistoric sites are distributed differently across the landscape from one area to another and their relationship to existing environmental variables is neither obvious nor simple. Data available for analysis is often unreliable due to uncontrolled biases through probabilistic samples and incompatible methods of observation.
One reason is that most models are constructed inferentially, starting from a sample of archaeological sites in a region and generalizing to an unknown population of sites in that same region (Kohler 1988:19). Environmental variables are often not used in most predictive location models because archaeologists simply do not know how to use them (Kohler 1988:20).
The model must be subjected to rigorous testing to confirm its accuracy. This demonstrates a major limitation on any predictive model: predictions are based on known data and are incapable of dealing with unique or extraordinary occurrences. By basing the model on sample data we only intensify the problem (Klesert 1987:230).
Two methodological issues arise from the foregoing: how to characterize the nature of the background environment on variables under examination, and how to compare archaeological samples against the background environment.
Aside from relevant statistical literature regarding sample size and the power of tests, there has been little study or guidance in the archaeological literature regarding the nature or size of samples needed to adequately characterize the often considerable variation present in the background of regions, particularly when they are large.
One-sample testing strategy for regional archaeological analysis avoid many difficulties by treating the background environment (on any variable) as a constant. Archaeological samples then may be compared against this background referent to determine whether they are unusual or deviate in some way from this norm. Stated differently, the background environment of a defined region of study is the same population of interest. One-sample tests examine whether characteristics of an archaeological sample depart significantly from the population.
Two-sample tests can only represent the background environment imperfectly through random sampling. This introduces additional variance to an analysis because sampling variation in both the archaeological class as well as the background class must be dealt with by the statistical test. Less powerful conclusions must result when compared with one-sample-testing approaches, to the same problems, that focus only on variation in a single archaeological sample (Kvamme 1990a:368).
For continuous environmental variables such as elevation, slope, aspect, distance to water, and local relief, a one-sample chi-square test is possible, but it is undesirable because the continuous variables must be categorized into relatively few classes whose areas in the region of study can be determined on maps (ie., area of level ground vs. area of steep ground; area within one kilometre of water vs. area within two kilometres of water vs. area beyond two kilometres of water). "This amounts to the re-scaling of interval-level concepts to a lower level resulting in the throwing away of information that can lead to less-powerful inferences" (Kvamme 1990a:369).
Predictive modelling as employed by most archaeologists, until recently, looked for sites in likely locations and ignored places not considered to be likely for sites. Archaeologists are now beginning to realize that the "modelling" which has been advocated in the past has a surprisingly, and perhaps dangerously, simplistic foundation beneath all of the mathematical discussions.
How are models evaluated? The answer often depends on the questions asked. For example, "successful modelling" for the archaeologist who wishes to predict location of sites to minimize construction impact may appear as a "failure" to the archaeologist attempting to interpret the processes that drive humans to select past settlement locations.
An important consideration for evaluating models is their ability to take into account rare sites. These sites constitute a very small portion of the site population either by virtue of their own characteristics or their location in relation to the environment. A site type can be rare without being impossible to model, but most models do not address these sites, because their low numbers make most statistical techniques unusable. The rare site problem increases when sample inventories at low sampling rates are used to generate the data base for model development.
Models should be evaluated for their completeness. Did they address changes in the environment through time? Are there biases in the' sample design that might affect the reliability of the data? Also, the resolution of the model is important. If the management need is for statements specific to linear corridors for example, broad zonal models may not be useful (Kincaid 1988:566).
On-site characteristics (such as slope) and resources, distances of sites to environmental features or resources, and characteristics of catchments surrounding sites, all share the feature that they are site focused. "To be measured, such variables demand identification of a specific location. By contrast, most predictive models focus on characteristics of the quadrats used for survey or for prediction" (Kohler/Parker 1986:408).
Few predictive models use immediate location characteristics of sites (such as aspect, view, and on-site vegetation), because of the high spatial resolution for prediction these require. Nonetheless, settlement-pattern analyses have repeatedly demonstrated that sites may be non-randomly located in relation to very local features. More intuitively, locations for rock-shelters, quarries, petroglyphs, weirs, and other specialized site types readily demonstrate the potential importance of very local environmental features to site location (Kohler/Parker 1986:409).
Research questions are often changed or refined as knowledge increases. "We need a lot more basic data before attempting to satisfactorily answer many of the research questions posed today, and we cannot hope to anticipate all the types of questions that might and should be asked once an area has been surveyed" (Ambler 1984:142).
If theory establishes the rules for the relationships among variables, models enable us to view the implications of changing the rules, to view the outcome of assigning specific values or conditions to those variables. Thus, models are not only manifestations of theory; they actually allow us to probe the limitations and sensitivities of theories.
Models are frequently criticized for ignoring variables, or for oversimplifications. Such criticism overlooks the paradox of models: their greatest strength, the ability to simplify reality and make problems tractable, is also a weakness. Such simplification sacrifices some aspects of reality. Any individual modelling effort, therefore, is incomplete.
Successful model applications generally result from approaching a problem with several different models or with several variations of a single model. Models can be viewed as heuristic devices, as a means for examining the implications of theory (Keene 1985:241).
With the ability to create potentially powerful archaeological resource models through a GIS, discipline wide attention must be given to how these models are employed by officials in government agencies responsible for the protection of irreplaceable resources. There is natural concern that agencies will use such models to form decisions resulting in a loss of resources in predicted low site probability regions. This perception tends to misplace blame on models, or model makers, rather than on decision makers.
Land managers may make unreliable decisions with regard to the resource base with or without the use of location models of any kind. It is hoped that managers will choose from among the responsible decision alternatives. If they have not always done so, this should not be seen as the fault of the methodology on which they based their information. If managers are making unreliable decisions with regard to the archaeological resource base, then attention should be paid to defining the responsibility. The question becomes whether (GIS based) site location models can be designed to give better site location predictions than would be possible without the use of such models.
Several archaeological GIS studies already have appeared that contain more computer generated "gloss" than substance. The lure of producing beautiful maps and data models is great, but some thought and concern must be given to the quality of the data and nature of the assumptions used to produce them. Without careful consideration these models will contribute to unreliable land use decisions.
This potential problem area will become more severe as sophisticated, non-archaeological computer consultants, who do not understand the nature of the archaeological data, processes, and needs, become increasingly involved in complex database, analysis, and modelling situations on a contractual basis (largely in cultural resource management contexts). In these cases archaeologists must play a large role and provide informed guidance; to be informed, however, requires some knowledge of GIS technology, the computational procedures that go into them, and analysis and modelling procedures. Consultants are usually hired because archaeologists lack the technical knowledge (Kvamme 1989:188-189).
The use of inappropriate sampling techniques, failure to differentiate significant temporal and functional subsets of sites, failure to consider how representative variables really contribute to location decisions, low spatial resolution, inappropriate statistical tests, and little consideration for model validation have often weakened the usefulness of these models for both management and research. All of these weaknesses may not be found in any one model, but at this stage in the development of predictive modelling, it can hardly be expected that any model would be entirely free from these difficulties (Kohler/Parker 1986:440).
Finally, the interpretability of the model is important. Is the model simple enough to be understood and explained in anthropological terms? Does it relate environmental and site variables to the everyday world? If not, it may not be usable by future researchers in a cultural resource management environment (Kincaid 1988:567).