Tutorial 1a – Introduction to ASP Data and ArcGIS
Andrew Bevan and James Conolly (15.12.07, updated 15.01.08)
with additional thanks to Varina Delrieu.
I. Introduction
This tutorial is an introduction to ASP datasets using ESRI’s ArcGIS 9.2 software. It and the datasets it uses can be found on the ‘Downloads’ page of the ASP website (http://www.ucl.ac.uk/asp/ and/or http://naxos.tuarc.trentu.ca/~aspweb/) and will also be archived with the UK Archaeology Data Service (http://ads.ahds.ac.uk/) from 2009. This particular tutorial seeks to convey the key features of ASP’s primary datasets, how they are integrated with one another and how they might be most easily manipulated. ASP methods sought to strike a balance between maintaining some direct comparability with a range of existing survey datasets in the Aegean area, developing practices that streamline field recovery methods, facilitating meaningful spatial statistical analysis and making our results relatively easily understood and used by others. As a result of the compromises that such a balancing act requires, the recovery and recording methods are certainly not claimed as a single form of best practice for such research, though we are happy that they nonetheless offer a reliable basis for more substantive archaeological interpretation.
This tutorial is also certainly not meant to provide a full overview of ArcGIS functionality, but nonetheless assumes little or no previous knowledge of this package. ArcGIS is the overall name for a suite of proprietary, commercial software components made by Environmental Research Systems Inc. (ESRI, http://www.esri.com). This suite has three main levels that offer ever more complicated levels of functionality: in increasing order, ArcView, ArcEditor and ArcInfo. These levels are confusing for those familiar with older ESRI products, because the first and last names used to be standalone applications. Now however, Arcview refers to the simplest level of the ArcGIS hierarchy, providing basic GIS functions such as mapmaking, querying and simple editing of spatial data. ArcEditor offers great capabilities for editing and creating spatial data, and ArcInfo offers additional spatial analysis tools (e.g. statistical tools). In turn, each of the above three hierarchical levels of ArcGIS Desktop consists of several mini-applications: the two most common are ArcMap and ArcCatalog, with many additional ones besides.
processed, as well as with what level of accuracy).
II. ASP Data Setup
Please download the following datasets (about 18.5 Mb in total) from the ‘Downloads’ page of the ASP website (http://www.ucl.ac.uk/asp/ and/or http://naxos.tuarc.trentu/~asp/). If, for whatever reason, you are not able to get them from this source, they will have been bundled with this tutorial or, from 2009 onwards, will also be archived with the UK Archaeology Data Service (http://ads.ahds.ac.uk/).
Survey Units – this will be a file called survey.zip Database - this will be a file called db.zip
Quickbird Satellite Imagery - this will be a file called qb.zip
Full Resolution Quickbird Sample - this will be a file called qb10.zip
When integrating raster and vector in an ArcGIS environment, it is sensible to adopt filenaming conventions that avoid Windows long-name aliases; more precisely, it is good to keep file and folder names to seven or fewer characters in length with no capitals and no spaces. In most cases, more complex names will not cause problems, but being very conservative with these filenames nevertheless short-circuits one common source of trouble.
1. So please extract ASP data to a folder path on your computer that conforms to these requirements (e.g. C:\asp\ would be good). Within this folder, you should end up with extracted datasets that have the separate sub-folder names \survey, \db, \qb10 and \qb (you could then delete the original zip files to save space).
2. Now Launch ArcCatalog from the ArcGIS section of the Start Menu Programs. This file manager is organizing into left-hand and right-hand windows. In the left-hand window, browse through the directory tree to /survey/tracts.shp and left click once on this file. On the right hand side, there are three tabs: ‘Contents’ shows you some basic information about the file in question, ‘Preview’ shows you a snapshot of the data or table, and ‘Metadata’ provides a much fuller description of the known background to the spatial dataset. For example, a closer look at this metadata (after having left-clicked on the metadata tab in the right hand window) reveals that the dataset is a set of polygons which represent ASP’s first stage of surface collection. Further detailed information is also available: for example: left-clicking on the ‘Geographical’ tab (or in ArcGIS v.9.1, the ‘Spatial’ tab) shows that, in common with all ASP data, this dataset is stored in the projected coordinate system known as Universal Transverse Mercator (UTM; WGS84 Zone 34N) primarily because this facilitated easy integration of Quickbird imagery and handheld GPS readings in the field. If you follow the links under ‘FGDC lineage’ you will also see information on the processing history of this dataset which includes digitisation onto a Standard Tasked Quickbird image and then semi-automated cleaning to remove polygon slivers and overlaps. All other downloaded datasets have also been given metadata of this kind, so please take a moment to explore the other datasets that you have downloaded.
III. ArcMap: the basics
There is a typical Windows menu at the top, and then a series of button-based toolbars below this. Sometimes one or more of the toolbars is ‘floating’, as is the ‘Tools’ menu above. If you cannot see the toolbar, you can re-display it by going to ViewToolbars. The Tools menu gives you basic ways to zoom in and out, select features and extract attribute information about datasets in the map window. The best way to explore these features is to experiment. A handy button to use if you get lost while zooming in and out is the one in the shape of a globe which will zoom you back to the full extent of the data shown in your map window.
ESRI products work with several native spatial formats, but we will work primarily with the long-established, traditional ESRI vector datasets called shapefiles, and with the ESRI raster datasets known as Grids. Shapefiles are usually referred to by their .shp suffix but both these and Grid datasets actually comprise a number of related files that work together (or, alas, not at all if they have become separated through bad file management).
1. In ArcMap, please click on the Add Data button in the main button bar.
scale in the white box and your view will change accordingly. Likewise, the Measure tool can be used to measure distances across the map (which are reported in the bottom lefthand corner of the ArcMap document).
2. Now click on on the Add Data button again and add \survey\tracts.shp. Right click on the name of the tracts shapefile in the Table of Contents and then left-click on the Properties option. This provides you with a wide variety of information pertaining to this individual layer in your ArcMap document. Now choose the Symbology tab and you will be able to see that the default display mode is to show this dataset as a single symbol (though the default colour is chosen randomly and thus varies). The shape, size and colour of vector and raster datasets can be modified in a wide range of ways using these options. Left-click on the current symbol in the middle of the dialog and then on the right-hand side of the resulting dialog, set the Fill Color to ‘No Color’, the Outline Color to ‘Seville Orange’ (mouse over the colour to get its name) and the Outline Width to 0.40. Then click OK.
The set of polygons that you now see outlined in orange maps out ASP’s stage-one survey method which is a slight variation on a well-established Mediterranean survey technique which divides the landscape into sub-hectare collection units (that are sometimes arbitrary in shape and sometimes follow real-world zones such as patches of similar vegetation or walled agricultural fields). ASP and several other Aegean surveys refer to these polygonal units as ‘tracts’ and each tract delimits a zone that was walked across in straight lines by surveyors spaced a certain distance apart (in our case every 15m). Overall, we have walked over 95% of the island in this manner, with the only unwalked areas being some exceptionally steep scarps and coastal cliffs. Each surveyor recorded separate counts per tract of the total pottery and lithics they observed and also made a permanent collection of any pottery sherds that were bases, rims, handles or decorated pieces (what ASP for convenience has termed ‘feature sherds’) as well as any worked lithics that they encountered.
3. Now click on the Add Data button again and add \survey\grids.shp. This is the set of 10x10m grid squares mapping out ASP’s stage-two survey method. Right click on the name of the grids shapefile in the Table of Contents, left-click on the Properties option and then choose Symbology again. Go to the main icon showing the appearance of the grid dataset and, in the resulting dialog, set the Fill Color to ‘No Color’, the Outline Color to ‘Mars Red’ (mouse over the colour to get its name) and the Outline Width to 0.40. Then click OK. The sets of squares now outlined in red reflect the second stage of our intensive surface survey which has involved the collection of a systematic sample of artifacts from 57 different locations across the island. A particular emphasis was placed on exploring prehistoric scatters in an attempt to mitigate the lower diagnostic visibility of these earlier periods in the tractwalking record. Such stage-two collections were organised on a 10x10m grid, with the centre of each grid square defined by a four-digit UTM coordinate pair (in a multiple of five, e.g. 3456, 0125). Not only did this simplify recording, but because it located artefacts to within ca. 10m of their actual position, it also integrates easily with our stage-one tractwalking collection (as above). Within each section of the grid, a circular area of 5 sq.m was completely vacuumed of cultural material over a timed 5-minute observation period, and then, in the rest of the square, grabbed all worked lithics, metals etc, and any ‘feature sherds’ (see above for this definition). Overall, some 1,700 squares were collected (just less than a 1% sample of the island's entire extent), providing a more detailed impression of the size and function of the numerous prehistoric activity areas observed across the island.
5. Now turn off your coast.shp layer by unticking the box next to it in the Table of Contents. This should leave just the tracts and grids visible.
So far we have been dealing exclusively in vector datasets whose geospatial character is defined using coordinate geometry. In contrast, this is a raster dataset, which is a different type of geospatial data model made of pixels or cells of a certain size arranged in a grid. This is pan-sharpened image from the Quickbird satellite, made available by kind permission of Digital Globe and Eurimage. It has been downsampled (i.e. degraded in quality) to a 10m pixel or cell resolution to conform to their copyright restrictions, but remains useful for general orientation of landscape features on Antikythera. Your ArcMap document should now look something like the one below:
7. To save this overall set of files and the way they are displayed in the map window, go to FileSave and save it in your main asp directory as tut1a.mxd. Note that this is an overall Arcmap Document file and does not save the actual geospatial data with it. If you wishes to pass on this project to another user, you would have to give them the whole asp folder and they would need to reset several directory paths to allow the project to work on their own computer (GIS projects of this kind are therefore not as trivial to share as the ordinary single file formats with which people are very familiar such as MS Word documents etc.).
8. Now use the Add Data button and browse to \asp\qb\qbtc.tif. In the resulting dialog, left-click on the image once and then click Add to bring this into your ArcMap document (if you double-click by mistake you will be prompted to bring in only one of the spectral bands of the image which is not what you want; also, if prompted, decline the opportunity to build pyramids at this stage). This is a sample 2x2km portion of a pan-sharpened Quickbird image provided at full resolution (again, with the kind permission of Digital Globe and Eurimage).
choose the option ‘Zoom to Layer’. You should allow you to see the detail of the image, along with the tracts and grids – untick the box next to qb10tc.tif in the Table of Contents and your map window should now look like the one below:
10. To save what you have done so far, choose FileSave.
We will now have a look at how to combine ASP’s many data records with the spatial units that you have already added to your ArcMap document. In \asp\db\ you will see there are a series of three .csv files containing the key aspatial data associated with ASP field and laboratory work (with individual data values delimited by comma separators). In database terms, they have a series of simple relationships to one another that are described in the accompanying text document (readme.txt).
IV. Databases: Tract-Level Records
To begin with, we will focus on the tract-level records kept as part of our stage-one survey method.
1. Use the Add Data button to add \db\t_tractrecord.csv to your ArcMap document. This will automatically switch the Table of Contents view from ‘Display’ to ‘Source’ (visible through the tabs at the bottom left) that, for our purposes here, merely means that you can see in the ToC any aspatial datasets that you have added.
walked, the team and team leader involved, an assessment of ground surface visibility, a description of local land cover, counts of standing features such as buildings, etc. (for further details see the readme.txt file associated with the \db directory).
3. We will now join this data to the tract polygons so we can plot it spatially. Please close the attribute table of t_tractrecord.csv that you have just opened and then right-click on tracts (the polygon layer not the table) in the Table of Contents and choose Joins and RelatesJoin. This brings up a dialog with a series of questions: please make yours look the same as the one below:
4. Now click Ok. This will have joined the attribute table of the tract polygons to the data table with the tract-level information, via the tract ids that both tables share. Now right-click on the tracts polygon layer and choose ‘Open Attribute Table’ and you should be able to see that the tract id field from the polygon dataset (now labeled ‘tracts.TRACT_ID’ for clarity) and the tract id field from the tract record database (‘tract’) have been joined in a one-to-one relationship which then allows us access to all of the associated tract-level recording. Once you are happy with this, please close the attribute table.
Below, this, make the interval size 10 and then click Ok to the entire dialog. If you have done these steps correctly, your previous dialog should now look like the one below (though the colour ramp may be different):
On this map, areas of good surface visibility (i.e. those not obscured by vegetation cover etc.) are shaded yellow while poor visibility areas are shown in red. More precisely, the method of estimation used here expresses the percentage of ground surface visible to the whole survey team as they walked across the tract (i.e. it is an average estimate usually decided upon by the team leader in consultation with the other surveyors in the team), and is by now a relatively well-established indicator adopted by many surface surveys in the Mediterranean. In fact, ASP is not entirely comfortable with the reliability of this measure, not least because it can be shown to exhibit quite a lot of inter- and intra-observer variation depending on who is recording it and at what time of the day, In fact, for much of our analysis, we prefer measures based on reclassification of high-resolution remote sensing imagery to suggest levels of visibility as affected by ground cover.
Take a minute to use the zoom tools to explore visibility across the whole island (you may also want to turn off some of you other layers to speed up the process). When you are finished, proceed with the next part below.
V. Databases: Walker-Level Records
During stage-one tractwalking, individual walkers also recorded their own smaller forms for each tract, noting an estimate of the distance they covered as part of that tract (a rough conversion to metres based on a count of their paces which was then converted to metres using an averaged estimate of their own individual stride length), a count of the pottery and other finds they observed.
includes more than one record for every tract unit in the survey, given that more than one surveyor was usually involved in each tract. ASP has developed an automated method of plotting individual walker lines within each tract polygon (based on tract shape as well as database information about walking direction and walker order from left to right) which allows us to make use of these walker records and plot artefact densities per walker rather than per tract – an example of this is shown in the figure below. For our purposes here we will aggregate this data by summarizing it per tract.
An example of walker lines plotted within tract polygons. Each line has a unique id and can be further sub-divided by walked 10m segment for the purpose of plotting collected artefacts (see section VI below).
2. Find the field (i.e. the column in the table) called ‘tract’ in this attribute table that you have just opened and right-click on the top of it. Then choose the option ‘Summarize’. This brings up a dialog that asks you about what data you would like to include in your summary table.
3. In the middle portion of the dialog please expand the section called ‘distance’ and tick the box next to ‘Sum’ to include a totaled value of distance walked by all walkers during the tract in the resulting table. Likewise, expand ‘sherds’ and choose to include a ‘Sum’ of pottery counted by all walkers during the tract in the resulting table. Make sure to specifying that the output tabled will be saved in your asp folder (preferably not in a subfolder) with the name sum_pot.dbf. If prompted agree that you want to add the new table to your ArcMap document.
N.B. Make sure that at the bottom of your table it does indeed say “Records (0 out of 3610 Selected)”. If it does not, go back and start section V.1 again.
By default, such a summary table provides you with a row for each unique value in the ‘tract’ field on which you originally chose to summarise (i.e. a row for each tract number) and then a count of the number of walker records for that tract (in other words, the number of walkers who walked it). However, we also chose to include further information which gave us the total distance walked (a field called ‘Sum_distance’) and the total pottery counted (a field called ‘Sum_sherds’) for that tract.
3. We will now join this summarized data to the tract polygons so we can plot it spatially. First, please close the attribute table of sum_pot.dbf that you have just opened and then right-click on tracts (the polygon layer not the table) in the Table of Contents and choose Joins and RelatesRemove Join(s)Remove All Joins. This gets rid of the link you created in section IV.3 above and allows you to stipulate a different join, this time between the tract polygons and your summarized walker-level data.
Click OK and if prompted agree to create an index for the resulting join.
5. Now please right click on qbtc.tif and choose ‘Zoom to Layer’ to make sure that you are zoomed in to a suitable area of the map.
6. Now right-click again on tracts (the polygon layer not the table) in the Table of Contents and choose ‘Properties’. Then click on the Symbology tab at the top of the resulting dialog and choose Quantities on the left-hand side and then the sub-option ‘Dot Density’. In the middle of the dialog under Field Selection choose ‘Sum_sherds’ and move it across to the right-hand window by clicking on the button with a right-pointing arrow.
This has randomly placed a single dot in a given tract for every counted sherd (regardless of the chronological date of the sherd). This is one useful way to display the density of sherds across the landscape, revealing that there are a limited number of high-density areas (you could also calculate a measure of pottery density per tract by dividing Sum_pot by Sum_distance and displaying this as a Graduated Colour scheme, but we will not cover this procedure here).
Take a minute to use the zoom tools to explore ceramic density across the whole island (you may also want to turn off some of you other layers to speed up the process). When you are finished, proceed with the next part below.
VI. Databases: Artefact-Level Records
So far we have explored datasets without regard to individual artifacts or their chronological date. However, ASP places great emphasis on permanent artefact collection that can facilitate laboratory analysis and allows us slowly to improve our chronological understanding over the course of further study. We feel that this is a critical feature of survey analysis that could not by achieved by trying to date these in the field alone (and makes sure that surveyors are free to concentrate on quality of recovery rather than on analysis). Our artefact database is broken down by material category and is still being updated (at the time of writing). As a preliminary example however, we can consider the prehistoric pottery recovered from around two Bronze Age scatters in the western central part of the island.
1. Click on the Add Data button again and choose the file t_pottery.csv from the \db folder. Right-click on this table in the Table of Contents and choose ‘Open’. This is part of a record made in the finds laboratory for each potsherd that ASP collected, but note again that here you have only a tiny preliminary sample of 531 sherds (a full dataset will be available by the end of 2008). Similar tables have also been made for other find categories (e.g. worked stone) but are not yet available online. You can see that each sherd has a rather complicated unique id (‘uid’) which is an amalgam of a range of records relating to the survey unit it comes from. For various reasons (see above), and regardless of whether the finds involved are from tractwalking, grid collection or GPS grabs, we can suggest a UTM location (suggX, suggY) for each find that should have a relative accuracy of ca.±10m. Thereafter, for each artefact we have recorded a range of other information on the recovery type (tract, grid or grab), vessel shape, fabric, dimension etc. (each of the fields is described in the readme.txt accompanying the \db folder). Perhaps the most unusual aspect of this recording however is the way in which finds are dated. A series of fields beginning with fn_eb1, eb2… and going though to ‘other’ represent commonly identifiable chronological periods in the south-west Aegean. For each artefact, one or more specialists has suggested the probability that it belongs to a particular period (hence for each row the overall probability sums to 100). This allows us to analyse and map out the uncertainty that is always present in artefact dating, especially in the context of often undecorated, abraded or coarse survey finds.
2. We will now plot the finds for the sample area. Close the table if you have it open in front of you and right-click on t_pottery.csv and choose the option ‘Display XY data…’. In the resulting dialog pick suggX as the X field and suggY as the Y field and then click OK. This will plot the finds at their appropriate locations.
4. This is best seen if you turn of the tract polygons with the original density information, so please untick the box next to tracts in the Table of Contents. Then right click on ‘t_pottery.csv Events’ in the Table of Contents and choose Zoom to Layer. If you have done this correctly your view should look like the one below:
Note that this shows the prehistoric pottery from both stage-one tractwalking (shown as orange crosses) and stage-two gridded collection (shown as brown dots) around two small scatters. These are located at either end of a thin patch of alluvium and preliminary study suggests that they are of Cretan First- to Second Palace date (ca. 1950-1450 BC).
6. Now click on the ‘Selected’ button at the bottom of the table to isolate just those that you have selected. Scroll across to the right hand side and note that none of these sherds have a definite fpal (First Palace or Cretan Protopalatial) or spal date (Second Palace or Cretan Neopalatial) but can be attributed to these or other prehistoric phases with varying certainty (with the latter certainty depending on diagnostic features associated with the sherd’s shape, fabric, decoration etc, and hopefully improving over time as we study the material further). Note that, for this sample dataset, we have removed the date fields used for later periods, but do apply the same principle when dating sherds from any period.
Take some time to zoom in on one of the scatters and note that most of the sherds have a relatively low probability of being assigned to any particularly period. This is because many are heavily abraded surface finds and not very diagnostic (particularly in the vacuum circles of the stage-two grid squares where all material is collected, including small, shapeless coarseware fragments). In fact, this, rather than easily dated finds, is usually the reality of surface material, though the degree of ‘diagnosticity’ does of course vary by context, period, observer, etc and hopefully improves with further study.
Despite this uncertainty, we suspect that these two particular scatters represent small, mid-second millennium BC farmsteads (perhaps for one or two families?) and the lower level material residue of the agricultural activities that went on around them.
VII. Databases – Archived Artefact Photos
ASP has made an archive photograph of every finds bag brought in by surveyors and makes these publicly available by anonymous ftp. Here we will have a look at the tractwalking finds we selected earlier in section VI.5 and that were collected by walker 57 about 20-30m into tract 12042.
These photos are labeled by concatenating the collection unit components that describe where they were found (i.e. combining tract-walker-pass-segment for tractwalking finds and square-segment for finds from gridded collection). This particular bag includes some Late Roman material and also the five sherds (shown on the left-hand side in this case) of prehistoric pottery we considered earlier. Such an archive allows ASP pottery specialists to make quick checks on their records even when they are not in the lab and also allows us to ground the otherwise rather bland and disassociated GIS data with the actual recovered material.
