|
LBCS METADATA PRIMER Guide to Documenting Land-Use Data Using Land-Based Classification Standards
Abstract Documenting land-use data is key to its proper usage. The quality and reliability of the land-use data can vary widely and knowing the characteristics of these data is essential. Planning studies depend on data about change in land-use patterns over time. This is a practical guide for the formal documentation of digital land-use data that employs LBCS. Formal documentation is metadata, or "data about data." Since much of the metadata is on the Internet, this guide also gives extensive links to other online sources. This guide is meant for local planning agency staff collecting or maintaining land-use data. Others using such well documented data can also benefit from this metadata guide. We begin with a discussion of the role and importance of metadata in sharing digital data. This is followed by an overview of the Content Standards for Digital Geospatial Metadata (CSDGM) adopted by the Federal Geographic Data Committee (FGDC) followed by the steps required to begin collecting and using metadata. The next section deals with how to select the proper metadata creation tool from the growing number being developed. It is followed by a tutorial for implementing CSDGM metadata for digital geospatial data utilizing LBCS. The guide concludes with a discussion of other relevant metadata issues. Table of Contents
NOTE: Some links to tools and external documents in this document are broken either because newer tools have replaced the versions mentioned here or the servers have changed. Use the broken URLs to find updated tools in Google or other search engines. Planning and management of land resources require the acquisition and integration of large amounts of spatial data to be effective. Increasingly, geographic information systems (GIS) are being utilized to manage, integrate, and analyze spatial data for use in land-use planning and management. As the use of GIS becomes more widespread, interest in sharing spatial data for use in land-use planning and management is increasing. The quality and reliability of land-use data vary widely. It is critical that these data be well-documented, so that informed decisions can be made about their use. The Land-Based Classification Standards (LBCS) Project is designed to support sharing of land-use data among a variety of organizations. The project, undertaken by the American Planning Association with support from eight Federal agencies, is developing a new multi-dimensional land-based classification standard for the nation. The project focuses on three types of land-based information--land use (the built environment/human use of land), land cover (natural features, vegetation, surficial geology, etc.), and land rights (ownership, rights in land, zoning, etc.). The draft LBCS classifies land-based data in the following dimensions:
The LBCS can be applied to what are termed "taxonomic units" or "measurement units of data collection" (APA 1998, 6). The classification scheme for each of the five dimensions may be relevant to multiple taxonomic units. Taxonomic units include business units, floors of buildings, the entire building, parcels, parcel aggregates (census geographies or other units of analysis), grids, or other arbitrary units. When mapped in a digital form, these "taxonomic units" may be referred to as geospatial data. The term geospatial data is defined as "information that identifies the geographic location and characteristics of natural or constructed features and boundaries of the earth. The information may be derived from --among other things-- remote sensing, mapping, and surveying technologies." (FGDC 1997, 71). Another topic is especially important in the promotion of spatial data sharing -- metadata. Metadata can be defined as "data about data." Documenting data sets allows users to make informed decisions about the availability, transfer, and fitness for use of spatial data. The LBCS Metadata Primer is designed to provide a practical overview of the issues associated with developing and maintaining formal metadata for digital land-use data based on the Land-Based Classification Standards. The primary audience for the LBCS Metadata Primer is local planning agency staff, although it will be useful for other land-use professionals. The primer provides a "cook book" approach to the creation of metadata. Because much of the most current information on metadata resides on the Internet, the primer summarizes relevant material available from other World Wide Web (WWW) home pages. The LBCS Metadata Primer begins with a discussion of the role and importance of metadata in sharing digital data. This is followed by an overview of the Content Standards for Digital Geospatial Metadata (CSDGM) adopted by the Federal Geographic Data Committee (FGDC). Section 4 focuses on the steps required to begin collecting and using metadata. The next section deals with how to select the proper metadata creation tool from the growing number being developed. Section 6 is a tutorial for implementing CSDGM metadata for digital geospatial data utilizing the LBCS. The primer concludes with a discussion of other assorted metadata issues. The LBCS Metadata Primer has been modified from a similar report prepared for the National States Geographic Information Council (NSGIC). The NSGIC Metadata Primer is one phase of a larger metadata research and education project undertaken by NSGIC and funded by the Federal Geographic Data Committee's Competetive Cooperative Agreements Program (CCAP). NSGIC's mission is to encourage effective and efficient government through the coordinated development of geographic information and systems to ensure that information may be integrated at all levels of government. More information on the NSGIC Metadata Research and Education Project can be found at the following Web site (http://www.lic.wisc.edu/metadata/metahome.htm). While we call this a 'cook book' of sorts for metadata, the metadata recipes you derive from it will be flavored by your institutional arrangements, your GIS and operating platform, and your data variety and volume. Bon appétit!
Section 2 -- Metadata: What is it and Why is it Important? At first glance, the term metadata evokes a technical image and almost guarantees a trip to the dictionary. Metadata is not viewed as a "user friendly" topic, but this image is not completely deserved. Simply defined, metadata is "data about data." Used in the context of digital spatial data, metadata is the background information which describes the content, quality, condition, and other appropriate characteristics of the data. Paper maps contain metadata, primarily as part of the map legend. In this form, metadata is readily apparent and easily transferred between map producers and map users. When map data are in a digital form, metadata is equally as important, but its development and maintenance often require a more conscious effort on the part of data producers and the chain of subsequent users who may modify the data to suit their particular needs. Metadata serves many important purposes, including:
Metadata may soon play an important role in the provision of actual insurance policies within the GIS profession. Gary Hunter of the University of Melbourne recently wrote an article for URISA News on the implications of the increasing trend toward purchase of database insurance policies for spatial data sets used in high-risk GIS application areas, e.g. emergency response (Hunter 1996, 1-3). Hunter points out that insurance companies will likely require conditions for the issuance of such policies ranging from detailed background information on the organizations producing and using the data to a certified quality assurance program in place. Another obvious condition would be comprehensive metadata on the data sets in question. Metadata may exist in forms other than ones compliant with the Content Standards for Digital Geospatial Metadata. (The Content Standards will be described in detail in section 3.) Perhaps the most common form of metadata is a file folder filled with notes on data sources and procedures used to build the data. Less common is complete, organized metadata such as the Wisconsin Department of Natural Resources' GIS Data Users Guide. This catalog provides concise metadata in a form that is easy to read and comprehend. CSDGM compliant digital metadata may be created, stored, and used in a variety of formats. The most basic is an ASCII text document. An ASCII document is easy to transfer to other users independent of the hardware/software platform they use. Another common format is Hypertext Markup Language (HTML). HTML provides an attractive way to view metadata using a browser such as Netscape Navigator, Mosaic, or Microsoft Internet Explorer. Recently, there has been strong interest in creating metadata in Standard Generalized Markup Language (SGML). SGML provides an effective way to tag metadata elements. This will be important for indexing and searching metadata on Clearinghouses and to provide a means to exchange metadata between metadata users, metadata databases, and metadata tools. 2.4 Geospatial Data Clearinghouses A Geospatial Data Clearinghouse is a location, typically accessed through a home page on the World Wide Web (WWW), to search for spatial data sets. A Clearinghouse may contain specific data sets which can be downloaded or may contain information about data sets that aid the user in making a determination about whether it is worthwhile to obtain the data set and how to do so. The existence of many Clearinghouses of geospatial data and metadata may seem surprising to persons new even to the CSDGM. Well, Clearinghouses are out there and they have both data and metadata! If you are hesitant to jump into metadata, don't worry, the water is fine! Here are some robust examples of spatial data/metadata clearinghouses:
Table 1
2.5 Who is Going to Create the Metadata? This is not an idle question. Metadata creation is typically considered to be an obligation of the data producer. Although you may be a data producer and custodian, it may not be most effective for you to learn the CSDGM and produce the metadata if you only produce/document a few data sets per year. If your state has a GIS coordinating council, there may be individuals specially trained to help you produce metadata for your data sets if you will contribute the metadata to a Clearinghouse. 2.6 Additional Information on Metadata For additional background information on the importance of metadata, please review the following brochures and on-line material:
Section 3 -- Getting Acquainted with the Content Standards for Digital Geospatial Metadata (CSDGM) 3.1 Overview of Content Standards for Digital Geospatial Metadata Metadata, or "data about data," describes the content, quality, condition, and other characteristics of data. The Content Standards for Digital Geospatial Metadata, Veresion 2 (CSDGM) specify the information content of metadata for a set of digital geospatial data. The purpose of the content standards is to provide a common set of terminology and definitions for documentation related to these metadata. Information about what elements of the metadata are mandatory, optional, repeatable, or one of a choice are encoded in the production rules of the CSDGM. The first impression of the CSDGM is its apparent complexity; in printed form it is about 75 pages long. This is necessary to convey the definitions of the 334 different metadata elements and their production rules. Do not let the length dismay you; the CSDGM is meant to be a reference, not recreational reading! The content standards are meant to be a framework to convey those things you need to know about a data set to evaluate its usability, to obtain it, and to use it effectively. To help better understand the CSDGM, it is useful to break it down into its major sections, not all of which may necessarily be required or present in the metadata for a data set. 3.1.1 Major sections of the CSDGM
Additionally, the content standards define three 'floating' minor sections.
The minor sections are never used alone, but are always inserted into one of the major sections as a block. Because they are used multiple times, it made the content standards more compact to define them separately. Contact Information, in particular, is a section of which a site might have several instances, and common to all or many metadata documents. This situation may be used to advantage in site specific metadata template documents, or database based metadata tools. If only the 'mandatory' elements of metadata are included, the metadata is rather brief and may be quickly prepared as shown in this tongue-in-cheek 'minimum metadata document.' The minimum mandatory elements may be appropriate for initial data set documentation, but should not be substituted for complete documentation. Some elements of the content standards are termed compound elements, because they (parents) are composed of, or are containers for other sub-elements (children). Typically this parent/child relationship is indicated in metadata by indenting the child element one level deeper than its parent (hierarchical indentation), or with numbers, by adding another decimal point refinement to the element number, e.g. a 2.1 parent might have 2.1.1 and 2.1.2 children. Neither the indentation or the numbering is a required part of the content standards. 3.2 Resources that Document the Content Standards or Help in Understanding Them
3.2.1 Example metadata Metadata, even that conforming to the CSDGM, does not all look the same. This is because the CSDGM, as its name implies, specifies only the content of the metadata, not its format. In the past, this has led to a number of difficulties in incorporating data from different organizations who use different metadata tools or template documents into a common clearinghouse. The different forms might employ hierarchical indentation of the elements, the numbering system of the CSDGM, colons following the element names (or none), string the element names together with underlines (e.g. Identification_Information) or leave them as separate words. Luckily, there is a tool which will bring some order to this allowed chaos, Peter Schweitzer's cns. The cns software (which stands for chew 'n spit) reads in metadata files and produces output with consistent element names which can be read by other metadata tools. Some CSDGM compliant documents in various formats follow.
These are just a few examples of metadata conforming to the CSDGM. It appears that there is a lot of latitude in the form of the metadata. This is true, and content is much more important than form, but there are several points to keep in mind if you want your metadata to be easily incorporated into an National Spatial Data Initiative (NSDI) Clearinghouse:
There are innumerable other metadata documents which can be utilized as sample metadata, although one must be cautious about what is selected for use as an example. Just because a document is billed as metadata doesn't mean that it is CSDGM compliant. Peter Schweitzer of the USGS has put together a package of sample metadata documents which may be useful as examples for metadata. Although they are CSDGM compliant, there are no examples in the sample metadata of data sets used by local governments - e.g. tax parcels or road centerlines. The FGDC has recognized that example metadata is important to help understand the content standards. Moreover encapsulating what geospatial data are available will be helpful to users who do not currently have good Internet access. For these reasons, the FGDC has commissioned Applied Geographics, Inc. of Boston, MA to compile metadata from existing Clearinghouses and solicit metadata from the general NSDI user community for the production of a CD of metadata, with a (hopeful) estimated six month update cycle. When that becomes available, you will find information on how to obtain it here!
3.2.2 Printable
versions of the content standards
3.2.3 The Green
Book
Include in your order (as applicable) your Name, Position, Organization, Street Address, City, State, ZIP (or postal) code, Country, Telephone number, FAX number, and email address. You can also FAX your request for the Workbook to (703) 648-5755
3.2.4 The
On-Line version of the CSDGM
3.2.5 Graphical
Representation of the CSDGM
3.2.6 Colorized CSDGM
3.2.7 Power Point
presentations
3.2.8 Metadata Resource CD Like the example metadata CD, when the Metadata Resource CD becomes available you will be able to find out here how to obtain it. In a wicked recursion, this primer might also be found on the Resource CD.
3.2.9 Future Revisions to the Content Standards for
Digital Geospatial Metadata Another useful metadata exercise was been developed by Hugh Phillips of the Wisconsin State Cartographer's Office to support a Metadata Mentoring Workshop at the Wisconsin Land Information Association 1997 Annual Conference in Lake Geneva, Wisconsin. The definitions of the metadata elements in the CSDGM are succinct. Unfortunately, in their brevity, they sometimes lead to users wishing they had a little more expanded explanation or an actual example of what was being asked for. It was mentioned above that example metadata can be useful in this regard. Sometimes, though, one must just ask for additional explanation. Some of these questions come up on the listserver nsdi-l, however they have never been collected into a package. Eric Miller of Ohio State University and the Online Computer Library Center, Inc. has a searchable archive of postings to nsdi-l and GeoWeb which might uncover a question about a particular element. The Green Book contains a sprinkling of FAQs throughout its pages, and Peter Schweitzer of the USGS has compiled a collection of FGDC Metadata FAQs, including FAQs about his three metadata tools, cns, mp, and xtme. Ideally, the on-line version of the CSDGM would be expanded to include FAQs and examples. 3.5 General Metadata Resources
Section 4 -- Where Does One Begin? This primer started with a discussion of the importance of metadata. This was followed by a review of the Content Standards for Digital Geospatial Metadata. Now is the time to roll up our collective sleeves and tackle the task. This section outlines a strategy for collecting metadata at the agency or corporate level. The first step often involves getting an organizational committment to "do metadata." This may involve explaning the tradeoff of the short term costs versus the long term benefits of metadata implementation. Once an organizational committmemt is secured, the next step involves inventory of the spatial data holdings of the agency and prioritizing their importance for documentation. Examination of sample metadata can help one become more familiar with the range of approaches to documenting spatial data and to avoid later misinterpretation of the CSDGM. Finally, this section will examine the detail to which metadata can be collected, including complete (and exhausting) documentation, partial documentation through a template or profile of the CSDGM and thumbnail sketches of the most basic information about a data set. Spatial data sets seem to have the ability to multiply and fill up all available disk space and proliferate beyond the initial source of creation. Agencies often find it difficult just to keep track of the growing inventory of spatial data they have developed, much less fully document it. Although when one thinks of the significant resources that many agencies invest in database development, the time spent keeping a current inventory is certainly justified. For those using GIS software by Environmental Systems Research Institute (ESRI), there is a very useful spatial data inventory software program called Findarc by Geographic Designs, Inc. (web page no longer in service) Findarc is a proprietary UNIX-based product presently running on the SUN Sparcstation platform that will search a file space or directory and locate Arc/Info coverages, GRIDs, ArcView shapefiles and ArcView projects. Findarc identifies file names, software version, feature type, size, date and time of last edit, file ownership, status of topology, and information about map projections. This is the output from a Findarc session. The output is a comma delimited file that can be imported into ArcView, a spreadsheet, or a database for sorting and querying or filtered with a UNIX tool such as grep. With a spatial data inventory such as that produced by Findarc, one can begin the task of prioritizing the data most important to the agency. These data sets are prime candidates for early documentation, along with those that will be shared with other agencies or sold. There may be other reasons particular to an individual agency for placing a high priority on early metadata creation. This is a good point to discuss the timing of collecting metadata. Database developers often intend to document spatial data shortly after completion of data entry. However, it is human nature to put off technical tasks such as database documentation. Months or years may slip by before metadata creation is undertaken. Key attributes of the data may be forgotten in the ensuing time. The most efficient strategy for metadata creation is to make it an ongoing process during database development. 4.3 Metadata Examples An important step before actually initiating the creation of the metadata document is to review some sample metadata. Metadata may be collected in a variety of forms and to varying level s of detail. Examining how other agencies have documented their data holdings may provide insight into the most appropriate strategy for your agency. Examining sample metadata may also result in time savings through reduced effort associated with metadata creation. An example would be duplication of the documentation of a coordinate system already completed by another agency. Metadata may be collected at many different levels of detail, ranging from that supporting a quick "thumbnail sketch" of a data set to very detailed documentation of a data set which may support decisions involving life-threatening situations or protection of multi-million dollar investments. Some agencies may wish to go into great detail describing the purpose, access constraints, use constraints, or distribution liability associated with the data set. Other agencies may wish to document aspects of data sets which are not part of the content standards. Agencies may review the CSDGM and identify a minimal subset on which they wish to focus in order to minimize the effort associated with development of metadata. These subsets are often referred to as "core metadata." When structuring core metadata, agencies should make sure that the resulting metadata meets the needs of the agency in areas such as data management and archival, data sharing and transfer, clearinghouse support, and subsequent decisions on fitness of use in future situations. Core metadata for the sole purpose of minimizing metadata implementation effort may not serve the agency well. It should be noted that although the FGDC has considered the issue of 'core' metadata, there is resistance to defining such a set, i.e. there isn't an accepted set of 'core' metadata elements. The ' Dublin Core' is an example of a 'minimum searchable set' of metadata elements. The 'Metadata Summit' in Denver, February 1996, identified another set of metadata elements which is sometimes referred to as the 'Denver Core.' Although a 'minimum searchable set' and a 'core' set of metadata elements are probably closely related, most would agree that these would not be identical sets. Several states have developed metadata profiles to meet their specific needs. Metadata profiles are closer to the full CSDGM and may involve reordering or renaming FGDC metadata elements, adding or subtracting elements, or developing customized production rules or response lists for specific elements. The Minnesota Governor's Council on Geographic Information has developed the Minnesota State Geospatial Metadata Guidelines (WordPerfect document). The guidelines include a table that cross-references FGDC and Minnesota State metadata element names and identifies where there is either no FGDC equivalent of the state element or where there is a compound element related to the state element. The Idaho Geographic Information Advisory Council has developed the Idaho Metadata Profile in WordPerfect format (.wp) or Microsoft Word format (.doc) which includes a thesaurus (from the Directory Interchange Format manual) and production rules that relate the profile to specific FGDC metadata elements. The profile is accompanied by a metadata form in WordPerfect format (.wp) or Microsoft Word format (.doc) for users to print and fill out as hard copy. In general, metadata profiles serve a useful function, customizing metadata implementation for specific circumstances. Unfortunately, this can lead to metadata which may not be compatible with mainstream metadata software and the CSDGM. You have been exposed to the full CSDGM, 'core' metadata, and metadata profiles. It's now time to 'fish or cut bait,' i.e. it is time to pick one of these approaches to metadata. The authors recommend, in order to get started, that you stay with the CSDGM as opposed to defining any metadata profile which differs only slightly from the CSDGM.
Section 5 -- Select the Proper Metadata Tool 5.1 Deciding Between a Metadata Database or Discrete Metadata Documents Deciding between holding your metadata in a database or to produce discrete metadata documents for each data set is somewhat dependent on the variety and volume of your data sets, as well as how often they (and the metadata) are updated. This decision will determine which metadata tools are appropriate to consider for use. The FGDC recommends that metadata be stored in a database if your data sets are subject to frequent change, or if some of the metadata is common to many of your data sets (e.g. a data set maintained in tiles). With a database for metadata, it may be necessary to write a specialized output report generator to produce CSDGM metadata when needed or for submission to an NSDI Clearinghouse where it can be made searchable (Metamaker and NOAA metadata tools already have built-in report generators to produce CSDGM compliant, or nearly compliant metadata). On the other hand, if your site is an NSDI Clearinghouse, then it may be possible to write the appropriate SQL interface between your Isite server and your database to allow the database to be queried directly, without the need to produce any intermediate discrete metadata documents. If your data holdings have few metadata elements in common, then discrete metadata documents are a simple way to hold your metadata, and almost any tool can be used to produce it. Making the change from discrete metadata documents to a database system is entirely possible if your metadata is brought to a common form with cns and mp. It is likely that some standard metadata database structures will be developed and conversion programs will allow compliant metadata to be imported into those structures. 5.2 GIS / Operating System and Other Considerations The tools which are an option for your site depends on the hardware and operating systems that are available, what your GIS package is, and perhaps even the approval of management at your agency. It is certainly desirable to use a tool which operates on hardware and under the operating system that you are familiar with. If your GIS supports a metadata function, or there are metadata tools which are specific for it, then using that built-in or specific tool is probably the most efficient way to produce at least a portion of the metadata. Unfortunately, at the present time, there are no Geographic Information Systems with extensive or CSDGM compliant metadata functionality built-in. There are some metadata tools which are specific for workstation level Arc/Info. If you are working under a specific metadata profile, then there may be a tool which is tuned to it. Examples are the new Metamaker for National Biological Service extensions, DataLogr for the IMAGIN Data Sharing Network, and mdc for the Florida Data Directory. The following is a breakdown of metadata tools by GIS/platform/OS:
5.3 Categories of Metadata Tools Metadata tools may be separated into categories based on their operating characteristics and function. The following four categories of metadata tools seem distinct:
5.4 The Available Metadata Tools and How to Get Them Based on the above material, you may have now arrived at a specific category of tools, or possibly even the most appropriate tool. Now you may want some more in depth information about it, or perhaps even a critical review by someone who has used it. You are in luck -- there are several reviews available which also give links to where to obtain these tools.
Finally, keep in mind that metadata tools are evolving rapidly. Watch for postings to nsdi-l announcing new tools or new versions of tools which may not be covered in the above reviews.
Section 6 -- LBCS Metadata Tutorial The previous sections of the LBCS Metadata Primer described the role of metadata, provided an overview of the Content Standards for Digital Geopatial Metadata, developed a strategy for metadata implementation within an organization, and provided a review of metadata tools. Armed with this information, you are ready to begin experimenting with metadata creation. This section is a tutorial for metadata creation associated with digital land-use data. It starts with the selection and installation of a suite of metadata tools. This is followed by completeion on an exercise to learn the operation of the metadata tools. The third stage describes some useful resources for metadata implementation. Finally, a sample metadata document for digital land-use data for a study area near downtown Madison, Wisconsin is analyzed. 6.1 Download and Install the Metadata Tools The metadata tools selected for the LBCS Metadata Tutorial were developed by Peter Schweitzer of the U.S. Geological Survey. The metadata tool kit includes either Xtme or Tkme as the metadata creation software, mp as the metadata validation software, err2html to assist in review of metadata formatting errors, and (if needed) cns as a metadata pre-parser. Xtme is the metadata creation tool of choice for those running on a Unix platform. Tkme is a port of Xtme that runs under Microsoft Windows 95 or NT. Xtme/Tkme was chosen because the menu-driven interface is easy to use, there is an extensive help file system which incorporates the production rules of the content standards, its cut and paste function supports copying sections within or between sessions, and its output is readily ingested by mp.The metadata tools are all available at the USGS "Formal Metadata" home page (http://geology.usgs.gov/tools/metadata/). Assuming you will be running Tkme under Microsoft Windows 95 or NT, download the self-extracting archive file ("all_w95.exe") that contains Tkme, mp, cns, documentation, and source code. (If you will be running Xtme under Unix, select metadata tools for the appropriate operating system: DG/UX, HP/UX, Linux, Solaris, or IRIX.) In order to extract the metadata tools once they have been downloaded, double-click on the file "all_w95.exe" from Windows Explorer. The default subdirectory for the files is c:\usgs. The executable files will be placed in the \metadata\tools\bin subdirectory. The source code will be placed in the \metadata\tools\src subdirectory, and the documentation will be placed in the metadata\tools\doc subdirectory. The "winstart.txt" file in the \metadata\tools subdirectory will help describe how to set up the tools once they have been downloaded. It describes how to set up an Windows icon to run Tkme and how to run mp and cns from a DOS prompt. At this point, take time to download the " err2html.exe" file. from the "Formal Metadata" home page. Save it to the \usgs\metadata\tools\bin subdirectory. Now that the appropriate metadata tools have been obtained and installed, it is time to learn their o peration. An exercise that demonstrates the use of several metadata tools (including Xtme, mp, and cns) was developed by Hugh Phillips for the Metadata Mentoring Workshop at the Wisconsin Land Information Association 1997 Annual Conference in Lake Geneva, WI. The exercise begins with a review of the Content Standards and general concepts for metadata implementation. If you would like to refresh these concepts and learn about the use of other metadata tools such as MetaMaker, start at the beginning of the original "Metadata Mentoring Workshop" exercise. Otherwise, we will focus only on the metadata tools that we just downloaded. Three sections of the exercise have been extracted and modified to
6.2.1 Download sample metadata files Download the sample metadata files needed for the exercise. Use winzip to uncompress the files and place them in a subdirectory named \usgs\metadata\tools\exercise\. 6.2.2 Use of mp mp, the metadata parser, was written by Peter Schweitzer of the USGS. It can perform both of the following functions on properly formatted metadata:
mp is very picky about two things:
To begin, start up a MS-DOS session from either through the Start Button/Programs selection or by creating an Icon for the DOS prompt on the desktop. Navigate to the \usgs\metadata\tools\exercise\ subdirectory. Remember to include the \usgs\metadata\tools\bin subdirectory as a path in the autoexec.bat file, so you can run the metadata tools from any location. Type mp without parameters to see its command line options.
Validate some clean metadata Okay, first use mp on maratax.asc, a 'good' metadata file, to produce an error report, some nice HTML metadata, and metadata marked up in SGML by typing:
Take a look at the error report, maratax.err with a text editor like
Notepad or WordPad. The
warnings are just that - in the identification of the Marathon
County coordinate system under Other_Grid_System's_Definition,
mp has detected some valid metadata element
names that seem out of place (they follow a free text domain
element). The errors for Abscissa_Resolution and
Ordinate_Resolution are more serious - these are mandatory
elements that expect a real number, not the text string
'unknown.'
Take a look at the HTML output, maratax.htm in \usgs\metadata\tools\exercise\ with your Web browser. Take a look at the SGML (Standardized General Markup Language) output, maratax.sgm in \usgs\metadata\tools\exercise\ with a text editor. It looks pretty ugly, no? It wasn't meant for people to read or work with directly, but it makes it much easier to index the metadata. To those familar with HTML tags from Web pages, SGML markup will look similar, with start and end tags around elements. This is the last we will look at SGML. Validate some 'slightly dirty' metadata The next file we will work with has some errors in it that I hope you can find and correct. Run mp on the file apbad in the \usgs\metadata\tools\exercise\ subdirectory and produce an error report:
The error report should look like this. Try to figure out what caused
these errors, fix them, and run apbad through
mp again. Less errors this time, I hope?
Remember that often errors are related, and that by fixing one
key problem, many others may disappear. Look for blocks of
errors, and usually the culprit will be revealed. Here is the
explanation of the error report for
the apbad file.
Did your repair work actually cause the reported number of errors to increase? If so, you are a natural for this stuff! It means you actually tried to do something, instead of saying "Yeah, yeah this is easy," and just skipping on ahead. Probably what happened was that you messed up the hierarchical indentation. cns, which we will look at next, can deal with that kind of problem. Don't spend a lot of time slaving on trying to get this data to pass mp perfectly, we are just sort of fooling around with these tools here, not too serious, and there is a bunch of other stuff to do! Extra Credit: mp with Customization The operation of mp can be customized through use of a configuration file, the use of which is detailed in mp's metadata (\usgs\metadata\tools\doc\mp.met). As an example here, we can produce the error file, sgml file, and html file (which is now customized with links to a description for each metadata element header) by specifying a configuration file and an input file, like so:
Get a directory:
Take a look at the HTML output file,
\usgs\metadata\tools\exercise\apgood.htm with your Web
Browser.
6.2.3 Use of cns cns is a formatter for metadata which makes it more acceptable to mp. Like mp, cns was written by Peter Schweitzer of the USGS. Some of the functions cns can accomplish are:
To begin, start up a MS-DOS session from either through the Start Button/Programs selection or by creating an Icon for the DOS prompt on the desktop. Navigate to the \usgs\metadata\tools\exercise\ subdirectory. Type cns without parameters to see its command line options.
Preclean the 'slightly dirty' metadata file In the last exercise with mp, several of the errors were caused by indentation problems, minor errors in the presentation of element names, and actual missing container elements. These are the sorts of things cns can help with. For an example, let's run a clone (apbad2) of the 'slightly dirty' file (apbad before we fixed it up) through cns. First, lets see that apbad2 really is 'slightly dirty.' Type:
The error report apbad2.err should look familiar.
Now, let's preclean apbad2 with cns, produce a leftovers file, and an output file. Type:
Now run the output from cns through
mp:
Look at apbad2.err. Not bad, huh?
cns, shining bright As we showed above, cns can clean up minor 'glitches' in metadata. However, it really shines when the formatting of the source metadata has no hope of getting through mp without hundreds of errors. We are talking here about metadata files that would take hours of editing with a word processor to make acceptable to mp. The output from Metamaker (a leading DOS/Windows metadata tool) is a prime example. First, let's try to jam some Metamaker 2.0 output, gmgb93, directly through mp. Type:
Look at the error report,
\usgs\metadata\tools\exercise\gmgb93.err.
Yowza! 315 errors! Okay, now let's try again after passing gmgb93 through cns.
Take a look at the formatting of the output
(\usgs\metadata\tools\exercise\gmgb93.out) from
cns. It's quite a bit different from the
source gmgb93, no?
Pass the output from cns through mp.
Now look at the error report from
mp for the output from cns.
It looks much better now, only 37 errors,
most of them due to a characteristic of Metamaker
2.0 to pass through some empty container elements to
the output report. Metamaker 2.1 is improved
in that respect.
If you do process much MetaMaker output, then you should use an element name alias file to help cns recognize and convert some of the non-CSDGM element names which MetaMaker uses in its output report. Up to this point we have been ignoring what 'chew and spit' (cns) puts in its leftovers file. (Doug Nebert has suggested to rename this file the 'spittoon' file) The leftovers file is where cns puts stuff it can't figure out what to do with, or that it has concluded is chaff. You need to insure that cns hasn't thrown away anything of value, because what it throws into the 'leftovers', doesn't come through in the output file.
Annoying 'feature' of cns: Reserved element names that begin a line in a free text passage cause that line (and all lines that follow, until another valid element name can be recognized) to be dumped into the leftovers file.That may be a little hard to picture, so here is an example of the problem from cns' author, Peter Schweitzer. There is no clean way to deal with this problem (yet), so I suggest any of the following quick (and dirty) kluges if you find cns is throwing stuff in the leftovers file:
6.2.4 Use of Tkme Tkme is a Windows-based metadata editor. It is specifically designed as a port of Xtme to Microsoft Windows 95 and NT. This may be starting to sound like a broken record, but like mp and cns, Tkme was also written by Peter Schweitzer of the USGS! Tkme supports the entry of values for metadata elements by direct keyboard entry of the value or by pasting in a Window. Its most powerful function however, is its ability the manipulate the structure of a metadata document. In fact, if the hierarchical structure of metadata could be likened to a tree, then the function of Tkme could be likened to the skills of an expert gardener in its ability to graft and prune parts of the metadata tree. In this part of the exercise you will explore some of the powerful functions of Tkme, not with the intention of creating a some metadata you will save, but rather with the idea of putting Tkme through its paces. Although this exercise will demonstrate some of the functions and operational characteristics of Tkme it is by no means an exhaustive demonstration. You should examine the metadata for Tkme, \usgs\metadata\tools\doc\ for other details. Start up Tkme by clicking on the icon that was created as part of the setup process. Tkme puts up two windows, the application window and a "console" window for displaying error messages. You can safely minimize the console window while running Tkme. The application window should open with "Untitled" in the title bar across the top. The application window is divided into two parts. The top half is shaded in gray and has a menu bar at the top. The bottom half has a white background. Click on Add in the menu bar and select Metadata. This adds "Metadata" as an entry in both the top and bottom windows. Note that the entry in the top window is highlighted in blue. Click again on Add in the menu bar and move down the choices to (all of the above). In the upper window you should now see the seven major sections of metadata listed hierarchically under the word Metadata. The upper window will always show the structural skeleton of the metadata, that is the metadata element names which are present. The lower window is the edit window in which you can type or paste values for metadata, unless the element highlighted in the upper window is a container element. If the latter is the case, then the lower window mirrors what is highlighted in the upper window and acts as a reminder that this element does not require a value to be entered. I may refer to the upper window as the 'element window', and the lower window as the 'value window' in the following material. File Menu Before you go any further, you should give your file a name. Position the mouse pointer over the word 'File' on the menu bar, hold down the left mouse button and drag down to 'Save As.' A message box will pop up, prompting you for a file name. Let's use 'tester' for the file name. Also, periodically during this exercise, you should revisit the File menu and select 'Save.' Highlight Identification_Information in the element window and notice that the element name is mirrored in the value window (its a container element). Go to Add on the menu bar and hold down the left mouse button. Notice that the Add menu has now changed to reflect what can be added as elements under Identification_Information. Select (all of the above) again. These new elements add hierarchically under Identification_Information in the element window. There are probably now more elements than the element window can display so a scroll bar may have appeared on the right side of it. To scroll the element window up or down position the mouse pointer over the dark part of the scroll bar, depress your middle mouse key and drag up or down. Highlight Keywords in the element window, then from the Add menu, select Theme. With Theme highlighted in the element window, go to the Add menu and select Theme_Keyword_Thesaurus. This new element appears in the element window. Notice that now, the value window is blank -- it does not mirror the metadata element name. This indicates that this is an element which requires a value (it is not a container element). Mouse click in the value window (changes focus to lower window) then type in:
We now need to add a Theme_Keyword, so we go back to the Add menu, but alas, Theme_Keyword is not available as a pick. In fact, all the menu says is 'Enter Value Below.' Here is the reason: In order to add a child element, its parent must be selected in the element window. Select Theme in the element window and from the Add menu, select Theme_Keyword. It now appears in the element window, and the value window is blank. Mouse click in the value window and fill it by typing:
Duplicate
We want to add another Theme_Keyword, and we could do it by
repeating the above procedure. There is an easier way,
though. Assuming that Theme_Keyword is still highlighted in
the upper window, select Duplicate from the Edit menu.
Another Theme_Keyword is added into the element window, and
Help Select Version from the Help menu. A new button should appear in the Windows Task Bar at the bottom of the screen which gives hints about using Tkme. Click on the Help button to view this new window. Grab the title bar of this new window and drag it away from Tkme's editing window so it doesn't obscure it. Dismiss the Help window to make it go away. Select Element from the Help menu and see that it gives the definition and production rules for the element selected in the element window. Select a different element and notice that the Help window reflects this newly selected element. What is displayed here comes from the help file specified when Tkme is started. ( Select Output from the Help menu. The Help window now displays all of the metadata elements that you have added, and all of the values that you have entered. The Output window reflects exactly the structure of your current metadata document which is just a plain ASCII document. Very nice. Note! Output doesn't update while you are entering values, but it will as soon as you select Update from its menu bar. View This is pretty self explanatory. View controls the hierarchical depth of metadata elements which are visible in the element window. View All, 1, 2, and 3 levels affects all elements. Hide and Show operates relative to the currently selected element and only affects the children of the currently selected element. 'View All levels' makes all elements to the lowest hierarchical depth visible in the element window. 'View 1 level' makes only the first hierarchical elements (like Identification_Information and Distribution_Information) visible. If an element has children which are not visible this will be indicated by the presence of a + sign to the left of the element name. Show reveals one more hierarchical depth below the currently selected element (assuming there are actually elelents to be made visible). Hide 'hides' all children elements of the currently selected element. Order Dependent Add Select 'All Levels' from the View menu, then select the Keywords element in the element window. From the Add menu select Temporal. Select the Keywords element again, then select Stratum. Select the Keywords element again, then select Place. So now the Keywords fragment of our metadata tree looks like so:
Keywords
Theme
Theme_Keyword
Theme_Keyword
Theme_Keyword
Temporal
Stratum
Place
Notice that these elements do not reflect the order in which
they are given in the Add menu, and the order in which they
are listed in the CSDGM. There is nothing in the CSDGM that
says anything about the order of sibling metadata elements, so
really there is nothing really wrong here. However for
consistency and to keep elements of metadata placed in the
order people come to expect them when reviewing metadata, it
would be helpful to reorder these Keyword elements into the
more standard order of Theme, Place, Stratum, Temporal. Swap
will do this.
Swap Swap exchanges the currently selected element with its nearest sibling element above it. Select the Place element, then Swap from the Edit menu. Repeat the operation twice more, moving the Place element above the Theme element (notice that all of Theme's children were swapped with it). Select Theme, then Swap from the Add menu to place Theme and Place in the standard order. Cut Cut (Edit menu) removes the currently selected element (and all its children). What gets 'cut' is still held in memory and is available to be 'pasted,' thus forming an alternate way to swap items without having to swap repeatedly. Cut the Temporal element from the metadata. Paste [Note: Paste from the Edit Menu did not work on Tkme version 2.2.3 as installed on Windows 95. Instead substitute Control-V for Paste in the following section. The notation (^V) will be added as a reminder. ] Paste (Edit Menu) attempts to paste the current selection (if it is a metadata subtree) appropriately relative to the currently selected element. If a sibling element is selected as a target the subtree is pasted immediately below the selected element. If a sibling isn't selected, but rather a parent element or the child of a sibling, then the element is placed at the bottom of all its siblings. For example: Select Keywords in the element window, then select Paste from the Edit menu (^V). Temporal is placed underneath Stratum. Cut Temporal again. Select Place, then select Paste from the Edit menu (^V). Temporal is placed underneath Place (a high speed swap!). Cut Temporal again. Paste is good about not letting you paste a subtree where it shouldn't go. For example, select the Spatial_Domain element, then Paste from the Edit menu (^V). Nothing should have appeared under Spatial_Domain, and a message stating 'Sorry, this information cannot be placed properly.' Nothing happened because Temporal is not a valid child element of Spatial_Domain. Select the Keywords element again, then Paste from the Edit menu (^V) to put Temporal back where it belongs before you lose it. Copy Copy makes a copy of the currently selected element and its subtree children available as a selection without removing it from the existing tree (very different from Cut). Select the Theme element, then select Copy from the Edit menu. Select the Keywords element, then select Paste from the Edit menu (^V). The Theme element and its children appear below Temporal. Select Output from the Help menu, and see that the values for the children of both instances of Theme are identical. Clear Clear 'zeroes out' the values for the elements under the currently selected element without removing the metadata element skeleton. Select the second instance of the element Theme then select Clear from the Edit menu. Select Output from the Help menu and observe that the Theme_Keyword_Thesa urus and Theme Keyword Elements no longer have values (under the second instance of Theme only, the first instance is unchanged). Prune Prune removes all elements below the currently selected element whose children are all devoid of values (i.e. there is no entered metadata in them, only the metadata element names are there). Hmmm... Remember back when we worked with cns, we showed that it could clean up a MetaMaker document pretty well, but that MetaMaker passed through a number of empty elements which cns also passed on but that mp complained about because they were empty? This is a quick job for Tkme.
Close whatever document you were working on, then make a
copy of that MetaMaker output that was
processed with cns
Now open this document up with Tkme. Select
Open from the File pull-down menu and navigate to
Select Output from the Help menu and scroll down to the end of the document. In the Metadata_Reference_Information section you will see a number of empty elements. Select the Metadata element in the element window, then select Prune from the Edit menu. Save prunetst, then Quit from Tkme. Run prunetst through mp
Look at the error file, prunetst.err. Now there is an
improvement - down to six errors from the former thirty seven!
Serious mucking about - pasting from one Tkme session to another Open your 'tester' document with Tkme.
Under the Description element in
Indentification_Information, Add an Abstract element. From
the value window you will see that Abstract is an element
that requires a value. We are going to paste it in from
another source. Open Notepad and select the
Okay, now lets do something similar from another
Tkme session. Close your
Notepad window, then start up another Tkme
session with the Cut the Distribution_Information element from your 'tester' document in your first Tkme session. Highlight Distribution_Information in the csdgmtab.txt document (second Tkme session) then select Copy from the Edit menu of that session. Mouse click back into the first Tkme element window and select Entity_and_Attribute_Information. Select Control-V (^V). The full Distribution_Information with all subtree children should now be appearing in your tester document. All Done? When you have finished editing a metadata file with Tkme, select Save, then Quit from the File menu. 6.3 Prepare to Create Metadata Metadata ResourcesThe following are general references you should have handy when preparing to create metadata:
Metadata Strategy Next, develop a strategy for which sections of the metadata document you will tackle first. The following is one possibility.
Metadata Sources Metadata creation is one subject where you won't be penalized for copying. Take advantage of |
Printer-Friendly Format
