Current Best Practices for Generalizing Sensitive Species Occurrence Data

Chapman AD (2020) Current Best Practices for Generalizing Sensitive Species Occurrence Data. Copenhagen: GBIF Secretariat. https://doi.org/10.15468/doc-5jp4-5g10.

Arthur D. Chapman

The document Current Best Practices for Generalizing Sensitive Species Occurrence Data is licensed under Creative Commons Attribution-ShareAlike 4.0 Unported License.

https://doi.org/10.15468/doc-5jp4-5g10

Version 1.0, November 2020

Based on the earlier publication by Chapman AD & Grafton O (2008) Guide to Best Practices for Generalising Sensitive Species-Occurrence Data. Copenhagen: GBIF Secretariat. https://doi.org/10.15468/doc-b02j-gt10.

Beneath the surface, Wollemi National Park, Australia. Photo 2015 Vern via Flickr, licensed under CC BY-NC-ND 2.0.

This document aims to provide best practice (or best current practice) for dealing with sensitive primary species occurrence data, and provide guidance on how to make as much data available without at the same time opening up the species to harm because data has been placed in the public domain.

It is now more than a decade since the first Guide was published, and this new publication is designed to bring those practices up to date and to incorporate the experiences gained by institutions that have implemented the Guide in whole or in part.

This work is designed for those who need, or want to know how they can best make as much data available on sensitive taxa as possible without that published data leading to harm to the species. This document is also for individuals or organizations faced with developing a policy on dealing with sensitive primary species occurrence data and writing in-house documentation consistent with current best practice.

Above all, this document will help end users of the data to understand the implications of trying to use records that may have been generalized to protect sensitive species, and how to understand the meaning of generalization at different precisions.

The term “best practice” generally refers to the best possible way of doing something. It is commonly used in the fields of business management, software engineering, and medicine, and increasingly in government. The term “current best practice” (or best current practice) is more specific in that it indicates the possibility for future developments and better practice. Due to the immaturity of this topic, this publication generally refers to “current” best practice and is sure to mature over time as more institutions adopt and adapt the principles outlined herein.

Two issues that this and the previous document have not covered are the issues of the privacy of living individuals and the development of data sharing and data licensing agreements. Both of these issues have legal implications and vary considerably from jurisdiction to jurisdiction. These issues have been covered in detail by others (e.g. Corti et al. 2000, Parry and Mauthner 2004, GBIF 2017, GBIF 2019, ALA 2018b, OEH 2019b).

Ethics and biodiversity is a topic that has received little coverage, although for hundreds of years biologists have followed implied ethics in their work. The management of data on sensitive data requires considerable ethical practice and in many cases, a lot of trust and collaboration. Often amateur biologists and citizen scientists are aware of the locations of sensitive taxa, and it is up to the biologists to work with these groups to ensure the continued survival of the species. This won’t always be possible as there will always be rogues, but collaboration can work. For example, working with amateur groups to ethically bring rare plants into cultivation so that there is less pressure on the wild populations.

Archaeology has been grappling with these issues for a long time. As Alison Wylie (1996) explains,

Nineteen years later, the archaeologists continue to grapple as pointed out by Frank et al. (2015) in discussing the results of 62 interviews of archaeologists and zoologists conducted by them. They concluded that “Researchers would generally prefer to restrict access to their data from the general public but maintain open data for colleagues”, while realizing the extreme difficulties in managing such a process.

This publication is largely focused on sensitive primary species occurrence data (e.g. museum and herbarium specimen data, observations, images, tracking data), but does have impacts on related data such as population sizes, numbers and viability, habitats and ecosystems, biogeography, traits, bycatch, biosecurity, etc.

This document only looks at taxon sensitivity and not sensitivity related to issues such as land or personal privacy. Different national and state jurisdictions have diverse legislation with regard to both land and personal privacy. Bearing that in mind, however, I would recommend following the same principles of generalization recommended in this publication wherever possible, rather than other methods such as randomization.

The National Biodiversity Network (NBN) in the UK (Countryside Agencies OIN 2007), and the Department of Environment and Conservation in New South Wales, Australia (Department of Environment and Conservation 2007) developed detailed sensitivity criteria, and the previous version of this publication (Chapman and Oliver 2008) relied heavily on the work of those two agencies. Since the publication of the previous Guide, both these agencies (DECCW 2009, NBN 2019a, NBN 2019b, OEH 2019a) along with the South African National Biodiversity Institute (SANBI 2010, SANBI 2016), the Atlas of Living Australia (ALA 2018a) and others, have given a lot of thought to criteria for determining sensitivity within their jurisdictions. Documentation from all of them have contributed greatly to this document.

A series of criteria for determining the sensitivity of taxa and data along with recommended metadata statements for documenting the reasons for the determination are set out in Table 1. The first two are for use by biodiversity data holders and those creating trigger lists of potentially sensitive taxa and refer largely to the taxa themselves. The last two are for use by biodiversity data holders and deal with an assessment of the data they hold and are considering making available – they are not suitable for the creation of trigger lists.

The criteria are used to determine:

A set of scenarios using Criteria 1 and 2 above to determine triggers for sensitivity of taxa is attached as an Annex to this document.

The first step in the process of determining sensitivity is to make an assessment on whether or not the taxon is subject to a harmful human activity or not and if the availability of related biodiversity data will increase the likelihood of the harmful activity occurring.

If it is not then there would appear no reason to list it as a potential environmentally sensitive taxon. It is recommended that you use the documented wording supplied but with additional supporting rational documenting the specifics of the threat, for example:

The next step is to determine if the taxon is sensitive to that human harm or whether they are suitably robust not to be adversely affected.

Once it has been decided that the taxon is subject to a significant risk and impact from harm or not, then a decision needs to be taken on whether the release of specific data on that taxon – or other related data – will increase the risk and impact of harm.

The final step is to make an overall assessment based on the three criteria above and to document the overall decision using the combined information documented in making each of the earlier decisions. Once it has been determined that the data should or should not be released, then it is important that a decision is made on the Category of sensitivity, and the level of generalization for the release of the data.

In the online survey (Chapman 2006), a number of respondents identified data awaiting publication, data subject to ongoing research, and incomplete or unchecked data as data that they would class as sensitive, and thus subject to restrictions on release. This is data whose sensitivity has a short time frame, and it is important that a time for release or review be clearly documented. They would most likely fall under criterion 3.3 above and would be documented accordingly with the supporting rationale being “awaiting publication”, etc.

The categories of sensitivity are largely based on those from the NSW Office of Environment and Heritage (DECCW 2009).

Few of the respondents to the online survey (Chapman 2006) recorded that they randomize data as opposed to generalizing it. Reasons for not randomizing included the extra work and computation involved, the increased chance of mistakes being made, and the less reliability that users may be able to place in the data.

Some respondents to the survey stated that they were comfortable with displaying presence/absence of sensitive data within large polygons or grids squares, because it still reflected the real data, but were aghast at the idea of deliberately ‘faking’ point coordinates such that locations appear as precise representations, but are randomly offset from the real data – i.e., they represent the deliberate introduction of error. Whereas generalization creates/retains ‘true’ data, randomization creates deliberately ‘false’ data.

In the fields of data mining and protection of privacy (including of individual privacy in census data), it is generally regarded that bottom-up generalization is far more practical and scientifically defensible than randomization (see for example, Wang et al. 2004, Dalvi and Keole 2015).

Another advantage of generalization is that it:

One of the most common requirements for generalizing sensitive biodiversity information is to generalize the spatial locality or geographic coordinates (Chapman and Wieczorek 2020). Traditionally this has been done in many ways, and there has been little consistency in methodologies and very little documentation as to what has been done in each case. This has considerably reduced the value of the data for analysis, and often users are unaware that the data has even been modified.

Generalization (at least in a spatial sense) is usually of one of two types, namely:

Many respondents to the survey (Chapman 2006) argued for the simplicity of generalization to a grid. The reasons given included:

Generalizing to a grid, while protecting the exact locations of sensitive taxa, also provides data in a format that is still useable for a majority of users, especially where a standard grid is used.

Figure 1 shows two methods that are commonly used to grid data. The first (recommended here) is a geographic grid (e.g. cartographic grids such as those based on geographic coordinates), the second-a metric grid (e.g. Europe Equal Area Grid 2001 – EPSG:19986). How each of these should be handled with respect to determining location and uncertainty, see Chapman and Wieczorek (2020). We recommend the use of a geographic grid, as discussed below, because of the ease of preparation and documentation, and because biological data being shared via the Darwin Core standard (Wieczorek et al. 2012) would need to be converted from a metric grid to a geographic grid before publication, with subsequent loss of precision. However, as noted in Chapman and Wieczorek (2020),

Where data is generalized to a geographic or biogeographic region (a polygon), the data has less usability for many analyses, but was seen by many as a more secure way of ‘hiding’ sensitive data locations. Currently, GBIF.org has only limited ability to incorporate polygon data. There are some parallels with this method with the reporting of census results in many countries where summaries are reported using Statistical Local Areas or census tracts to restrict possible identification of individuals (Australian Bureau of Statistics 2006). A key difference is that with census data results are summarized over many individuals within a region, whereas with biological data we want to hide the location of a single entity within an area. It does de facto produce a summary, but this is not the primary intent. One problem with this method is that there is no guarantee that political (or even biogeographic) boundaries will remain constant over time and this further reduces the value of the data for many purposes. This has already been found to be a problem when comparing census data over time where census districts or tracts have altered. (Noble et al. 2011).

If georeferences are given for data that is generalized to a biogeographic or political region, the result can be quite misleading – a coastal species, for example, may end up with a georeference that is hundreds of kilometres inland, reducing usefulness for analysis or data cleaning. Making such data available without suitable documentation can lead to quite disastrous results for users. It is often better in some cases to not supply a georeference, but if one is supplied, then documentation should be clear with either a large Uncertainty Radius, or well documented Spatial Fit (see discussion under Spatial fit below).

Good practice dictates that whatever you do to generalize the data that you document it so that users of the data know what reliance they can place in it. Note, that when documenting what has been done, it is essential that both the coordinates and the coordinate uncertainty in meters be recorded. How this is best done can be seen in the Georeferencing Quick Reference Guide (Zermoglio et al. 2020).

I recommend that data providers who are generalizing their data do so using a standard methodology (see below), and to document this accordingly. As most biodiversity data is currently made available using decimal degrees (Chapman and Wieczorek 2020), the recommended method means that protocols such as Darwin Core (Wieczorek et al. 2012) do not need modification, other than to allow for suitable metadata documentation.

The method recommended below allows for several levels of generalization that conform to Categories 1–4 described in Table 1.

The recommended method for generalization is:

The South African National Biodiversity Institute, in largely implementing the criteria as laid out in the previous document (Chapman and Grafton 2008), have implemented only two categories (SANBI 2010, SANBI 2016): the original, non-generalized data and data generalized to one-quarter of a degree (0.25 degree (QDS)). We believe that this is a more difficult generalization to implement, and, unless fully and clearly documented, it could lead to a misleading level of precision. It also reduces the flexibility provided by the above four-categorization method.

It is important to document the method and level of generalization so that users are aware of what has been done to the data, and what reliability they may be able to place in the data. Currently, neither Darwin Core nor the ABCD protocols provide fields for the recommended metadata. It has been proposed that these protocols be modified to accept such metadata (see Afterword), but in the meantime, it is recommended that the information be recorded using existing Darwin Core terms at the record-level (e.g., dwc:informationWithheld, dwc:dataGeneralizations or any of the 'Remarks' fields).

As far as the generalization of georeferencing data is concerned it is important to record that the data has been generalized using a ‘decimal geographic grid’ and record both:

The recommendations for metadata for inclusion in the Darwin Core Location Class (TDWG 2018) are set out in the Afterword. Once they (or similar) have been adopted, then it is recommended that the appropriate fields be recorded and distributed with the data.

With plants, especially, and with other taxa (like insects), collectors often gather multiple specimens (duplicates or parts of sets)—usually on the order of four to six, though examples of more than 80 have been cited (Paul Morris 2007, personal communication, April)—with these duplicates or parts of sets often sent to many institutions around the world. One problem that arises is originating institutions may lose control of what happens to the information (including locality information) distributed to collections from those secondary institutions – remembering that the duplicates may have been distributed prior to the taxon being identified as sensitive.

In most cases this exchange of information is not a problem, but with sensitive taxa, it often is. The secondary institution may not know what are regarded as ‘sensitive taxa’ in the jurisdiction of the originating institution or may not have flagged that information. Sensitivity is not always information that can be distributed along with the collections, as it may not be known until much later that the species is endangered and/or sensitive. This issue is a difficult one, as simply labelling a taxon as sensitive may not be the answer: a taxon may be endangered in its native area (and thus sensitive) and may be a weed or pest in other areas, with locality information important for its control in both instances.

Identifying duplicates across institutions is not easy, as, especially for historic and legacy collections, it is often difficult to determine duplicate specimens. Some institutions, such as Centro de Referência em Informação Ambiental (CRIA) in Brazil in its speciesLink project and the Atlas of Living Australia, use matching across a number of fields such as collector number, date and locality, while GBIF is developing an algorithm for data-clustering. Currently, however, there is no universal global system available. The use of unique, persistent and resolvable Globally Unique Identifiers (GUIDs) (Page 2009, Richards 2010, Richards et al. 2011) will aid these processes in the longer-term, but the implementation of specimen-level GUIDs still seems some way off. A recent paper by Nelson et al. 2018 makes a number of recommendations on minting, managing and sharing GUIDs for herbarium specimens, but until such techniques are more widely adopted, identifying duplicates across institutions will remain an issue.

Metadata has generally been used to refer to documentation of a whole dataset. Documentation at the record level has usually been referred to just in comments. I prefer, however, to term this ‘record-level metadata’, and to formalize the process. In the previous chapter a recommendation was made that where data is generalized for distribution, to document the level of generalization – for example, that the data had been generalized using a decimal geographic grid, and to record both the precision of the data provided and the precision of the data ‘as-held’ or stored. Also, in the chapter on Determining sensitivity, a series of documentation processes were recommended (Table 1). Some of these may be more appropriate for documenting the reasons for regarding a taxon as a potential environmentally sensitivity taxon (Criteria 1 and 2), while the others (Criteria 3 and 4) are appropriate to the data themselves and belong as part of the broader record-level metadata. To fully document the reasons for restricting data, however, it may be necessary to inherit the documentation from Criteria 1 and 2 to the record level – for example, the reason that data is restricted may include that the taxon is subject to harmful human activity.

At the moment, neither the Darwin Core nor the ABCD standards have fields for recording the type of record-level metadata that is recommended here. A number of recommendations have been made to Biodiversity Information Standards (TDWG) for the inclusion of extra fields to the Darwin Core Location Class (TDWG 2018) and are listed in the Afterword.

Until such time as these standards and protocols are modified, it is recommended that the data be documented using existing Darwin Core terms at the record-level (e.g., dwc:informationWithheld, dwc:dataGeneralizations or any of the 'Remarks' fields), and, as far as possible, to record the same type of information that would be included in the recommended fields, for example, that:

When noting access constraints in the metadata at the dataset level, you may include something like:

Spatial Fit (Chapman and Wieczorek 2006, 2020) is a concept that has arisen out of the BioGeomancer project (Guralnick et al. 2006) and provides a measure of how well a geometric representation matches the original spatial representation. Spatial fit is a value of either zero, one or greater than 1, where 1 represents an exact match (i.e. the data has not been generalized). Details on how Spatial Fit may be calculated can be found in Chapman and Wieczorek (2006, 2020), where the summary below can also be found:

With respect to generalization of data, the Spatial Fit can be seen as a Generalization Coefficient – i.e. to what extent a record has been generalized – a ratio of the generalized area to the true area.

An example of its applicability is where a georeference with an uncertainty radius of 1 km (using a point radius method (Wieczorek et al. 2004)) is made available using a 10 km² grid (which completely covers the uncertainty). In this case the Spatial Fit would be greater than 1 as it represents an area greater than the real uncertainty (as shown in Figure 2).

The smaller the grid size, the closer the Spatial Fit will be to ‘1’. Note that a record that has its georeference randomized or generalized such that a portion of the uncertainty radius falls outside the grid square would have a Spatial Fit equal to zero.

Data is already distributed around the globe through duplicate specimens, etc., and although data may be restricted from some institutions, others holding duplicates may be releasing the same information. This may be through ignorance of what may be regarded as sensitive in the home ranges of the taxon concerned as no universal list of what is regarded as ‘sensitive’ is currently available. Difficulties are compounded by the fact that a taxon may be sensitive in one area, but not in another (and indeed may even be an invasive weed or pest species in the second location). If identical data is publicly available through other sources, it cannot be considered sensitive (Australian Government 2016).

For these reasons, it has been recommended that a trigger list of potential environmentally sensitive taxa should be created and linked through GBIF’s Backbone Taxonomy. This would have the advantages of alerting data providers in other jurisdictions that a species is potentially sensitive, and via the Backbone Taxonomy would provide links to synonyms. It is important to note that the list should be regarded as a trigger to flag the need for a decision on the actual sensitivity of sharing information using the criteria in the previous chapter, and not for generating blanket restrictions. Not all endangered species are threatened through knowledge of their locations, or across the totality of their range, and so should not be regarded as sensitive per se and thus the list of potential environmentally sensitive taxa should be much smaller than any existing list of rare and threatened species.

The list should be created using Criteria 1 and 2 and scenarios in Annex 1 and include additional information, such as:

Jurisdictions may also wish to maintain a similar list for their own purposes, and it is recommended that if they do so, they include the above information in all cases. The advantages of making the information more broadly available is that it will alert other data custodians that your jurisdiction regards the taxon as potentially sensitive, and alert users that they should take the sensitivity into account when publishing the results of their analyses, etc.

A number of recommendations have been made to Biodiversity Information Standards (TDWG) for the inclusion of extra fields to the Darwin Core Location Class (TDWG 2018). The recommendations included: