learn

Factsheet: Can I Trust This Data?

Cawthron Institute have played a key role in LAWA by assessing how data is collected and analysed against best practice standards. How each site measures up to standards is explained in the Can I trust this data? link for the River Quality and Lake topics.

About Cawthron

Cawthron Institute  is New Zealand's largest independent science organisation, offering a broad spectrum of services to help protect the environment and support sustainable development.

Based in Nelson, Cawthron works with regional councils, government departments, major industries, private companies, and other research organisations throughout New Zealand and around the world. 

Cawthron is a diverse organisation employing almost 200 scientists, laboratory technicians, researchers and specialist staff from more than 20 different countries.

Cawthron’s scientists have unique expertise in aquaculture research, marine and freshwater resource management, food safety and quality, algal technologies, biosecurity and analytical testing. Their ground-breaking science is supported by substantial testing and research laboratories, state-of-the-art technology and a purpose-built aquaculture park.

They also run an extensive community and education programme to help foster the next generation of scientists.

Since establishment more than 90 years ago, Cawthron has strived to support New Zealand through applied science that works for the economy and environment.

What is Cawthron’s role in LAWA?

Cawthron’s role in LAWA is diverse. Primarily, their scientists make sure that you can trust the data presented on LAWA.  As an independent scientific institute, they also help to assure that the data is presented without any bias. Cawthron has worked alongside regional councils to verify the processes and methods used for data collection, laboratory analysis of samples collected and the statistical analysis and interpretation of the results presented.

If you want to find out how robust the information for a particular site is, simply click on the Can I trust this data? link and you will be able to read about the level of confidence you should have in the data. We say that the data received the 'Cawthron tick' and you can find ticks in the River Quality and Lakes topics on LAWA.  

The colour of the tick describes how robust the information is. For example, some sites will have a tick that is green (high quality data and analysis appropriate) and some sites will have an orange tick (conclusions should be made with caution). All information shown on LAWA has either a green or an orange tick - any data that might have received a red tick is not presented on LAWA.

 

Cawthron tick green High quality data and analysis appropriate
Cawthron tick orange Conclusions should be made with caution

 

 

RIVER QUALITY

Scientific Indicator Tick

This Cawthron tick assesses DATA COLLECTION, STATE & TREND ANALYSIS for water quality indicators.

DATA COLLECTION

Data collection needs to be done thoroughly and using appropriate protocols to obtain robust, standardised data. The Cawthron Institute assesses:

  • Whether council staff are adequately trained to collect water samples
    • All councils have internal field sampling protocols in place which outline field sampling programmes relevant to each council.  These programmes ensure that:

* All water quality parameters recommended in national guidelines are sampled

* Suitable sampling practice is followed (i.e., sampling occurs at the same site, ideally mid-stream)

* Appropriate sampling frequency (i.e., monthly, at approximately the same time of day) and sampling conditions (i.e., during all flows)

* Adequate sampling equipment and its regular calibration (i.e., prior to each sampling run for pH and dissolved oxygen probes, monthly for temperature probes)

* Adequate recording of sample results and site location (i.e., recording of metadata on field sheets, photographs)

* Adequate storage (i.e., chilled storage bins) and transport of samples (i.e., within 24 hours) to laboratories

  • All councils use International Accreditation New Zealand (IANZ) laboratories (www.ianz.govt.nz) which use approved methodologies and detection limits. There may be minor differences in laboratory analysis methods between different laboratories. Regional Councils and unitary authorities are currently working towards standardising laboratory analysis techniques based on national recommendations. This will increase the robustness of the freshwater quality data shown on LAWA.
  • Adequate field data entry processes are in place for each council before the data are sent to LAWA’s data processing team.

 STATE & TREND ANALYSIS

LAWA displays the current regional and national water quality (=state) of our rivers and streams to give you an idea about the effects of different pressures (i.e., urbanisation, farming, etc.) on the condition of our freshwater resource right now. Find out more here on how the state of water quality is calculated.

To see if water quality has improved or deteriorated at a site, LAWA also displays trends for the last five and ten years. The ability to detect trends is based on the length and frequency of the sampling record. For example, occasional samples collected over a short period can provide limited evidence that a meaningful change is occurring at a site, over and above expected daily/seasonal variability. Longer more frequent sampling records provide much better evidence. Monthly sampling is recommended for robust water quality trend analysis. The Cawthron tick assesses sampling frequency for each site and gives the appropriate tick colour. If you want to find out more about how trends are calculated have a look at our Trend factsheet.

Cawthron also assesses the method used for trend analyses. Ideally, a ‘flow adjustment’ process during trend analysis is used to compensate for differences in flow on the sampling days across the data record. For example, if water quality sampling happens during high flow on one day and during low flow on another day at a particular site, differences in the data are caused by different flows rather than consistent changes at the site. Unfortunately continuous flow data is not available for the majority of water quality monitoring sites.  Therefore, trend analyses reported on LAWA are not flow adjusted which is why Cawthron gives trend analyses an orange tick.

Macroinvertebrate Tick

The 2 overarching areas the Cawthron tick looks at for the macroinvertebrate tick are SAMPLE COLLECTION & PROCESSING and QUALITY CONTROL.

SAMPLE COLLECTION & PROCESSING

For the ‘Sample Collection Tick’, the Cawthron Institute assesses sampling frequency, sampling protocols, sample habitat, and stand-down periods after floods.

As macroinvertebrate density and abundance depend on the season (with higher abundance during spring and summer), sampling frequency and timing is very important. Because of this, councils sample sites regularly at the same time(s) and under the same flow conditions each year.  See LAWA’s macroinvertebrate factsheet which explains the importance of timing in more detail.

Samples need to be collected following certain sampling protocols to assure that all councils collect their data in the same way.  There are three accepted and widely used sampling protocols and you can find out more here. Although the Cawthron Tick is based on national recommended guidelines (i.e., NEMaR), it acknowledges the existing variations in sampling protocols in its tick colours. This means, for example, that councils which follow NEMaR recommendations score the same colour tick as councils that follow other accepted protocols (e.g., C1 as recommended by Stark et al. 2001). Councils are currently in the process of developing National Standards for macroinvertebrate sampling protocols.

Similar to sampling frequency and sampling protocols, all councils should use the same sampling habitat (i.e. where in the river is the sample taken – in riffles, runs or pools, or all three). Our macroinvertebrate factsheet explains this aspect in more detail.

Macroinvertebrates are very small, which means they are prone to being washed downstream and out of the river system during floods. Therefore, councils need to have a “stand-down” period for at least 2 weeks before they go back to a river for sampling following a flood greater than 3 times the median flow. This gives the animals a chance to recover and recolonise their habitats. Macroinvertebrate abundance and diversity data may be compromised if samples are collected within the first two weeks after a flood.

For the “Sample Processing Tick”, the Cawthron Institute assessed whether councils are following current best practice. Once macroinvertebrates have been collected, they get transported to a laboratory for sample processing. There are three accepted protocols which are widely used by councils to process their samples (i.e., P1, P2, P3 – Stark et al. 2001). See here for more information on these sampling protocols. Proposed national guidelines and leading literature are inconsistent in their recommendations about which protocol to use and councils are currently in the process of developing National Standards for macroinvertebrate sample processing.

QUALITY CONTROL

To really be sure that the data collected and processed by councils is of high quality, quality control measures need to be applied in the field and the laboratory. Similar to sample collection and sample processing, three widely used and accepted quality control protocols are available (QC1, QC2, QC3). Proposed national guidelines and leading literature are, however, inconsistent in their recommendations about which quality control protocol to use and councils are currently in the process of developing National Standards for quality controlling macroinvertebrate data.

 

MACROINVERTEBRATES - Comprehensive Cawthron Tick text

DATA COLLECTION 

  • Macroinvertebrates are sampled at this site as part of council's stream health monitoring programme. Proposed national guidelines recommend the sampling of benthic macroinvertebrates as an ecological stream health indicator. Macroinvertebrate data along with physical-chemical water quality data provide a comprehensive picture of stream health at a certain site.
  • This council does not monitor macroinvertebrates as part of their stream health monitoring. Proposed national guidelines suggest that benthic macroinvertebrate sampling should be included in stream health assessments to obtain a more comprehensive picture of stream health at this site.

SAMPLING - FREQUENCY

  • Macroinvertebrate sampling is done annually or more frequently at this site. Proposed national guidelines suggest that macroinvertebrates should be sampled at least once a year in late summer (i.e., January - March) to allow robust long-term trend analysis. Therefore, data shown here follow current best practice guidelines.

SAMPLING - PROTOCOLS

  • This site is a hard-bottomed site and appropriate sampling protocols have been applied. Proposed national guidelines (i.e., NEMaR) suggest quantitative protocol C3 for hard-bottomed streams should be used. Although the Cawthron Tick is based on national recommended guidelines (i.e., NEMaR), it acknowledges the existing inconsistencies in sampling protocols in its tick colours. This means, for example, that councils which follow NEMaR recommendations score the same colour tick as councils that follow other accepted protocols (e.g., C1 as recommended by Stark et al. 2001). Data shown here has been collected using current best-practice.
  • This site is a hard-bottomed site and sampling protocols follow current best practice. However, there have been inconsistencies in sampling protocols used during the past at this site (i.e., change from C1 to C3 in 2010).  Proposed national guidelines and leading literature are inconsistent in their recommendations about which protocol to use, therefore both, C1 and C3 are considered suitable. When analysing long-term data trends, however, this change in sampling methodology will need to be considered.
  • This site is a soft-bottomed site (i.e., >50% of the stream bed is naturally composed of silt, sand or pumice) and appropriate sampling protocols are being used. Proposed national guidelines suggest semi-quantitative protocol C2 for soft-bottomed streams should be followed. Therefore, data shown here has been collected following best-practice. 
  • Sample area at this site is defined by a timed sample of woody debris, macrophytes and overhanging vegetation. Proposed national guidelines suggest a fixed sample area of 0.3 m2 for soft-bottom streams as specified in C2. Sample area used at this site is therefore not following best practice and conclusions from this data should be treated with caution.
  • This site is a soft-bottomed site (i.e., >50% of the stream bed is naturally composed of silt, sand or pumice) and sampling protocols have been changed in 2010 from C2 to C4 (Stark et al. 2001). Proposed national guidelines and leading literature are inconsistent in their recommendations about which protocol to use, therefore both, C2 and C4 are considered suitable. When analysing long-term data trends, however, this change in sampling methodology will need to be considered.

SAMPLING - HABITAT

  • Macroinvertebrate sampling is done in riffle habitat at this site. Proposed national guidelines suggest riffle habitat as the most suitable sampling habitat to obtain comprehensive and high quality bio-monitoring results. Data shown here are therefore following best practice.
  • Macroinvertebrate sampling is done in all meso-habitats (i.e., pool, run and riffle), rather than just in riffle habitat, as current guidelines suggest. All-habitat sampling provides a more comprehensive description than riffle-only sampling of the invertebrate community at a stream site. However, it can be more difficult to compare ecosystem health among sites using all-habitat data, as variation arises from the range of habitats present. Thus the choice of method depends on the primary objective of the monitoring. Comparisons made between macroinvertebrate data collected in all habitat and data collected in just riffle habitat need to be treated with caution.

SAMPLING - STAND DOWN PERIOD

  • This council does not collect any macroinvertebrate samples for up to two weeks after a flood greater than three times the median flow. Macroinvertebrates are prone to being washed out of a river system during a large flooding event which is why 'stand-down' periods are necessary before the next sample collection. Recommended national guidelines suggest a minimum of two weeks 'stand-down' period after a flood greater than three times the median flow. Therefore, data collected at this site is following best practice.
  • Macroinvertebrate samples are collected within the first two weeks after a flood greater than three times the median flow at this site. Macroinvertebrates are prone to being washed out of a river system during a large flooding event which is why 'stand-down' periods are necessary before the next sample collection. Proposed national guidelines suggest a minimum of two weeks 'stand-down' period after a flood greater than three times the median flow. Therefore, data shown here is not following best practice and conclusions based on this data need to be treated with caution.

PROCESSING PROTOCOL

  • Samples at this site have been processed following sampling protocols P2 or P3. P2 is a 200-animal fixed count method and is a widely used sub-sampling technique both in NZ and overseas. Protocol P3 is a full count, quantitative method, which is appropriate for intensive impact assessments, especially where an anticipated impact is a change in macroinvertebrate densities. Proposed national guidelines suggest that all samples should be processed following the quantitative P2 protocol (200-count with scan for rare taxa) or where required Protocol P3. Data shown here have been processed following best practice.
  • Samples at this site have been processed following sampling protocol P1 (semi-quantitative - coded abundance). Proposed national guidelines suggest that all samples should be processed following at least the quantitative P2 protocol (200-count with scan for rare taxa). Data collected with P1 are, however, high quality data and can be converted back into semi-quantitative categories for time-trend analysis. Data shown here is not following best practice, but are still robust.
  • Samples at this site have been processed following protocol P2 which is recommended by proposed national guidelines. However, the individual count number has been reduced from 200 to 100 (with a scan for rare taxa). Proposed national guidelines recommend a 200-count (with a scan for rare taxa) over the slightly more rapid 100-count because of the information gains described in Stark et al. (2001). Data processed at this site are therefore not following best practice and conclusions based on this data need to be treated with caution.

QUALITY CONTROL - FIELD

  • Data collected at this site had some form of field quality control done. This may include re-sampling of field samples (10% of samples to be repeated by a second staff member - as recommended by proposed national guidelines), senior staff sampling supervision and/or regular training of field staff at the beginning of the sampling season. New Zealand councils are currently reviewing their quality assurance and control procedures to update current guidelines. In the meantime, data collected here are more robust than data with no field quality control.
  • Field data collected at this site had no quality control applied to it.  Proposed national guidelines suggest 10% of samples to be repeated by a second staff member. New Zealand councils are currently reviewing their quality assurance and control procedures to update current guidelines. In the meantime, data shown here are not following best practice and conclusions based on this data need to be treated with caution.

QUALITY CONTROL – LABORATORY

  • Data processed at this site has had one of the three Quality Control Protocols (i.e., QC1, QC2, QC3) applied. Proposed national guidelines suggest 'Quality Control Protocol QC2' to be followed (i.e., 10% of sorted samples including residue re-examined by external expert), however, laboratory QC amongst councils is inconsistent with different councils using different QC protocols. Councils are therefore currently reviewing laboratory QC procedures to update current guidelines and develop national processing protocols. Data shown here is more robust than data with no laboratory QC applied.
  • Data processed at this site has had no laboratory quality control applied. Proposed national guidelines suggest 'Quality Control Protocol QC2' to be followed (i.e., 10% of sorted samples including residue re-examined by external expert). New Zealand councils are currently reviewing laboratory QC procedures to update current guidelines and develop national processing protocols. Data shown here is less robust than data with no laboratory QC applied and conclusions based on this data need to be treated with caution.

 

LAKES

The three main areas the Cawthron Institute assesses relating to lake water quality monitoring are SAMPLE COLLECTION & PROCESSING, QUALITY ASSURANCE and TREND ANALYSIS.

SAMPLE COLLECTION & PROCESSING

National guidelines for the collection and processing of lake water quality data have not been developed for New Zealand yet. This means, that there is some variability in the way councils collect and process their data. Until national standards for sample collection and processing have been developed, there is no single agreed way to collect and process lake water quality data.

Although the Cawthron Tick is based on national recommended guidelines (i.e.,  NEMaR ), it acknowledges the existing variations in sampling protocols in its tick colours. This means, for example, that councils which follow NEMaR recommendations score the same colour tick as councils that follow other accepted protocols (e.g., Burns et al 2000).

QUALITY ASSURANCE

How do we know if the data LAWA shows can be trusted or not? One part of our review looks at processes behind the data collection and data analysis, which we refer to as Quality Assurance or QA. There are many different sub-categories that fall under QA, such as 1) data processing and 2) quality coding. For some of these sub-categories, national standards already exist (see here for NEMS Quality Coding Standards). Once national standards are developed, data will be collected and processed in the same way so all data throughout New Zealand will be comparable. However, there are no National Standards, yet for the ‘Quality Assurance’ aspect.

Regional Councils and the Ministry for the Environment are currently assessing the range of Quality Assurance and Quality Management Systems that are in place. Once this information is collected, a National Standard for Quality Assurance will be developed to provide confidence that national datasets can be compared.

1)    Data Processing

All councils should have robust data processing measures in place. National guidelines for data processing are currently being developed. Until then, the Cawthron Tick acknowledges councils that have some form of data processing measures set up. These include specific data entry procedures and protocols, high levels of training for staff involved with data entry and appropriate archiving of processed data.

2)    Quality coding

The quality coding of data is important to determine how much faith should be placed on a particular data point. National Environmental Monitoring Standards (NEMS) recommend that all environmental data should be quality coded following the NEMS protocol to allow for national comparison of environmental data. Some councils already have some quality coding procedures in place, but others are in the process of reviewing their existing procedures or introducing new quality coding procedures following NEMS recommendations. The Cawthron Tick for Quality Coding will be reviewed once all councils have adopted NEMS Quality Coding protocols.

TREND ANALYSIS

LAWA displays any regional trends for the last five and ten years. The ability to detect trends is based on the length and frequency of the sampling record. For example, occasional samples collected over a short period can provide limited evidence that a meaningful change is occurring at a site, over and above expected daily/seasonal variability. Longer more frequent sampling records provide much better evidence. Monthly sampling is recommended for robust water quality trend analysis. The Cawthron tick assesses sampling frequency for each site and gives the appropriate tick colour. If you want to find out more about how trends are calculated have a look at our Trend factsheet.

 

LAKE - Comprehensive Cawthron Tick text

DATA COLLECTION 

  • Lake water quality is being monitored in this region. Proposed National Guidelines suggest that all councils should monitor lake water quality as part of their State of the Environment Monitoring Programmes, if lakes occur in the region. There are different guidelines and processes for collecting and analysing lake water quality data. To find out whether the data shown for this site has been collected and analysed following best practice, look at the Cawthron Tick colour.
  • This site is only sampled during summer months to determine if it's safe for you to swim in. This data is collected by trained council staff and analysed by accredited laboratories. There is no data on ecological health available at this site.
  • Lake water quality is not monitored in this region. Proposed National Guidelines suggest that all councils should monitor lake water quality as part of their State of the Environment Monitoring Programme, if lakes occur in the region.

STATE – SAMPLING

  • All samples were collected using approved field protocols and have been analysed in accredited laboratories.
  • This site is sampled by helicopter, which means that only a subset of monitoring parameters can be measured. For example, water clarity measured by Secchi disc cannot be measured by helicopter, but is a key water quality variable and ideally should be included in routine water quality sampling. Data shown here is therefore less comprehensive than what would be collected by boat. This needs to be considered when making conclusions based on the data from this site.

STATE – QUALITY ASSURANCE - DATA PROCESSING

  • Data at this site has had some form of data processing measures applied. These could include specific data entry procedures and protocols, high level of staff training and/or appropriate archiving of processed data. Therefore, the data shown here are more robust than data with data processing measures applied.
  • Data at this site has not had any form of data processing measures applied. In particular, there are no specific data entry protocols after data collection. Proposed national guidelines suggest that data processing is necessary to produce trustworthy high quality data. Therefore, the data shown here is less robust and needs to be treated with caution.

STATE – QUALITY ASSURANCE – QUALITY CODING

  • Data shown for this site has had quality coding applied to it, following the NEMS recommendations. The data shown here is therefore robust and can be trusted. 
  • Data shown for this site has had some form of quality coding applied. National Environmental Monitoring Standards (NEMS) recommend that all environmental data should be quality coded following the NEMS protocol to allow for national comparison of environmental data. Therefore the data shown here is more robust than data without any quality coding measures applied.
  • Data shown here hasn’t had any kind of quality coding applied. National Environmental Monitoring Standards (NEMS) recommend that all environmental data should be quality coded following the NEMS protocol to allow for national comparison of environmental data.  Therefore, the data shown here is less robust than quality coded data. Councils are currently in the process of reviewing their existing or introducing new quality coding procedures following NEMS.

TREND – SAMPLING FREQUENCY

  • Samples are collected monthly at this site. Proposed national guidelines suggest monthly sampling is required for robust trend analysis. Therefore, any trends shown here are based on best practice in terms of sampling frequency.
  • ONLY FOR TAUPO - Samples are collected three-weekly at this site.  Proposed national guidelines suggest monthly sampling is required for robust trend analysis, but three-weekly sampling will provide equally robust data for trend analysis.
  • Samples are collected monthly (and/or) bimonthly (and/or) quarterly at this site. Proposed national guidelines suggest monthly sampling is required for robust trend analysis. Therefore, any trends shown here are less robust than those calculated from data that is solely collected monthly. New Zealand's councils are currently reviewing their sampling procedures and aim to adjust their sampling frequency to monthly. This will increase the robustness of the trend analyses and data shown here.
  • Samples are collected monthly at this site since 2013. However, sampling frequency used to be bimonthly at this site. Proposed national guidelines suggest monthly sampling is required for robust trend analysis. Therefore, any trends shown here are less robust than those calculated from data collected monthly. New Zealand's councils are currently reviewing their sampling procedures and aim to adjust their sampling frequency to monthly. This will increase the robustness of the trend analyses and the data shown here.