previous episode

OTN Node Manager Training

next episode

Near Real Time (NRT) Animal Movement Location Data Quality Control (QC) Process

Overview

Teaching: 120 min
Exercises: 0 min
Questions

  • What are NRT data and NRT data sources/vendors?

  • Why NRT data needs QC?

  • What is ArgosQC package?

  • ArgosQC’s workflow/features

  • Satellite vendors: SMRU and WildlifeComputers

  • How to configure ArgosQC (aniMotum) model parameters for various species?

  • What are the mandatory import files for ArgosQC?

  • What are the output files and how to interpreate ArgosQC results and diagnoses?

  • What are the signs, and how do you fine-tune model parameters?

Objectives

below topics can be on a separate lesson.

objectives:

keypoints:

ArgosQC for near-real-time data is an essential automated process that uses state-space models (calling the underneath aniMotum R package) to filter noisy Argos satellite location data from vendors like SMRU and Wildlife Computers. Its effectiveness relies on configuring species-specific movement parameters and interpreting diagnostic outputs to produce reliable animal movement tracks for ecological research.

NRT data and NRT data sources

  1. Near Real-Time data are transmitted by satellite-linked electronic tags, when animals are at the ocean surface, via the Argos satellite constellation.

  2. Currently, the ArgosQC R package can access & download NRT data from two animal tag manufacturers - SMRU (Sea Mammal Research Unit, St Andrews, UK) and Wildlife Computers. Typically, NRT data are downloaded & QC’d once every 24 hours until tag deployments have ended (e.g., due to tag battery failure, or animal recapture). SMRU tag data are made available on a server with a Web Application Firewall, which requires a user ID and password (provided to the tag owner) to access the tag data files (stored in a .mdb file). Once a node manager has access to a tag owner’s user ID and password, SMRU tag .mdb files can be downloaded via ArgosQC. Wildlife Computers tag data are accessed via a Data Portal, which requires both a user account (with user ID and password) to access the Portal AND explicit consent by tag owner(s) to share their tag data (set up by the tag owner on the Data Portal). Details on accessing tag data via the Wildlife Computers Portal are here. Once a user account is set up by the node manager and explicit data sharing is set up by the tag owner, data can be downloaded by ArgosQC via the Wildlife Computers API.

Quality Control for NRT data

  1. With the Argos satellite system, tag location is measured by tag transmissions received by polar-orbiting Argos-Kinéis satellites as they pass overhead, and relayed to a base in France. The Doppler shift in tag transmission frequency is used to triangulate position of the tag. These calculations are conducted in real-time by the French organization Collecte Localisation Satellites (CLS). This positioning technology is less precise than GPS and requires a statistical quality control process (provided by the ArgosQC R package) to obtain more reliable locations and estimates of their uncertainty.

  2. At a minimum, satellite tags transmit their location but, depending on their programming and on-board sensor capabilities, may also transmit summaries of behavioural data such as dive profiles or diving and surfacing activity summaries, and physical observations of water temperature, salinity and/or fluorimetry at depth (CTD/FTD profiles) as animals dive through the water column. Tag owners can obtain records of their tag(s) locations through time from CLS, but CLS also provides the location data and all tag transmission messages to the tag manufacturers in near real-time. The tag manufacturers decompress and organize these messages (typically) into distinct tag data files (e.g., one file per sensor data stream or behavioural activity) and make them available to the tag owners via an API, data portal, or secure Web-Accessible Folder (WAF).

  3. Typically, the behavioural and physical observations data files either have crudely interpolated locations or no locations associated with each record. The ArgosQC R package uses a statistically robust interpolation to append a location and its uncertainty to each record, based on their observation datetime, in these data files. This provides more accurate locations for each tag-transmitted observation and eliminates the need for subsequent users of the data to geolocate every tag-transmitted observation.

ArgosQC workflow and features

  1. ArgosQC workflows are intended to be run automatically via a scheduler and require minimal supervision. ArgosQC provides two workflows, one for SMRU tag data and one for Wildlife Computers tag data. Both workflows require the node manager to set up a JSON configuration file that specifies all required project information and QC parameters. In general, both workflows do the following:
    • Downloads specified tag data from the tag manufacturer
    • Acquires any available deployment metadata from the tag manufacturer & builds an operational deployment metadata file, or ingests a specified metadata file (CSV).
    • Prepares tag location data for state-space model (SSM) fitting
    • Fits the SSM in 2 passes to each tag location dataset. SSM fitting to multiple tag datasets is conducted in parallel across a number of available processors.
    • Reroutes any SSM-estimated locations that occur on land back into the ocean.
    • Interpolates & appends SSM locations to each record in each tag data file.
    • Generates diagnostic plots of the SSM fits to tag location data & a map of the SSM estimated tracks.
    • Combines all QC-annotated tag data files across individual tags & writes these aggregated files, plus SSM summary output & annotated deployment metadata, to CSV files as the final QC output.

The workflows process multiple tag datasets at a time, but must be limited to data from a single data owner/researcher and to data from a single species.

More specific details on the workflows are provided in the ArgosQC vignettes: SMRU and Wildlife Computers.

Configuring ArgosQC

The entire QC process is automated, but requires a number of initial, manual steps to gain access to the tagging data and deployment metadata, and to set up a configuration file. A JSON configuration file provides all required information to fully specify an ArgosQC workflow. The config files are slightly different for SMRU vs Wildlife Computers tags, but both have the same 4-block structure, within which different QC parameters are specified:

The setup block specifies the National Observing Program (e.g., atn, otn, imos) overseeing data assembly & the directory paths for downloading or accessing tag data files, accessing metadata & output directories. The harvest block specifies data harvesting parameters such as user access to the tag manufacturer’s data portal. The model block specifies model- and data-specific parameters required for SSM fitting. The meta block specifies species and deployment location information, but is only required when no metadata CSV file is specified in the setup block. The ArgosQC vignettes provide details on the block parameters specific to SMRU and Wildlife Computers config files.

The JSON file for NRT QC of Wildlife Computers (WC) tags looks like this:

{"setup": 
	{
	"program":"otn",
	"data.dir":"grse/data",
	"meta.file":null,
	"maps.dir":"grse/maps",
	"diag.dir":"grse/diag",
	"output.dir":"grse/output",
	"return.R":false
	},
"harvest": 
	{
	"download":true,
	"owner.id":"663d756e3cd05b1b7a0ff568",
	"wc.akey":"[Wildlife Computers akey]",
	"wc.skey":"[Wildlife Computers skey]",
	"tag.list":"grse_tags.csv",
	"dropIDs":null
	},
"model": 
	{
	"model":"rw",
	"vmax":3,
	"time.step":6,
	"proj":null,
	"reroute":true,
	"dist":20,
	"barrier":null,
	"buffer":0.25,
	"centroids":true,
	"cut":false,
	"min.gap":72,
	"QCmode":"nrt",
	"pred.int":6
	},
"meta":
	{
	"common_name":"grey seal",
	"species":"Halichoerus grypus",
	"release_site":"Sable Island",
	"state_country":"Canada"
	}
}]

In the setup block the parameters are:

In the harvest block the parameters are:

In the model block the parameters are:

In the meta block the parameters are:

The meta block is only required when no metadata file is supplied in the setup block. In this case, ArgosQC obtains available tag deployment metadata from the tag manufacturer, restructures it, and appends the attributes listed in the meta block.

How to set up an ArgosQC config file for a NRT QC workflow

Setting up a config file for the first time requires a number of steps. Config files for Wildlife Computers tags and SMRU tags are similar but require slightly different information in their harvest blocks. Tags deployed across multiple species, even if associated with the same data owner, must be split into separate QC workflows (i.e., have different config files). The config file setup steps are documented below and organized by config file block. The harvest block setup differs between Wildlife Computers and SMRU tags, this section has been separate by tag manufacturer.

To setup a new config file, the node manager must first create a new JSON file and then:

setup block - all tags

  1. Update the setup block, ensuring the directory and file paths point to the correct locations. Note, all sub-directories and files must be contained within a single working directory. ArgosQC will create the sub-directories inside of this working directory, if they do not exist.

harvest block - Wildlive Computers tags

  1. Ensure they have access to the WC Portal and its API by first registering for a WC Portal account: {width=400}

  2. Login to the WC Portal account (a) select “Account Settings” (b), select “Web Services Security” (c.1) & add an access and secret key pair (c.2) to securely download data via the WC Portal API: a. {width=400} b. {width=400} c. {width=400}

Steps 1 and 2 only need to be done once.

  1. Ensure the data owner(s) have explicitly set up data sharing with the node manager within the WC Portal. The data owner(s) will need the email address the node manager used for their WC Portal user account. Each time a data owner has new tags registered in the WC Portal, those tags will need to be explicitly shared with the node manager. This document (p 19) provides details on how data owners can set up data sharing within the WC Portal.

  2. Set harvest:download to true if the data are to be downloaded from the WC Portal.

  3. Once data sharing has been setup, the node manager must find the data owner(s) WC Portal ID(s). This can be done within R using the ArgosQC utility function wc_get_collab_ids:

    ArgosQC:::wc_get_collab_ids(a.key = "...", s.key = "...")

    Where the a.key and s.key values are obtained from the WC Portal (step 3c, above). Executing this function with R returns a data.frame of all data owners’ (collaborators) ID’s and email addresses who have set up data sharing with the node manager:

    {width=300}

    Choose the appropriate owner.id and copy it into the ArgosQC config file. Only one owner.id can be used per config file:

    {width=400}

  4. Copy and paste the WC access and secret keys that were generated in step 3 into harvest:wc.akey and harvest:wc.skey, respectively:

    {width=400}

  5. Get the WC dataset UUID’s from the Portal to populate the harvest:tag.list file, using the ArgosQC function wc_get_uuids: 
    ArgosQC:::wc_get_uuids(a.key = "...", s.key = "...", owner.id = "...")

    Where a.key and s.key are the node manager’s keys, and owner.id is a single id obtained from step 6. Executing this function in R returns a data.frame of all the owner’s datasets on the WC Portal; one per record:

    {width=800}

    The key variables in this data.frame are:

    • id - the dataset uuid
    • owner - the owner’s email address (to confirm that the correct data owner is listed)
    • tag - the WC tag serial number

    Other variables listed are not fully parsed into human-readable form.

  6. To conduct a QC workflow on a subset of the listed tag datasets, copy their corresponding ids into a CSV file with a single variable names uuid:

    {width=200}

    Move this CSV file into the QC working directory and copy the file name into harvest:tag.list:

    {width=400}

    This approach is required, for example, when the owner’s tags have been deployed on multiple species. In this case, separate QC workflows need to be run for each species. The WC Portal typically does not contain information on the species tags were deployed on, so the node manager will require some minimum deployment metadata from the tag owner to identify which tags (e.g., by tag serial number &/or deployment date) were deployed on which species. If all the owner’s tags were deployed on a single species in a common geographic locations then they can likely be QC’d in a single workflow. To conduct a QC workflow on all the owner’s tag datasets, harvest:tag.list should be set to null.

harvest block - SMRU tags

First, obtain the deployment metadata from the researcher, along with their SMRU username and password to access their data on the SMRU server. The minimum deployment metadata must include the species name, the SMRU campaign id for the tags being deployed, e.g. ct189, the deployment site name (must be same for all tags in the deployment campaign), and the deployment country.

  1. To download data from the SMRU server, set download to true.
  2. Set the cid to the SMRU campaign ID. If multiple active campaigns ID’s exist for the researcher then these must be set up in separate config files so the QC process runs separately for each.
  3. Set the smru.usr and smru.pwd to the researcher’s SMRU username and password.
  4. Set the timeout to 180 s. This can be increased for slower internet connections to ensure that file download from the SMRU server completes.
  5. Set dropIDs to null initially. If some SMRU tags fail to become active, then they can be removed from the QC process by listing the SMRU Ref ID in a .CSV file. Supply the filename here.

  6. Set the p2mdbtools directory path on your machine so it points to the locally installed mdbtools software (required to access data within the SMRU .mdb files)

    {width=400}

model block - all tags

  1. The model section of the config file provides parameters and information to conduct the QC, including: fitting the state-space model (SSM) to the tag location data; rerouting locations off of land; interpolating SSM locations to the times of each record in the various tag data files. A good starting place for parameter values is provided in the example config file:

    {width=400}

    See the Configuring ArgosQC section (above) for an explanation of the typical parameter values. Generally, the config parameters only need to be set once per species but arriving at reasonable parameter values usually requires at least one QC test run. The node manager should examine the QC-generated map of SSM-estimated locations & diagnostic plots to ensure the QC results are reasonable before operationalizing the workflow. Below are some examples of the maps & diagnostic plots with varying time.step parameter values.

    {width=400} {width=400} {width=400} {width=400}

    Notice how the 12-h time.step results in jagged tracks that lack the movement detail of the tracks QC’d at shorter time.step’s. This is an indication that the 12-h time.step is a bit too coarse given the temporal resolution of the Argos locations. Conversely, the 1-h time.step results in some very obviously straight lines along which the SSM-estimated locations fall. This is an indication that the time.step is too fine relative to the frequency of the data and the SSM is interpolating too much between the Argos locations. If you look carefully, you can also see a hint of these straightlines in the 3-h time.step tracks, suggesting that the 6-h time.step may be the better choice for these data. In practice, either the 3- or the 6-h time.step will work almost equally for the NRT QC of these data. If a lot of tags (e.g., > 20) are being processed in one QC workflow then it will be a bit more efficient to choose the 6-h time.step.

    A check of the model fit diagnostic plots (just latitudes for the 2 grey seal tracks shown below) indicdates the model is fitting the Argos data reasonably well because the SSM-estimated latitudes (red points) generally smooth through the Argos latitudes (blue points). Bad fits would be characterized with numerous red points outside of the blue points (ie. above or below) - this case would suggest a slightly coarser time.step may be needed. Note, these diagnostic model fit plots do usually not change with differing time.step’s.

    A final note on re-routing locations off of land. The example config file has the rerouting parameter model:reroute set to true, but a significant number of the seals’ locations occur on Sable Island because the seals return to land periodically during the tag deployments (see above maps - Sable Island is roughly at the map centre where there is a large concentration of locations). A model run with re-routing turned on results in some re-routing artefacts (denoted by red arrows): {width=600}

    For this reason, model:reroute was set to false and we accept that some small portion of locations may erroneously fall on land. A better solution for this issue would be to generate a custom land polygon shapefile of the Nova Scotia mainland without a polygon for Sable Island. The is custom shapefile could then be added to the config file by providing the filename and path to the model:barrier parameter and setting model:reroute back to true. Other re-routing config parameters - dist, buffer, centroids - need not be changed from the provided defaults.

meta block - all tags

  1. The meta section of the config file is only required if no metadata file is provided in the setup block. When no metadata file is provided, ArgosQC uses the fields in the meta block plus partial deployment metadata downloaded from the WC Portal. ArgosQC assembles these metadata attributes and writes a deployment metadata CSV file as one of the QC outputs. This file, if sufficiently complete, can be used as an input to subsequent QC runs by supplying the CSV file in the setup block. When both a metadata file & completed meta config block are provided, the attributes from the metadata file will supersede those in the meta block.

ArgosQC Key Features

https://github.com/ianjonsen/ArgosQC

  1. Comprehensive Automated Workflow: ArgosQC now provides a fully automated, end-to-end process. It handles everything from accessing and organizing complex multi-file data structures from vendors (SMRU and Wildlife Computers), to fitting state-space models (SSMs) and finally writing out appended, quality-controlled data.

  2. Simplified Execution with New Functions: The process can be run easily with manufacturer-specific functions like smru_qc() and wc_qc(). These functions use a simple configuration file (e.g., test_conf.json) to manage project settings, making the QC process repeatable and easier to manage.

  3. Enhanced Data Integration: A key new feature is the ability to collate tag data with associated deployment metadata. The output is then enriched by appending SSM-estimated locations to every tag-measured event record (like CTD profiles, dives, haulouts, and raw Argos/GPS locations) into a single, comprehensive .csv file.

  4. Refined Data Filtering: Recent updates (February 2026) include a practical fix to remove erroneous locations originating from the manufacturer’s headquarters (e.g., SMRU HQ), a common issue in raw data streams.

  5. Improved Data Handling: Recent commits have focused on robustness, including fixes for Wildlife Computers (WC) UUID parsing issues (January 2026) and the implementation of checks to ensure data integrity.

  6. Expanded Documentation and Vignettes: The package now features detailed vignettes that guide users through the config file structures and specific workflows for both SMRU and Wildlife Computers data, making it easier to get started and customize the process.

Feature Request and Bug Report

As an open-source tool under active development, user feedback is essential for improving ArgosQC. If you encounter problems or have ideas for new features, you are encouraged to contribute through the following channels:

Report Bugs via GitHub Issues: If you find a bug (e.g., a workflow error, data parsing problem, or unexpected crash), please submit a report through the repository’s Issues tab. When reporting, include:

A clear and descriptive title.

A step-by-step description of how to reproduce the issue.

Your session information (output of sessionInfo() in R) and the version of ArgosQC you are using.

Any relevant error messages or log outputs.

Suggest Enhancements: For new feature requests or ideas to improve existing workflows (like support for additional tag vendors or new diagnostic plots), you can also use the Issues tab. Please tag your issue with an “enhancement” label if possible, and clearly explain the proposed feature and its potential use case.

Review Current Priorities: Before submitting, it is helpful to review the Issues page to see if the bug or feature has already been reported or is currently being addressed.

Code Contributions: If you are interested in fixing a bug or adding a feature yourself, please consult the repository’s Contributing guidelines (linked on the main page). The standard practice is to fork the repository, create a branch for your changes, and submit a Pull Request for review by the maintainer.

Key Points

previous episode

next episode