This lesson is still being designed and assembled (Pre-Alpha version)

OTN Field Reporting Standards

Background

Overview

Teaching: 5 min
Exercises: 0 min
Questions

  • What is the Ocean Tracking Network?

  • How does your local telemetry network interact with OTN?

Objectives

Intro to OTN

The Ocean Tracking Network (OTN) supports global telemetry research by providing training, equipment, and data infrastructure to our large network of partners.

OTN Field Technicians

The OTN Field team are a group of experts whose combined years of experience can be leveraged, with study design advice and field assistance, by OTN reporting researchers. The OTN field team deploy and maintain scientific infrastructure around the coasts of Canada, as well as in various regions around the world.

OTN has recently added to its scientific fleet several ROVs and a side scan sonar unit. These tools allow for site evaluations as well as search and recovery of lost equipment.

All of these field endeavours generate an enormous amount of data and metadata records - irreplaceable and of great value to researchers.

OTN Data System

OTN and affiliated networks provide automated cross-referencing of your detection data with other tags in the system to help resolve “mystery detections” and provide detection data to taggers in other regions. OTN’s Data Managers will also extensively quality-control your submitted metadata for errors to ensure the most accurate records possible are stored in the database. OTN’s database and Data Portal website are excellent places to archive your datasets for future use and sharing with collaborators. We offer pathways to publish your datasets with OBIS, and via open data portals like ERDDAP, GeoServer etc. The data-product format returned by OTN is directly ingestible by analysis packages such as glatos and resonATe for ease of analysis. OTN offers support for the use of these packages and tools.

Better Together

The OTN Field and Data teams need to work closely together through much of their workflows. Researchers often request information about OTN-managed infrastructure as soon as the field work is complete. Records need to be passed from the field team to OTNDC as soon as is feasible and with as high a degree of accuracy as possible. Current workflows may need adjustment to better serve the needs of the research community.

Intended Audience

This set of workshop material is directed at OTN Technical staff from the Field department who want more information about how to report to OTN, and what happens behind the scenes in the OTN data system.

In this walkthrough we will be looking at:

The walkthough will have many opportunities for discussion and feedback as we strive to improve these processes moving forward.

Key Points

  • OTN is a global organization.

  • OTN supports research in ways both technical and data related

  • More information is available on the Data Portal

Introduction to the Data Portal

Overview

Teaching: 30 min
Exercises: 0 min
Questions

  • What is the Data Portal?

  • What features will be most useful to me?

Objectives

What is the Data Portal?

OTN’s Data Portal website is similar to DropBox or another file repository service. This is built using the open-source Plone software, and is sometimes called Plone around OTNHQ. While there are helpful links and tools to explore, this site is mainly used to hold private repository folders for each project. In a project folder, you can add files which can be viewed ONLY by users who have been given access. These folders are also where the Data Manager will upload Detection Extracts when they are ready.

OTN’s database is built on PostgreSQL/PostGIS and is hosted on OTN hardware at Dalhousie University. Many partner Nodes are hosted at other locations. Users do not have direct write access to the database: the files posted in a Data Portal folder will be downloaded, quality controlled and loaded into the database by a Data Manager. OTN Field Techs cannot access the database directly unless they are trained to do so.

Useful tools for Field Techs

OTN’s Data Portal website is used extensively by OTNHQ staff members for many purposes. A few of these are highlighted below!

Project Pages

Each project which has registered with OTN or a partner Node will have a public project summary page. These are searchable from the Home page using the interactive map tool, or the map search option. An example project page for project NCAT is here https://members.oceantrack.org/OTN/project?ccode=NCAT. You should make yourselves familiar with these webpages to share with partners.

This page contains the following sections:

  1. Title, citation, Node membership information and collaboration type (Tracker, Deployment or Data) - this will tell you lots about the project!
  2. An interactive map showing non-clickable tagging sites and clickable station-history information (note: some Nodes do not share these locations so they will be absent from the map)
  3. A list of the species studied by the project (provided by the PI)
  4. Contact information for Principal Investigators and Points of Contact
  5. Related URLs including:
    • for logged in users (unless project is public) - a link to the project’s private repository
    • for logged in users (unless project is public) - a link to the Detection Extracts subfolder in the project’s repository
    • a stations KML file containing the GPS locations of all stations (only if this project is part of a Node which shares these locations)
    • a deployments CSV containing a summary of all deployments (not the same format as the metadata – not provided if this project is part of a Node which doesn’t share these locations)
    • if a project is public - a KML and CSV file containing information regarding the project’s tagged animal records
  6. A summary of all the tag-projects whose tags have been detected on the selected array and the array-projects which have detected their tags.

You will likely be most interested in the information from item 6 - this can be used to learn about detections without contacting OTNDC

Example Project Page

Project repository

A project’s repository folder is only accessible to logged-in Plone users who have been given specific permission to view the folder. OTN Field staff should have access to all folders.

This is where researchers, and OTN staff, can share files with each other in a secure manner, ensuring they are archived properly.

Example Repository

Below, a few relevant features are highlighted.

Data and Metadata

This is a standard subfolder available in all project repository folders. This is the location for all tagging and deployment metadata, offloaded data files, and other useful documents. For OTN projects, often this folder is organized into missions and/or years for better coordination.

Files and folders can be added using the Add new... option on the left-hand menu. The folder organization is customizable, like Dropbox/Google Drive or similar programs.

TIP: Batch Uploading

If you are uploading multiple files at once, into the same folder, you can use the Batch Upload function to save time.

  1. Choose the Contents option on the left-hand menu
  2. Choose the Upload option
  3. You can browse for files or drag-and-drop them into the Upload window, until all relevant files are selected.
  4. Click the Upload button and wait for the upload to complete before navigating away.
  5. When you are finished, return to your standard folder display by choosing View from the left-hand menu.

Please do not Batch Upload single VRL files - these should be zipped together

TIP: Editing/Renaming

You can edit a file or folder after you’ve created it, using the Edit option on the left-hand menu.

Editing a folder:

Editing a file:

Plone has a version-control history, so it is possible to revert back to a previous version of a file if you edited it by mistake.

OTNDC staff receive an email each time you edit, delete or post a file.

TIP: Finding a current file

Within the Data and Metadata subfolder, it can often be difficult to find the current year/mission folder. This is because the default sorting is based on the date the file/folder was created, from oldest to newest. We have manually re-sorted for many projects, so that the newest folder is at the top for easier access, but this is not the default.

Please ensure you scroll to check all available mission/year folders before you determine which is the newest.

Inside the newest folder, there may be multiple files. If the titles/descriptions do not make it clear which one is newest you can always review the last modified date to determine this.

When in doubt: please ask OTNDC staff to help you find the newest file or export a clean file from the database for you, so you do not accidently use an outdated metadata sheet for mission planning. This will make everyone’s lives easier!

Detection Extracts

OTN and its partner Nodes create Detection Extracts on a regular basis, every 4 months (Push months are February, June, and October), following a cross-Node coordinated detection matching event known as a Data Push. These Detection Extract files contain only the detections for the year corresponding to the suffix on the file name. See the detailed detection extract documentation for more information.

If you are ever interested in learning more about what a project has detected, you are welcome to check out the content in the Detection Extracts. Reminder that you can not share this information without reviewing the OTN Data Policy or contacting OTNDC staff for guidance.

Equipment

This subfolder is the location where Shipping Lists can be posted and reviewed, if relevant. The shipping/inventory tracking workflow has moved to the SnipeIT inventory system and Asana.

Background

This subfolder is the location where OTN Project Management staff will upload loan agreements, project plans, and legal paperwork, if relevant. This folder is only visible to OTN staff members by default.

FAQ page

OTNDC has created a Frequently Asked Questions page on the Data Portal. This will contain useful information about OTN’s policies, Nodes, metadata, and data products. It is recommended you refer to this often and familiarize yourself with the answers to some of these questions, in case you are asked by partners.

If there are topics you think should be added, please reach out to OTNDC - this page can be updated at any time to remain valuable to users.

Current FAQs

Statistics Page

OTNDC has created a Statistics page on the Data Portal. These statistics are updated every 4 months during a Data Push, and can be used for presentations, exploration and general interest. Summaries include:

Additional summaries or plots can be created by OTNDC staff upon request, and if the need arises, even incorporated into this webpage. Reach out if you have ideas!

Example from Statistics Page

Key Points

  • The Data Portal has many useful features

Reporting to the Database

Overview

Teaching: 30 min
Exercises: 0 min
Questions

  • How do I report to the OTN Database?

  • Why should I report my data?

  • How do I receive my detection matches?

Objectives

Metadata Reporting and Best Practices

Reporting Data to an OTN Node

Researchers who are part of the OTN Network are encouraged to register their projects and report data and metadata in a timely manner to their Data Manager. This will benefit all researchers in the region through the database’s detection-matching system.

This presentation Reporting to OTN will cover some of the following topics.

You are encouraged to read the OTN FAQs Page for more information.

How to register with the OTN Database

In order to register a project with OTN, we require 3 metadata types:

  1. project metadata
  2. instrument deployment metadata
  3. tagging metadata

See the templates here. OTNDC@DAL.CA is the best contact for assistance

What is the benefit of registering with the OTN Database?

OTN and affiliated networks provide automated cross-referencing of detection data with other tags in the system to help resolve “mystery detections” and provide detection data to taggers in other regions. OTN’s Data Manager will also extensively quality control submitted metadata for errors to ensure the most accurate records possible are stored in the database. OTN’s database and Data Portal website are excellent places to archive datasets for future use and share with collaborators. We offer pathways to publish datasets with OBIS, and via open data portals like ERDDAP, GeoServer etc. The data-product format returned by OTN is directly ingestible by analysis packages such as glatos and resonATe for ease of analysis. OTN offers support for the use of these packages and tools.

Metadata Tips and Tricks

Below are some field-specific guidelines for completing metadata. The OTN Field team has their own metadata format, based on the Shortform Template, but containing OTN-specific columns, which is used for many projects.

The OTN-Field deployment metadata sheet is available HERE

If you are unsure of what information to put in a column, refer to the Data Dictionary tab of the metadata for other instructions.

Upon Deployment

When a mission contains deployments, there is some information needed:

OTN staff have had great success with recording deployments properly.

Upon Recovery

When a mission contains equipment recoveries, there is some information needed:

OTN staff have had great success with recording recoveries properly.

Download-only Missions

When a mission contains stations which are remotely downloaded, metadata still needs to be completed:

TIP: For deployments where multiple remote downloads occur (5-year+ VR4 moorings), on each download, you replace the previously listed download date in the metadata with the new date and accompanying comments.

Lost or Found Gear

Generally, equipment is marked with a y in the recovery indicator column if recovery is successfully completed during a mission, or n if recovery has not yet been attempted for the specific mooring. There are other cases which need special consideration in the metadata:

Recovered equipment:

Non-recovered equipment:

Found equipment:

Other accepted terms include moved, which is used when a mooring is recovered successfully, but was a substantial distance from its deployment location suggesting fishing interactions or movement during a storm.

OTN specific-columns

Mission IDs for both the recovery and deployment missions are helpful to ensure we’re able to link the mission report (notes) to the deployment history. These fields aren’t mandatory but can be very useful internally!

Consecutive deployment number is helpful for tracking stations which have been rolled-over and those which haven’t yet, but is not a mandatory field.

Raw Field Notes

Often, typos are introduced into metadata sheets while being transcribed. This means that the true information is only found in the raw field notes, contained in muddy write-in-the-rain notebooks. These are easily lost and not easily searchable.

For these reasons, it is strongly encouraged to take a photo of the field notes at the end of the mission and upload it to the project’s Data Portal repository folder for safe-keeping. These can be used to untangle metadata mysteries that might occur, and will preserve this information when projects are handed between technicians.

Example Raw Metadata Image

Key Points

  • Metadata templates are used to report records

  • OTN matches data and metadata across numerous geographic areas

  • Detection extracts are returned to researchers after data pushes

Snipe IT Inventory System

Overview

Teaching: 30 min
Exercises: 0 min
Questions

  • How do I add assets

  • How do I check in or out?

  • How do I fix my mistakes?

Objectives

SnipeIT Inventory System

In 2017, OTN deployed the SnipeIT Inventory system to track equipment shipments to partners and internally. This is an open-source tool with an underlying database, which OTNDC can access for internal metrics. To ensure its success, there are a couple of additional features we should add to our regular repertoire.

Snipe Dasboard

General Tasks

OTN Field technicians utilise SnipeIT on a regular basis, mainly doing the following activities:

Checking gear in/out

When a shipment is sent to/returned from a partner, or deployed/recovered by OTN Field Ops, its important to update the project the equipment (asset) is associated with.

Checking out equipment:

If you don’t know the correct collectioncode for a project please reach out to OTNDC staff to clarify.

Checking in equipment:

Changing status of equipment

Sometimes, you need to change the status of equipment in order to checkout the asset. The status will also need to be changed for units which are lost or being shipped for RMA.

Batch Import

When a new order of equipment has been received, its important to import the assets to the inventory as quickly as possible, so they can be shipped and deployed without issue.

Import Template

In order to use the batch import function, a .csv file needs to be created, following this standard template - Snipe Asset Import Template

The mandatory columns and their formats are:

You may edit the file in Excel, but please save as a .csv when you are finished. This will eliminate all special characters and formatting. If you have any issues, please reach out to OTNDC staff.

Batch Import Steps

If you have any issues, please reach out to OTNDC staff.

Creating New Users

When there is a new project receiving OTN equipment, this project will need a User in SnipeIT.

Creating a new user:

If you have any issues, please reach out to OTNDC staff.

Fixing Errors in SnipeIT

The SnipeIT underlying database is backed up periodically by Dal IT staff, so the history is possible to recover if you make an error. However, its possible that the problem is easily fixable, depending on what happened! If you’re unsure, please reach out to OTNDC staff. Currently, Brian Jones is the staff member responsible for Snipe.

Key Points

  • Inventory is important to keep up to date

ROV and SSS Reporting

Overview

Teaching: 30 min
Exercises: 0 min
Questions

  • Why is it important to keep ROV/SSS records?

  • How are ROV/SSS missions reported?

  • How can we improve this workflow?

Objectives

Record Keeping for Robotics Missions

ROV camera and remote sensing data are important contributors to our understanding of biological presence or absence in difficult-to-sample parts of the ocean. It can also make important contributions to habitat mapping or species composition/interaction observations, even when this was not the intended aim of the deployment. This creates an obligation to our many partners as well as our funders to record and document all ROV missions thoroughly for future re-use of this valuable data. This will include proper archiving of all collected data, video footage and other samples along with complete metadata to give temporal and geographical context.

Operators are also responsible for the constant verification and re-evaluation of the workflow to ensure we continue to safely and productively meet these obligations while adhering to the latest in industry reporting standards.

Current OTN ROV Reporting Requirements

The previous set of instructions for ROV/SSS Reporting was available on the Field Ops GitLab.

This is a new data format for OTNDC! We want to work with the Robotics Team to figure out what works. We do know that we need to archive the data somewhere, and ensure we have a record of all missions completed. We can use this records to show our “effort” to our funders, and document any lost/damaged equipment which may need replacement.

Currently:

  1. When a request for an ROV project is received and scoped, a project needs to be entered into the Mission Planning project in Asana. This information will be used by the Field Team Lead to work with the Robotics Team and Project Management to identify the requirements and any legal agreements required.
  2. Once an ROV mission has been completed, the following information must be added by the Operator to this OneDrive folder. Direct uploading is necessary because the size of most of these files precludes them being emailed or posted on our Data Portal website.
    • Data files from any instruments
    • Video files from the instruments
    • Any video clips that have been deemed relevant/interesting, or notes describing any significant video sections (for the Communications team).
    • GPS trackfile of the flightpath (if available), or start/end coordinates for the mission
  3. Once an ROV mission has been completed, the following information is emailed to OTNDC@Dal.ca or posted in the relevant Data Portal folder.

Missions completed for other partners will get their own project created in the OTN database, while all other OTN-led projects will be added to an OTN-Robotics catch-all project.

Establishing Standards Based on Industry Practices

OTNDC’s questions for the Robotics Team include:

We are excited to expand our robotics fleet and are keen and willing to change our expectations based on the experience of professionals. We currently do not have any tools built to process ROV-specific metadata (only gliders) making this an ideal time to identify standard formats.

Key Points

  • Data files, along with metadata about the time and location, are needed for each mission

  • We have a current workflow for reporting, but are open to adapt

Introduction to Asana

Overview

Teaching: 30 min
Exercises: 0 min
Questions

  • What is the purpose of Asana?

  • How can the Field Team use Asana?

Objectives

OTN’s Asana

At OTN, Asana is used for internal record keeping and collaborative action item tracking.

Asana allows us to be accountable, thorough, and comprehensive in our processes. This is essential for smooth communication and acts as a searchable archive of OTN’s institutional memory. Asana has replaced Podio and GitLab for the Field Team.

Why Asana?

OTN has several relevant Asana Teams, beyond Field Operations (described below), that you will need to interact with.

Field Operations Team

As the Field team has grown, job descriptions have expanded, and remote work has become more common it is now important to start using some of the lessons learned by the Data team to improve Field Ops communications. For this reason, there is now a Field Team Asana - https://app.asana.com/0/1201938039482799/overview.

Each OTN Field team member should have an Asana account created - please contact Project Management.

FieldOps Asana Home Page

This Field Ops Asana has been created as a place to track complex / important / outstanding issues so that they can be contributed to collaboratively and archived for future reference. This will be used in addition to Slack to make sure all Field Team members are communicating and information does not slip between the cracks of our busy schedules.

Asana projects include:

You will likely use the Field Shipping, Mission Planning, Found Gear Reporting and TOIL projects most often.

Exploring Projects

Each Project in Field Operations should include information in an Overview tab. This will detail exactly how to use this project, and how to complete the Tasks within it. This will differ for each project and therefore will not be covered in this curriculum.

Current members of this Asana Project include: all Field staff and members of the Project Management Office. Please tag them in any tasks (using the @ symbol) to prompt them for input.

Project Overview

Tasks

Within each project there are Tasks - these are the action items, and often contain subtasks with deadlines for completion. A Task may be defined differently in different projects - Ex: in Found Gear Reporting each task is one “washup”. In Receiver Inventory it’s a specific model of receiver.

Often, there is automation built into the creation of Tasks or Subtasks in a project. This helps us remember when we can complete each step of our action item. A great example is the Field Shipping project. Once a shipment’s status is changed to Active shipment then additional subtasks are populated for the assignee to follow. Similarly, if the shipment is labelled as Incoming vs. Outgoing there are different subtasks automatically created and so forth.

Tasks can be reassigned when needed to ensure they’re not forgotten when a staff member goes on vacation or on a field mission.

You can see all Tasks and Subtasks assigned to you, with their due dates, in the left-hand menu option My Tasks. You need to check these every day to ensure you are not missing tasks assigned to you by Project Management or other teams.

Asana Structure

More Information

The Field Ops Asana has a Onboarding Powerpoint with all the details needed to learn how to navigate Asana!

Key Points

  • Asana helps organize and archive tasks

  • There are several ways Asana is used by the Field Team

Ops Dashboard

Overview

Teaching: 30 min
Exercises: 0 min
Questions

  • What is the OTN Ops Dashboard?

Objectives

KPI Dashboard

OTNDC has developed an interactive web-based dashboard for Project Management staff called the OTN Ops Dashboard. It is available at this link and has a shared username and password for all OTN Staff.

This curriculum is public, so we cannot add the login credentials here, but please email OTNDC@Dal.ca for information if you are interested.

Note: this is for internal use only! The output from any of these KPI tools can not be shared externally without first discussing with Project Management or OTNDC.

Dashboard Banner

View KPI

The default summary page is the KPI Dashboard. This has a drop-down menu for each fiscal year, and contains Key Performance Indicator statistics which are required for annual reporting to CFI. These are pulled from the KPI reports on Asana as well as the annual report survey sent out to all partners.

Metrics here include:

Exporting Tool

The Exporting Tool menu option is to export the results from the KPI Dashboard in an Excel document. This is used by managers for reporting.

Admin Tools

The Admin Tools menu option is for OTNDC and Project Management staff only - allowing them to edit the targets for each KPI and other settings.

Receiver Report Tool

The Receiver Report Tool has a direct link to the OTN Database, and can be used to search all the deployments in our database. You can search in three ways:

  1. Search by collection codes (and/or year) to show all deployed receivers (owned by OTN or not!) for a certain project(s)
  2. Search by receiver serial numbers (and/or year) to show all deployments of a certain receiver(s), across all projects
  3. Search by both collection code and receiver serial number

You are able to export as an XLSX, and soon as a CSV as well.

TIP: use this to track down shipping chaos - where has a receiver been reporting?

Receiver Report Tool

Receiver Efficiency Index Tool

This Receiver Efficiency Index Tool was developed to determine which stations are performing the best and worst in an array. This is based on the calculations presented in Kendall et al. 2021

This tool allows you to choose a collection code and date range, and either view or export an interactive map displaying the “hot spots” of detection activity (yellow colour) in your array. This is important for OTN to use for array reconfiguration evaluations.

Receiver Efficiency Index Tool

Bibliography Graph

The Bibliography Graph is the newest tool added to the Ops Dashboard. It is an interactive network plot showing the publications by OTN researchers, with links between researchers who have co-authored papers. Researchers who publish more have larger dots in the graph. The researchers are coloured according to their Network membership.

There several interactive filter options including by year. There are likely to be more developments on this tool in future.

Bibliography Graph

Future Tools

This Dashboard is meant to serve as an internal mechanism for non-Data colleagues to view and understand data held in our databases. Are there more summaries you’d like to see here? Some ideas are:

This is meant to serve you! You can voice ideas at anytime by reaching out to OTNDC or creating an Issue in the kpi-dashboard GitLab project.

Key Points

  • There are several features on the Ops Dashboard that are useful to the Field team

Introduction to Nodes

Overview

Teaching: 30 min
Exercises: 0 min
Questions

  • What is an OTN-style Database Node?

Objectives

  • Understand the relationship between OTN and its Nodes.

What is a Node?

OTN partners with regional acoustic telemetry networks around the world to enable detection-matching across our communities. An OTN Node is an exact copy of OTN’s acoustic telemetry database structure, which allows for direct cross-referencing between the data holdings of each regional telemetry sharing community. The list of OTN Nodes is available at https://members.oceantrack.org. Data only needs to be reported to one Node in order for tags/detections to be matched across all.

How does a Node benefit its users?

OTN and affiliated networks provide automated cross-referencing of detection data with other tags in the system to help resolve “mystery detections” and provide detection data to taggers in other regions. OTN Data Managers extensively quality-control submitted metadata to ensure the most accurate records possible are stored in the database. One of the biggest benefits of interconnected Nodes is that researchers only need to report their records to ONE OTN style Node. OTN’s database and Data Portal website are well suited for archiving datasets for future use and sharing with collaborators.

The OTN system includes pathways to publish datasets with OBIS, and for sharing via open data portals such as ERDDAP and GeoServer. The data-product format returned by OTN is directly ingestible by analysis packages including glatos and resonATe. OTN offers continuous support for the use of these packages and tools.

For your interest, we have included here a presentation from OTN and FACT Managers, describing the relationship between OTN and its Nodes, the benefits of the Node system as a community outgrows more organic person-to-person sharing, as well as a realistic understanding of the work involved in hosting/maintaining a Node.

Node Managers

To date, the greatest successes in organizing telemetry communities has come from identifying and working with local on-the-ground Node Managers for each affiliated Node. The trusted and connected ‘data wranglers’ have been essential to building and maintaining the culture and sense of trust in each telemetry group.

Node Managers have been trained from groups including FACT, ACT, SAF, PATH, PIRAT and MigraMar. OTNDC staff assist with node management for NEP, SAF, MigraMar and OTN.

Nodes and Data Partners

Node Training

Each year OTN hosts a training session for Node Managers. This session is not only for new Node Managers, but also a refresher for current Node Managers on our updated tools and processes. The curriculum is available here.

Data Partners

In addition to OTN-supported nodes, we partner with numerous telemetry networks to ensure that effort and science is shared regardless of disparate database structures. Data system cross-walk efforts continue with ETN, IMOS and GLATOS.

Key Points

  • Nodes help telemetry researchers stay connected

  • Nodes are fully compatible

  • OTN supports Node Managers

OTN System, Structure and Outputs

Overview

Teaching: 25 min
Exercises: 0 min
Questions

  • What does an OTN-style Database look like?

  • What is the general path data takes through the OTN data system

  • What does the OTN data system output?

Objectives

  • Understand the OTN Database structure on a high level

  • Understand the general path of data of data through the OTN system

  • Understand what the OTN system can output

OTN Data System

The OTN data system is an aggregator of telemetry data made up of interconnected Node Databases and data processing tools. These work together to connect researchers with relevant and reliable data. At the heart of this system are Nodes and their OTN-style Databases.

Affiliated acoustic telemetry partner Networks may become an OTN Node by deploying their own database that follows the same structure as all the others. This structure allows Nodes to use OTN’s data loading processes, produce OTN data products, and match detections across other Nodes.

This lesson will give a short overview on the OTN Database and the way data is processed through it to create meaningful detection matches, summaries, and website updates.

Basic Structure

The basic structural decision at the centre of an OTN-style Database is that each of a Node’s projects will be subdivided into their own database schemas. These project schemas contain the relevant tables and data to that project. The tables included in each schema are created and updated based on which types of data each project is reporting.

Projects can have the type tracker (tags only), deployment (receivers only), or data (tags and receivers).

In addition to the project-specific schemas, there are some important common schemas in the Database. These additional schemas include the obis, erddap, geoserver, vendor, and discovery schemas. These schemas are found in each Nodes and are used during data processing and to create important end-products.

The amount of information shared through the discovery tables can be adjusted based on sharing and reporting requirements for each Node. For example, the OTN data policy allows OTN to share receiver locations, and project bounds, while the FACT data policy does not allow receiver locations, and so the discovery tables in FACT do not have that information.

OTN Database - path of data through the system

The Path of Data

The OTN data system takes 4 types of data/metadata: project, tag, instrument deployments, and detections. Most data has a similar flow through the OTN system even though each type has different notebooks and processes for loading. The exception to this is project metadata which has a more unique journey because it is completely user-defined, and must be used to initially define and create a project’s schema.

Project Data

Project metadata records are the first thing that need to be submitted and loaded, so that there is a home for all the other records to follow. Project data has a unique workflow from the other input data and metadata that flows into an OTN Node, it will be used to create the new schema in the Database for a project. The type of project selected (tracker, deployment, or data) will determine the format of the tables in the newly created schema. The type of project will also impact the loading tools and processes that will be used.

Tag, Deployment and Detections Data

Even though tag, deployment, and detections data all have their own loading tools and processes, their general path through the database is the same.

OTN Database - structural ER diagram

OTN Data Products

The OTN Database has specific data products available, based upon the clean processed data, for researchers to use for their scientific analysis.

In order to create meaningful Detection Extracts, OTN and affiliated Nodes only perform cross-matching events every 4 months (when a reasonable amount of new data has been processed). This event is called a synchronous Data Push. In a Data Push:

Summary Map

Backing Up Data

OTN data systems are designed with redundancy that ensures zero data loss in the event of any hardware failure. OTN has a plan in place for data archiving with CIOOS, should the network cease to be supported by our funding agency.

Key Points

  • All OTN-style Databases have the same structure

  • Databases are divided into project schemas which get certain tables based on the type of data they collect

  • Data in the OTN system moves from the raw tables to the intermediate tables to the upper tables before aggregation

Data Push

Overview

Teaching: 10 min
Exercises: 0 min
Questions

  • What is a Data Push?

  • What are the end products of a Data Push?

Objectives

What is a Data Push?

A Data Push is when the OTN data system is re-verified and any new relevant information is sent to researchers. New data being brought in is cut off so that what’s in the system can be reliably verified. This way any issues found can be fixed and the data can be in the best form based on the information available at that moment. Once verification is done detections are matched across nodes and detection extracts are sent out to researchers. This is also the time when summary schemas like discovery, erddap, and geoserver are updated with the newly verified and updated data.

What is the Push Schedule?

Push events happen three times a year. They start on the third Thursday of the “push month” which are February, June, and October. This date is the cut-off date for all data-loading: no records can be loaded after this until important verification and processing tasks are completed.

With the increased number of Nodes joining the Pushes, we are announcing the schedule for the next year. Please prepare in advance and mark your calendars.

Example Push schedule through 2025:

As OTN Field Techs you will have a submission cut-off date one week before the actual Push begins

Activities During the Push

Both node managers and data analysts have important roles during a push:

Node Managers need to ensure that the following is done:

  1. The first job is to get the Node’s data loaded in time for the cut-off date. Data is submitted by researchers on a continuous basis, and generally increases just before a cut-off date. We prioritize loading data as it arrives, to attempt to prevent a backlog near the Push date.
  2. The second job for Node Managers is to create and send out Detection Extracts when they are ready to be made.

Once the cut-off date has passed Node Managers are “off duty”, then when it’s time for Detection Extracts to be created and disseminated that task is assigned to the Node Managers, but this does not signify the end of the Push. There are several more “behind the scenes” steps required.

Data Analysts have many jobs during a Push, including:

  1. verify all data in the database
  2. perform cross-node matching
  3. perform robust back-ups
  4. repopulate our Data Portal website

Detection Extracts

Detection Extracts are the main output of the Push. They contain all the new detection matches for each project. There are multiple types of detection extracts OTN creates:

Detection Extract files are formatted for direct ingestion by analysis packages such as glatos and resonate. They use DarwinCore terminology where relevant.

These files are uploaded to a projects Data Portal folder and can only be shared in accordance with OTN’s Data Policy.

Key Points

  • A Data Push is when we verify all the data in the system, fix any issues, and then provide detection matches to researchers

  • Detection Extracts are the main end product of the Push

Beyond Telemetry

Overview

Teaching: 0 min
Exercises: 0 min
Questions

  • In what other ways does OTN support animal telemetry researchers?

Objectives

Beyond Telemetry Reporting

OTN serves the telemetry and broader scientific community through its primary purpose, as a data aggregation node, but that is not where the community support ends. In addition to curating and brokering interproject telemetry records, OTN has the following aspects:

OBIS publishing

OTN is an OBIS thematic node, tasked with curating and coordinating the publishing of aquatic animal tracking data. OTN adheres closely to Darwin Core standards throughout the data reporting process.

OTN is involved with the training of OBIS members and development of tools and processes, through training workshops and code sprints.

DOI minting

OTN creates DOI data sets when requested by researchers. These DOIs primarily support publications and descriptive links are held on the OTN Data Portal publication data repository.

Study Halls

To reduce the stress of working in isolation, especially for early career researchers, OTN established a weekly ‘Study Hall’ session. Networking and collaborative problem solving in a virtual setting are hosted with a broad geographic attendance. We strongly encourage Field Techs to attend! They occur every Thursday from 2-4 Atlantic. Contact OTNDC for a Zoom link.

study hall map

Workshops

OTN has hosted many workshops in the past which contain different code sets relating to telemetry analysis. Many of our Intro to R workshops are based upon this curriculum from The Carpentries.

Past custom workshops with repositories and webpages are listed here

Key Points

  • OTN provides support to researchers in many ways

OBSOLETE - Introduction to GitLab

Overview

Teaching: 30 min
Exercises: 0 min
Questions

  • What is the purpose of GitLab?

  • How can the Field Team use GitLab?

Objectives

#This lesson is obsolete - nearly all of these tasks are now done by the Field Team in Asana

OTN’s GitLab

At OTN, GitLab is used for internal record keeping and collaborative code development. The main purpose (for non-computer scientists at OTN) is the ability to use GitLab Issues containing templates of task-lists to ensure the DAQ team NEVER forgets a step in data loading, and that no file is ever lost/forgotten in an inbox. We track all communications with researchers and archive all important information for future use.

GitLab allows us to be accountable, thorough, and comprehensive in our data-loading process. This is essential for smooth communication and acts as a searchable archive of OTN’s institutional memory.

Field Ops Gitlab

As the Field team has grown, job descriptions have expanded, and remote work has become more common, it is now important to start using some of the lessons learned by the Data team to improve Field Ops communications. For this reason, there is now a Field Team GitLab project - https://gitlab.oceantrack.org/otnfield/OTN_Field_Ops.

Each OTN Field team member should have a GitLab account created - please fill out this signup form for an account on GitLab if you don’t have an account already.

This Field Ops Gitlab has been created as a place to track complex / important / outstanding issues so that they can be contributed to collaboratively and archived for future reference. This will be used in addition to Slack to make sure all Field Team members are communicating and information does not slip between the cracks with our busy schedules.

Gitlab Issues should be created to track:

Current members of this Gitlab Project include: all Field staff, members of the Project Management Office, some members of the Data Team. Please tag them in an issue (using the @ symbol) to prompt them for input.

Field team members leave (for vacation, or permanently) and so a shared record of important events will be essential to ensure knowledge-transfer. Having the information in your email is a great first step, but it may become lost or buried and is not easily shared/searched by colleagues.

FieldOps GitLab Home Page

How to Use GitLab

Menu on the left-hand sidebar:

Top menu banner:

You will use the Issues and Wiki left-hand sidebar options, and the personal Issue Dashboard in the top menu banner (right side) most often.

Issue Templates

For tasks that are repeated (Canadian shipments, international shipments, returning equipment, washups, Fedex insurance claims, etc) the same set of steps are often required each time. GitLab Issues have an option to populate with a standardized template checklist.

Currently, the Field Ops GitLab only has one template, for found_gear, and some common shipping tasks. However, if this is deemed successful we can expand.

Using Issue Templates

When you navigate to Issues on the left-hand sidebar menu, there will be an option to create a New Issue. By choosing the New Issue button in the top-right of your screen, you will be taken to a new, blank issue form. To fill out the fields you will need to do the following:

Now, with all information completed, you can select Create Issue.

GitLab Issue Creation - with templates

GitLab Wiki

The Field Ops GitLab has a Wiki page where important files, templates, and procedures are outlined. Currently, there are several resources available to assist with day-to-day activities for the Field Team:

Theoretically, this curriculum should cover the same content as the Wiki, but the Wiki is a good reference as well and can be edited to contain any information deemed important.

Key Points

  • GitLab helps organize and archive tasks

  • There are several ways GitLab can be used by the Field Team