The U.S. Geological Survey Upper Midwest Environmental Sciences Center is developing deep learning algorithms and tools for the automatic detection, enumeration, classification, and annotation of seabirds and other marine wildlife from digital aerial imagery — advancing cutting-edge research in collaboration with the Bureau of Ocean Energy Management (BOEM), the U.S. Fish and Wildlife Service (FWS), and the University of Michigan.
Remote sensing and automation technologies are rapidly becoming essential tools to support critical wildlife monitoring objectives. Areas of the ocean and Great Lakes that are important for wildlife are now being surveyed using aircraft equipped with cameras that capture high resolution imagery generating large amounts of data. Deep learning and computer vision tools can be used to efficiently detect and classify wildlife found in high resolution aerial imagery, mitigating the need for labor-intensive data processing. This project seeks to improve the efficiency and accuracy of airborne wildlife population surveys with the use of remote sensing and machine learning. The methodology developed in this project will generate refined wildlife counts from aerial imagery, which will help inform harvest and other regulatory decisions, improve environmental assessments, and support impact analyses of wildlife exposure to offshore wind energy development projects and other activities in U.S waters.
We actively engage with all components of this work, including data collection and archival, annotation workflows, deep learning model development, and model deployment on large image datasets from aerial surveys. With these components in mind, we develop cost-effective, modular, and reproducible methods that other researchers and practitioners to apply these data and tools to their research questions and information needs.
Image Datasets
High resolution digital imagery collected during aerial surveys of the Atlantic Outer Continental Shelf and the Great Lakes provide the data for algorithm development, as well as baseline information on wildlife distributions and abundance. This project incorporates imagery from several sources, including previous BOEM supplied datasets and ongoing image collection work by the FWS.
Annotation Process
UMESC collects annotation data by hosting the Wildlife Annotation Tool (WAT), a customized annotation tool based on an open-source tool called CVAT (GitHub - opencv/cvat). The Wildlife Annotation Tool allows both internal DOI and public users to visualize imagery and provide annotation data based on their wildlife expertise. For this study, we are annotating to the lowest taxonomic level and incorporating information on age, gender, and activity when resolvable. There are currently over 70,000 manually developed annotations representing approximately 205 unique taxonomic classifications of birds, sea turtles, marine mammals, bony and cartilaginous fish.
Model development
We are developing models in two stages. The first stage focuses on object detection, with the goal of detecting any wildlife present in the image. The second stage expands the model to incorporate object classification and produce a refined wildlife identification value for each detected wildlife object. Using the annotated data, detection models are trained to draw bounding boxes around visible wildlife targets including mammals, reptiles, birds, bony fish, and cartilaginous fish. Once detected, these images are passed through a classification workflow for a more specific label. For more information regarding our classification work, see our publication (https://doi.org/10.1002/rse2.318).
Videos
View a recording of a presentation by of Kyle Landolt given at the 2020 Workshop | ETWG (nyetwg.com)
State of the Science Virtual Session - YouTube
- Overview
The U.S. Geological Survey Upper Midwest Environmental Sciences Center is developing deep learning algorithms and tools for the automatic detection, enumeration, classification, and annotation of seabirds and other marine wildlife from digital aerial imagery — advancing cutting-edge research in collaboration with the Bureau of Ocean Energy Management (BOEM), the U.S. Fish and Wildlife Service (FWS), and the University of Michigan.
Remote sensing and automation technologies are rapidly becoming essential tools to support critical wildlife monitoring objectives. Areas of the ocean and Great Lakes that are important for wildlife are now being surveyed using aircraft equipped with cameras that capture high resolution imagery generating large amounts of data. Deep learning and computer vision tools can be used to efficiently detect and classify wildlife found in high resolution aerial imagery, mitigating the need for labor-intensive data processing. This project seeks to improve the efficiency and accuracy of airborne wildlife population surveys with the use of remote sensing and machine learning. The methodology developed in this project will generate refined wildlife counts from aerial imagery, which will help inform harvest and other regulatory decisions, improve environmental assessments, and support impact analyses of wildlife exposure to offshore wind energy development projects and other activities in U.S waters.
We actively engage with all components of this work, including data collection and archival, annotation workflows, deep learning model development, and model deployment on large image datasets from aerial surveys. With these components in mind, we develop cost-effective, modular, and reproducible methods that other researchers and practitioners to apply these data and tools to their research questions and information needs.
Image Datasets
High resolution digital imagery collected during aerial surveys of the Atlantic Outer Continental Shelf and the Great Lakes provide the data for algorithm development, as well as baseline information on wildlife distributions and abundance. This project incorporates imagery from several sources, including previous BOEM supplied datasets and ongoing image collection work by the FWS.
Annotation Process
UMESC collects annotation data by hosting the Wildlife Annotation Tool (WAT), a customized annotation tool based on an open-source tool called CVAT (GitHub - opencv/cvat). The Wildlife Annotation Tool allows both internal DOI and public users to visualize imagery and provide annotation data based on their wildlife expertise. For this study, we are annotating to the lowest taxonomic level and incorporating information on age, gender, and activity when resolvable. There are currently over 70,000 manually developed annotations representing approximately 205 unique taxonomic classifications of birds, sea turtles, marine mammals, bony and cartilaginous fish.
Model development
We are developing models in two stages. The first stage focuses on object detection, with the goal of detecting any wildlife present in the image. The second stage expands the model to incorporate object classification and produce a refined wildlife identification value for each detected wildlife object. Using the annotated data, detection models are trained to draw bounding boxes around visible wildlife targets including mammals, reptiles, birds, bony fish, and cartilaginous fish. Once detected, these images are passed through a classification workflow for a more specific label. For more information regarding our classification work, see our publication (https://doi.org/10.1002/rse2.318).
Sources/Usage: Public Domain.A screenshot of the Wildlife Annotation Tool hosted by UMESC to let users visualize imagery and provide identification values using their wildlife expertise.
Sources/Usage: Public Domain.A collage of oceanic wildlife captured from aerial imagery including seabirds, turtles, bony fish, and other aquatic species (imagery provided by BOEM)
Sources/Usage: Public Domain.A collage of oceanic wildlife detected by machine learning methods and classified at a high level (e.g. bird and reptile with “confidence” values) from aerial imagery (imagery provided by BOEM) Videos
View a recording of a presentation by of Kyle Landolt given at the 2020 Workshop | ETWG (nyetwg.com)
State of the Science Virtual Session - YouTube
- Data
- Publications