Pixel by Pixel: How the USGS, U.S. Forest Service Checked the Work of a Computerized Landsat Mapping Project
LCMAP Reference Data
About a year ago, scientist Bruce Pengra sat at his desk with a conundrum.
As part of the team working on the reference dataset for the USGS Land Change Monitoring, Assessment, and Projection (LCMAP) initiative’s land cover mapping project, Pengra was staring at a single speck of satellite data, a nondescript blotch of color representing a 30-meter plot of ground in Washington state.
What was this land used for? Getting it right was critical for the reference dataset—a tool to check the accuracy of LCMAP and provide precise estimates of land cover and change. Such datasets have been produced for years through a variety of methods for a variety of maps, but Pengra’s job wasn’t to make a call on a single point in time. He was reviewing its history across 33 years.
Pengra had three computer screens open to help him decide. On the left, he had the area’s satellite-recorded history since 1985, viewed through a U.S. Forest Service (USFS)-designed program called TimeSync. In the center were high resolution images from Google Earth, linked to the plot’s latitude-longitude coordinates. On the right, a geographic information system (GIS) program hummed along, also linked to the location, full of data layers like precipitation and forest type.
The plot—destined to be one pixel out of billions in a national map—looked like a pasture. The grass didn’t disappear at harvest time. There were no roads, no patches of trees, no pools of water.
But there was something curious. At least once since 1985, the pixel’s spectral signature and the land surrounding it suggested bustling human activity. With a little investigation, Pengra discovered that the area had been used as a military training site.
Therein lay Pengra’s conundrum.
“Once you know that, then you have to figure out, is that a developed use or is that a pasture?” said Pengra, a contractor at the USGS Earth Resources Observation and Science (EROS) Center. “If it was just a one-time use, it’s just a pasture. But if it’s used every year or every other year? That’s something you have to consider.”
The answer, arrived at with the help of the exhaustive set of criteria that guided the work of Pengra and his teammates, was pasture.
And with that, Pengra moved on. There were tens of thousands of pixels left, after all, and plenty of other conundrums to attend to.
The anecdote offers a small window into what it took to create the reference dataset for LCMAP—an initiative unlike any produced before in that it is designed for a set of maps that track land cover change to every pixel in the Conterminous U.S. (CONUS) for more than three decades.
“This is one of the few that’s been independently collected for this many years over this large of a geographic area,” said Jo Horton, an EROS contractor and remote sensing scientist who served as a senior interpreter alongside Pengra. “There have been some attempts to do a global reference data collection, but they don’t cover this span of years.”
Because of its unprecedented scale, the resulting dataset has piqued the interest of multiple other remote sensing scientists and promises to offer insight and guidance to a host of land change projects, both in the U.S. and around the globe.
Human Eyes, Checking a Computer’s Work
The massive undertaking saw Horton, Pengra and just over a dozen other interpreters at EROS and USFS contractors at Utah State University tagging a total of 25,000 pixels, one by one, to estimate how close LCMAP’s automated sorting of satellite data had come to accurately classifying land cover conditions on the ground.
The work could be tedious and difficult—one interpreter bowed out due to frequent headaches—but it was also critical to LCMAP’s scientific goals. LCMAP will soon release a 10-product suite of mapping data that characterizes the nation’s landscapes an annual basis from 1985-2018, leaning on USGS Landsat satellite observations to map land cover and track change across some 9 billion plots of land.
The reference dataset produced by EROS and its partners over the past five years amounts to a yardstick for product quality, offering assurance to scientists that the data is right for the job.
As the need for an LCMAP version of the quality control tool first emerged, the USFS was in the midst of a similar project designed to detect tree cover disturbances through Landsat data. The agency had already produced the TimeSync tool, and the needs of the projects overlapped, so the partnership was born.
“The combined effort saved both agencies time and money, significantly reduced duplicated efforts, and the project ultimately represented a smart approach for meeting mutual objectives,” according to Dr. Thomas Loveland, the former EROS Chief Scientist who led LCMAP efforts until his retirement in 2018.
Collaboration on reference datasets is nothing new, but LCMAP presented unique challenges. Small area projects might collect ground data at the time of a satellite overpass, but such an approach is resource-intensive and impractical for a nationwide project. Large-area projects rely on interpreters and ancillary data, such as high-resolution aerial imagery.
Even in that, however, the LCMAP work was unique. The project not only looked at CONUS, but looked at CONUS annually, back through time. Aerial photos from 1985 might not match up in quality or consistency with imagery from 2012, for example. Some of the ancillary data sources used for recent years didn’t exist three decades ago.
Solutions to these challenges came in the form of human capital, interagency collaboration, and meticulous record-keeping.
LCMAP’s human interpreters from the USGS and USFS put eyes on individual Landsat pixels and sorted them into land cover classes like developed, trees, or wetland. The 25,000 pixels were selected at random. A subset of them—selected based on their difficult-to-classify nature—went through a second round of independent interpretation, with disagreements between interpreters settled through consultation with a senior interpreter.
The exhaustive process will be traceable by users through the 300-odd pages of validation material that will accompany the public release of LCMAP’s Collection 1 product suite.
“If you give me one of the plots, I can tell you who did the original interpretation, if it was a duplicate interpretation, when or if it was reviewed based any of our (quality assurance) criteria – I can backtrack all of that,” Horton said. “Right now, we’re condensing all of that information so people who don’t know my shorthand will be able to do that, as well.”
‘Like Visiting 25,000 Places from Your Desk’
Horton can also tell you stories about the land that can only come from the plot-by-plot, year-by-year nature of the work. Taken as a whole, for example, the reference dataset captures a drop in the expansion of developed land that accompanied the 2007-09 housing crisis. Horton, sitting at her computer with three screens and pixels to sort, saw some of that at the neighborhood level.
“I had a plot where you could tell that they were cutting down the trees (for development). They had put in the cul-de-sac road, and then they just left,” she said. “If you look at the timing of the photo imagery, they left in that 2007-2008-2009 timeframe. And they never came back. The last time I looked at that plot, it was just going back to shrub.”
Stories like that point to the reason some remote sensing scientists are drawn to a job that would see them staring at screens to sort pixels of satellite data at a clip of a few dozen a day.
“It’s like getting to visit 25,000 places around the country, but do it from your desk,” said Jesslyn Brown, the USGS lead for LCMAP.
Photo interpretation was once a more prevalent part of remote sensing work. Brown’s first exposure to EROS came while interpreting photos that had been archived at the Center for an archaeology mapping assignment, long before her arrival at the USGS.
Open data, powerful desktop and cloud computing systems and widespread high-speed internet connectivity have resulted in more automated imagery interpretation. But the expertise of photo interpreters remains a necessary backstop for automated mapping, and Brown said the ones who labored to produce the LCMAP reference dataset have proven their worth many times over—point by point, pixel by pixel.
“It takes a certain kind of person to do this,” Brown said. “They’re sort of like teachers – they don’t get near enough credit for what they do. It’s sort of amazing, to be honest.”
Below are other science projects associated with this project.
LCMAP Validation Data Captures Impact of U.S. Economic Downturn
About a year ago, scientist Bruce Pengra sat at his desk with a conundrum.
As part of the team working on the reference dataset for the USGS Land Change Monitoring, Assessment, and Projection (LCMAP) initiative’s land cover mapping project, Pengra was staring at a single speck of satellite data, a nondescript blotch of color representing a 30-meter plot of ground in Washington state.
What was this land used for? Getting it right was critical for the reference dataset—a tool to check the accuracy of LCMAP and provide precise estimates of land cover and change. Such datasets have been produced for years through a variety of methods for a variety of maps, but Pengra’s job wasn’t to make a call on a single point in time. He was reviewing its history across 33 years.
Pengra had three computer screens open to help him decide. On the left, he had the area’s satellite-recorded history since 1985, viewed through a U.S. Forest Service (USFS)-designed program called TimeSync. In the center were high resolution images from Google Earth, linked to the plot’s latitude-longitude coordinates. On the right, a geographic information system (GIS) program hummed along, also linked to the location, full of data layers like precipitation and forest type.
The plot—destined to be one pixel out of billions in a national map—looked like a pasture. The grass didn’t disappear at harvest time. There were no roads, no patches of trees, no pools of water.
But there was something curious. At least once since 1985, the pixel’s spectral signature and the land surrounding it suggested bustling human activity. With a little investigation, Pengra discovered that the area had been used as a military training site.
Therein lay Pengra’s conundrum.
“Once you know that, then you have to figure out, is that a developed use or is that a pasture?” said Pengra, a contractor at the USGS Earth Resources Observation and Science (EROS) Center. “If it was just a one-time use, it’s just a pasture. But if it’s used every year or every other year? That’s something you have to consider.”
The answer, arrived at with the help of the exhaustive set of criteria that guided the work of Pengra and his teammates, was pasture.
And with that, Pengra moved on. There were tens of thousands of pixels left, after all, and plenty of other conundrums to attend to.
The anecdote offers a small window into what it took to create the reference dataset for LCMAP—an initiative unlike any produced before in that it is designed for a set of maps that track land cover change to every pixel in the Conterminous U.S. (CONUS) for more than three decades.
“This is one of the few that’s been independently collected for this many years over this large of a geographic area,” said Jo Horton, an EROS contractor and remote sensing scientist who served as a senior interpreter alongside Pengra. “There have been some attempts to do a global reference data collection, but they don’t cover this span of years.”
Because of its unprecedented scale, the resulting dataset has piqued the interest of multiple other remote sensing scientists and promises to offer insight and guidance to a host of land change projects, both in the U.S. and around the globe.
Human Eyes, Checking a Computer’s Work
The massive undertaking saw Horton, Pengra and just over a dozen other interpreters at EROS and USFS contractors at Utah State University tagging a total of 25,000 pixels, one by one, to estimate how close LCMAP’s automated sorting of satellite data had come to accurately classifying land cover conditions on the ground.
The work could be tedious and difficult—one interpreter bowed out due to frequent headaches—but it was also critical to LCMAP’s scientific goals. LCMAP will soon release a 10-product suite of mapping data that characterizes the nation’s landscapes an annual basis from 1985-2018, leaning on USGS Landsat satellite observations to map land cover and track change across some 9 billion plots of land.
The reference dataset produced by EROS and its partners over the past five years amounts to a yardstick for product quality, offering assurance to scientists that the data is right for the job.
As the need for an LCMAP version of the quality control tool first emerged, the USFS was in the midst of a similar project designed to detect tree cover disturbances through Landsat data. The agency had already produced the TimeSync tool, and the needs of the projects overlapped, so the partnership was born.
“The combined effort saved both agencies time and money, significantly reduced duplicated efforts, and the project ultimately represented a smart approach for meeting mutual objectives,” according to Dr. Thomas Loveland, the former EROS Chief Scientist who led LCMAP efforts until his retirement in 2018.
Collaboration on reference datasets is nothing new, but LCMAP presented unique challenges. Small area projects might collect ground data at the time of a satellite overpass, but such an approach is resource-intensive and impractical for a nationwide project. Large-area projects rely on interpreters and ancillary data, such as high-resolution aerial imagery.
Even in that, however, the LCMAP work was unique. The project not only looked at CONUS, but looked at CONUS annually, back through time. Aerial photos from 1985 might not match up in quality or consistency with imagery from 2012, for example. Some of the ancillary data sources used for recent years didn’t exist three decades ago.
Solutions to these challenges came in the form of human capital, interagency collaboration, and meticulous record-keeping.
LCMAP’s human interpreters from the USGS and USFS put eyes on individual Landsat pixels and sorted them into land cover classes like developed, trees, or wetland. The 25,000 pixels were selected at random. A subset of them—selected based on their difficult-to-classify nature—went through a second round of independent interpretation, with disagreements between interpreters settled through consultation with a senior interpreter.
The exhaustive process will be traceable by users through the 300-odd pages of validation material that will accompany the public release of LCMAP’s Collection 1 product suite.
“If you give me one of the plots, I can tell you who did the original interpretation, if it was a duplicate interpretation, when or if it was reviewed based any of our (quality assurance) criteria – I can backtrack all of that,” Horton said. “Right now, we’re condensing all of that information so people who don’t know my shorthand will be able to do that, as well.”
‘Like Visiting 25,000 Places from Your Desk’
Horton can also tell you stories about the land that can only come from the plot-by-plot, year-by-year nature of the work. Taken as a whole, for example, the reference dataset captures a drop in the expansion of developed land that accompanied the 2007-09 housing crisis. Horton, sitting at her computer with three screens and pixels to sort, saw some of that at the neighborhood level.
“I had a plot where you could tell that they were cutting down the trees (for development). They had put in the cul-de-sac road, and then they just left,” she said. “If you look at the timing of the photo imagery, they left in that 2007-2008-2009 timeframe. And they never came back. The last time I looked at that plot, it was just going back to shrub.”
Stories like that point to the reason some remote sensing scientists are drawn to a job that would see them staring at screens to sort pixels of satellite data at a clip of a few dozen a day.
“It’s like getting to visit 25,000 places around the country, but do it from your desk,” said Jesslyn Brown, the USGS lead for LCMAP.
Photo interpretation was once a more prevalent part of remote sensing work. Brown’s first exposure to EROS came while interpreting photos that had been archived at the Center for an archaeology mapping assignment, long before her arrival at the USGS.
Open data, powerful desktop and cloud computing systems and widespread high-speed internet connectivity have resulted in more automated imagery interpretation. But the expertise of photo interpreters remains a necessary backstop for automated mapping, and Brown said the ones who labored to produce the LCMAP reference dataset have proven their worth many times over—point by point, pixel by pixel.
“It takes a certain kind of person to do this,” Brown said. “They’re sort of like teachers – they don’t get near enough credit for what they do. It’s sort of amazing, to be honest.”
Below are other science projects associated with this project.