Eyes on Earth Episode 45 - Harmonized Landsat-Sentinel

JOHN HULT:

Hello everyone and welcome to another episode of Eyes on Earth. We're a podcast that focuses on our ever-changing planet and on the people here at EROS and across the globe who use remote sensing to monitor and study the health of Earth. I'm your host for this episode, John Hult. We talk a lot about Landsat on this show. That is the joint NASA/USGS satellite program that's been monitoring the Earth for nearly 50 years. We also talk occasionally about the European Space Agency Sentinel-2 satellites, which are similar to Landsat and used for similar purposes. But similar is not same. The imagery isn't interchangeable. That's why scientists have spent several years working to bring Landsat and Sentinel into alignment. Doing that would mean more frequent observations and more chances to track change, ranging from deforestation to disaster impacts and recovery, in near real time. On today's show we're talking with Jeff Masek and Brian Freitag. Two members of the NASA team behind the Harmonized Landsat Sentinel-2 project., with data that will be distributed by the NASA Land Processes Distributed Archive Center or LP DAAC. LP DAAC is located at EROS, by the way. Masek is a NASA Landsat project Scientist and a member of the Landsat Science team. He helped build the algorithm for the new data set. Freitag is leading the effort to process the data set in the cloud and make it accessible to the research community. Jeff, Brian, welcome to Eyes on Earth.

JEFF MASEK:

Thanks so much.

FREITAG:

Thanks a lot, John.

HULT:

Alright, well let's get into this here. First off, let's talk a little bit about Landsat and Sentinel. How are these satellites systems similar and how are they different?

MASEK:

From an applications perspective, both Sentinel and Landsat serve a lot of the same needs. They're used to look at land characteristics and land change worldwide. Things like global agriculture, natural disasters, changing icecaps, urbanization. Basically, when you need to get down to a spatial resolution fine enough to see human activities and the consequences of human activities. That said, there are some differences, technical differences, between the systems. Landsat sees objects on the ground as fine as about 30 meters, and Sentinel-2 is a little bit finer resolution. It gets down to about 10 meters.

HULT:

Just to break in there quickly. 30-meter resolution, we're talking about pixels on an image around the size of a baseball infield for 30-meters?

MASEK:

Correct

HULT:

And a little smaller for Sentinel?

MASEK:

Correct

HULT:

Ok

MASEK:

They have a lot of spectral bands in common. So that's basically where the electromagnetic spectrum we are actually observing the light being reflected off the Earth. But the band placement is a little bit different between the two systems. They also are in different orbits. So, they're observing the same target with slightly different viewing geometry and again that effects the brightness of the imagery that comes out for a particular target.

HULT:

If I can break in there again. This is something that if we're trying to explain it would be like the difference between, taking a picture of a stop sign from 50 yards away. One person is standing directly in front of it and the other person is sort of standing to the left and they take photos. There is going to be a slight difference there. Is that kind of a way to describe that?

MASEK:

Yeah. Exactly. If you have ever looked at a lawn. If you look at it with the sun behind you it's really bright. Then if you turn around and look at it with the sun in front of you, it's really dark because you are looking at shadows. So, the angle that you are looking at something really effects the overall brightness and what we call the spectral response.

HULT:

Those are the differences we are looking at. Different angles, different spectral bands. With Landsat we have a thermal band as well, right?

MASEK:

Correct. Right. So that's another difference between the two systems. Landsat has the thermal infrared bands where we can look at surface temperature. Sentinel-2 does not have those bands. That's a unique aspect of Landsat that we only get from that system.

HULT:

Can we talk a little bit about what Landsat and Sentinel imagery is used for today?

MASEK:

So Landsat is our longest-lived remote sensing system. First launched in 1972 and so we have going on 50 years now of long-term record of how Earth's land environment has changed. So, we can look at deforestation. We can look at changes in agriculture and irrigation. We can look at changes in glaciers and ice caps. In addition, both Sentinel and Landsat are used for land management. So, if you want to understand what crops are growing where, what areas need more irrigation what areas needs more fertilizer, both Landsat and Sentinel data are used for that as well. So they are used extensively both for scientific research and land management. 

HULT:

So, what is Harmonized Landsat Sentinel Data? What is this that you are creating?

MASEK:

What we are trying to do is to bring the two data sets together and do the necessary geometric and radiometric adjustments so they can be used completely interchangeably. What that means in practice is we take the input data, we map them to the same grid, we map them to the same 30-meter spatial resolutions, so they physically over lay each other. And then we adjust the spectral values the radiometry, so they look like each other. We do the angular correction. We do the band pass corrections and produce the products that are easy for people to use. The idea of putting Landsat and Sentinel together is that you can actually start to get a temporal resolution that's down closer to that daily interval, but with a higher spatial resolution.

HULT:

What would that allow people to do, first off. And secondly, why can't we do those things now? What stands in the way?

MASEK:

In the case of Landsat, we only get an image every 16 days, or 8 days if you have 2 satellites in orbit. There's a lot of phenomena that vary on time scales that are shorter than 16 days or 8 days, right? You could look at an inundation event, you could look at the progression a fire, crop harvesting, issues of water quality. What you really want is imagery that captures that rapid variability. We don't have that from Landsat alone. So the idea of Harmonized Landsat Sentinel is you bring both data sets together and you end up with a data set which provides you an observation at that high resolution every two to three days. Which is phenomenal.

HULT:

So, we're looking at instead of seeing something every five days, as long as there are no clouds, with Sentinel, or seeing something every 8 to 16 days with Landsat, we are looking at maybe kind of a three-day window or so? Are there other things that will help you do, operations or applications that stand out?

MASEK:

The big one is agriculture. That was really the big push for HLS. Crops can vary really from day to day in terms of their condition. And in addition, a lot of crops are managed very aggressively. Alfalfa, for example, might be cut 2,3, or 4 times in a growing season. For a bunch of reasons, you want to look at croplands much more frequently than we can look at with Landsat throughout a growing season. We basically call that phenology. It's the green up and brown down of vegetation during a single season. If you can characterize the phenology curve really well, then that tells you a lot about, first of all, what's growing there, because that's diagnostic. But it also tells you how well it is growing. That leads right into information that's relevant for agriculture management. Should I water? Should I fertilize? Should I plant a particular crop here this year?

HULT:

And this gets into something that's kind of interesting. This is happening, right? People are using remotely sensed data to track things like Normalized Difference Vegetation Index (NDVI). You know, crop health, phenology. We're using satellite data and we have been for a while. But typically, those sorts of daily repeat sets of data, they're very course resolution. This would be a publicly available data set where you can see smaller portions of a field. Is that kind of what we're talking about here?

MASEK:

Or just individual fields. Our daily record of something like NDVI, previously has come from MODIS or VIRRS or AVHRR, sort of in that one kilometer, half-kilometer spatial resolution. And so you can't see individual field real well, unless they're really big fields. But with Landsat resolution you can actually see fields and even within fields.

HULT:

How long has this been in the works? And maybe Brian, if you want to jump in here, what have you been working on to make this happen?

MASEK:

We started HLS back in 2014 before Sentinel-2 launched.

HULT:

Wow, that's really interesting. So you were actually thinking about this before Sentinel-2 went up. You thought, "well how can we work together with Landsat?"

MASEK:

Well, yes. For example, the Landsat Science Team had recommended trying to move toward daily observations from future missions. And we kind of looked at that and thought, well, that would be really nice but really expensive and we have one Landsat satellite that costs almost a billion dollars. It gets data every sixteen days. You launch sixteen of them, and you can do the math. So that seemed sort of prohibitive. If you combine the data from other systems that are already out there, from other countries.  Then you get that daily repeat cycle almost for free, relatively speaking. So that was the impetus. We knew that was going to be an important asset, even before Sentinel-2 launched.

HULT:

Brian, anything you want to jump in there and add on Harmonized Landsat and Sentinel?

FREITAG:

Jeff started developing this in 2014 as kind of a beta product. And it was a targeted area they were actually doing the production. So I think they were doing most production over North America. They were doing some other regions around the globe, smaller areas that were of interest to different product scientists. As part of the Satellite Needs Working Group, NASA goes back and they interview or survey a number of federal agencies for what they need to perform their operational tasks. And HLS was identified from the 2016 cycle as something that would be really useful, not only over North America and the United States, but as a global product. Starting in 2017-2018, NASA's perspective shifted from using this targeted domain that Jeff had used previously as kind of a beta release, they wanted to expand that out to global. The global production of this really kind of started in 2019. And that is where my group, the IMPACT Project, started assist in taking the algorithm and scaling it up.

HULT:

Ok, let's stick with you Brian. You're talking about the IMPACT Project. Tell us a little bit more about that. Is there any sort of simple way to explain what it is that you are doing when you talk about an algorithm that crunches this data together?

FREITAG:

The IMPACT Project is the Interagency Implementation in Advance Concepts Project. We're located at Marshall Space Flight Center. And our role is basically supporting NASA's Earth Science Data Systems. What they tasked us with doing was basically taking the existing algorithm that Jeff and Junchang Ju had developed at Goddard and move then that into the cloud, and then make it more cloud friendly so we kind of get an idea of what that looks like. Typically, when you have an algorithm and you're running that on a high-performance computing (system), it's a block algorithm. You kind of just shove it through, you get input/output, and you have status: success or fail. What we wanted to do was basically benchmark the different steps of the algorithm so then we can track the performance of the algorithm through time by separating out the different components. It is almost like you are taking the full cake and then breaking it down into individual ingredients. And then you're able to check each individual ingredient as you develop the cake so that in the end, the cake that you have from the new process matches the cake that you had before.

HULT:

And the cake we are talking about here is Harmonized Landsat and Sentinel.

FREITAG:

Correct. Yeah.

HULT:

Just so I can wrap my head around it. Harmonized Landsat and Sentinel, this will be a situation where you go in and you search for a particular spot on the globe, and you see Landsat images and Sentinel images that are available, but they're interchangeable?

FREITAG:

Right now we have two data products. We have the Landsat component and the Sentinel component. And we've actually done something where we're doing imagery through Worldview. NASA has a nice visualization platform where you can have these data layers that give you quick look, browse imagery to see what the scene may look like that you might be interested in. And so, we've got those two layers uploaded into NASA Worldview. You can look at that to kind of get an idea of what this looks like. But then you have the general full constellation of what the two data products look like combined together. And then as you focus on a particular area as a function of time, you will see the two- to four-day revisit period that we'll get over a particular location.

MASEK:

People often ask what we mean by harmonized. What we are talking about is to the user it doesn't matter to them which satellite the observation comes from. That they can use the observations interchangeably for the common bands, and increase the temporal density that way. That's the philosophic definition of harmonized, in this case.

HULT:

So, if I'm a person who uses a GIS software program, in the past, I would get one Landsat scene and one Sentinel scene, I'd throw them both into Arc, and they're both for the same location but they don't quite match up. Here, we're talking about a Landsat and Sentinel scene will match up as well as a Landsat and a Landsat or a Sentinel and a Sentinel. That's ultimately what you are going for here?

MASEK:

Exactly. And when we say "match up," we mean geometrically so the pixels overlay exactly-the tiles are in the same area-but also, what we think of as radiometrically. So the reflectance data is on the same scale. It's calibrated. We do a band pass adjustment, so it looks like they had the same spectral band pass, and we correct this angular difference that we talked about in the beginning.

HULT:

Let's talk about the why. What do you hope to see done with this HLS data? What does success look like five years from now, from your perspective? Are we talking daily deforestation alerts in the Amazon? Weekly algae bloom updates? Urban heat island tracking? What do you want to see? What's it going to look like to you five years from now to say, "this was worth the work, this is why we did this?"

MASEK:

Yeah. All of that. I think to see HLS or spin-offs from it incorporated into real land management activities. Like having the U.S. Department of Agriculture downloading this on a regular basis and doing crop forecasts and crop type analysis. To see the U.S. Forest Service, look at the daily scale of recovery from a forest fire. That to me would be the ultimate payback in the work. Because then people are really using it and using it to its fullest extent.

HULT:

Brian, anything you want to add there?

FEITAG:

Seeing the number of users increase. Seeing general access and the users coming in and learn how to access the data through cloud, learn how to shift their workloads a little bit. I think just getting that perspective and maybe shifting from, as you send out these data access surveys, seeing people shift from the https-based downloads to cloud-based analysis. Seeing that shift over the next five years is something that we really hope to pioneer with HLS.

HULT:

More people using the data, and more people using this more efficient method of doing their analysis.

MASEK:

I would also mention the international aspect, a little bit of a "kumbaya" moment. We have the opportunity to bring together data from multiple international programs. You're going to be seeing more and more of that, right, because international remote sensing programs are becoming more and more diverse, more and more robust. And there's that opportunity to combine data from these programs. To increase the frequency of observations or to do new types of observations. So, HLS is very much in that mode, and I think that's pretty exciting, too.

HULT:

Let's talk a little bit about Landsat 9. That's launching this fall. We have Sentinel-2C, which is set to go in 2023, as I understand it. What are the plans for incorporating these new sources into the HLS data set? And how big a difference will that make when that happens?

MASEK:

We are planning to incorporate both Landsat 9 and Sentinel-2C into the HLS. That will really bring us down to pretty much daily observations at that resolution.

HULT:

Can I just stop you there? I want to put a finer point on that. We are talking in the future about near daily observations-from civilian satellites, accessible to the general public, open data. Daily observations. That's a possibility here?

MASEK:

Exactly. It is worth remembering how different that is compared to what we had twenty years ago. I remember Landsat 7. I'm dating myself here. But I remember Landsat 7. The science goal was to get an observation every season. Now we are talking about an observation every day. So, it's definitely a new world.

HULT:

Let me ask. Do you guys have any closing thoughts? Anything in particular you would like to point out?

MASEK:

I hope people use HLS. I hope they get on LP DAAC. Just do the search through the data set and start to play with it and see what it's all about. What's out there today is a provisional product. This spring we're going to be re-processing what we started to process in October and release it as a final science quality product. And then for the remainder of 2021 we are going to be doing back processing, to basically create an entire archive going back to Landsat 8 launch in 2013. 

FREITAG:

When we go and start to do the re-processing, we are going to be looking at something like a 4 petabyte data set. So getting spun up on cloud based access and familiarizing yourself with the resources that LP DAAC makes available will really help once that full archive becomes available early next year. Doing your time series analysis, which is the bread and butter of HLS data product, is really only going to be possible with that cloud-based analysis. Using this intermediate time to get yourself familiar with cloud-based solutions, I think, is going to be extremely critical.

HULT:

So get your foot in the door now and you will be in a much better position going on down the road.

FREITAG:

Exactly.

HULT:

We've been talking with Jeff Masek and Brian Freitag about Harmonized Landsat Sentinel data, which will soon allow for more observations of the Earth's surface for researchers and scientists around the world to use. Jeff, Brian, thank you for joining us.

MASEK:

Thanks so much.

FREITAG:

Thanks a lot, John.

HULT:

Be sure to watch for our next episode of Eyes on Earth. You can follow us through Apple podcasts or Google podcasts, and all of our shows are available at usgs.gov/eros. That's u-s-g-s-dot-gov slash EROS. This podcast is a product of the U.S. Geological Survey, Department of Interior.