Eyes on Earth Episode 134 – Data Accuracy: The Calibration and Validation of Landsat

TOM ADAMSON:
Hello everyone, and welcome to another episode of Eyes on Earth, a podcast produced at the USGS EROS Center. Our podcast focuses on our ever changing planet and on the people here at EROS and across the globe, who use remote sensing to monitor and study the health of Earth. My name is Tom Adamson.

Landsat provides us with pretty pictures, but besides that, it's the incredibly accurate scientific measurements that people rely on for a multitude of applications. Landsat is the gold standard for Earth observations. That means the data is recognized as trustworthy. The USGS EROS Center has been collecting and archiving Landsat data since 1972, but we have also been carefully calibrating that data since then. In this episode, we talk to Cody Anderson, the project manager for the EROS Calibration/Validation Center of Excellence. He talks about the numerous activities that go into calibrating Landsat data to make sure it is accurate and can be trusted.

Cody, what's your job title here?

CODY ANDERSON:
So my job title is the EROS Cal/Val Center of Excellence project manager. So watch over everything Landsat, Cal/Val, as well as JACIE commercial data quality activities. I've been at EROS for about seven and a half years. I've been on the government side for about six and a half years of that.

ADAMSON:
What kind of degree do you need to be a Cal/Val manager?

ANDERSON:
Well, I have degrees in electrical engineering and mathematics. We do have lots of physics, science degrees that work on the team. Mathematics, data science type folks, software engineers, have a couple of those. So as long as you really like fine details and working with numbers, you can do Cal/Val.

ADAMSON:
Where did you go to college?

ANDERSON:
South Dakota State University in Brookings, South Dakota.

ADAMSON:
You're local, all right. That's pretty cool. We were talking about the term Cal/Val,
which is just short for--

ANDERSON:
Calibration and validation. The definition of those are always a little fluid, and they bounce around a little bit depending on who you ask. Calibration is trying to make things accurate or true. And validation is trying to independently verify that what you did for calibration worked. So trying to back up your claims of truthfulness of the data. Want to make sure that users can trust the data. We don't hide anything. We don't hold anything back. We definitely try to err on the side of over informing our users. And usually the response we get is, I'm glad you're doing that so I can ignore it. And that's a good, good spot for, I think, us to be in.

ADAMSON:
Why do Landsat satellites need calibration and validation?

ANDERSON:
Oh, that's a good question. I would say any time you want to use some type of data, some type of acquisition, some type of measurement, some type of image for scientific purposes, you have to calibrate it. If you and your friend each take a picture of the same thing, and you look at those pictures side by side, they'll be different. They'll be different for many different reasons, right? The angle of what you were taking a picture of will change a little bit between them. If you're standing next to each other or if you're even farther away. The type of cameras that you both have will be different. Different doesn't always mean right or wrong. It just means what you actually acquired was different than what the other person acquired. It gets difficult, and where calibration and validation comes in is if you want to compare those two images with each other in a scientific meaning. You want to make sure that the representation of those two different images are actually signifying the same scientific quantities. So calibration is all about the quantities of the data that you're trying to represent. Correcting for or accounting for those view angle differences, if that's what it was. Time differences, those software differences. It's trying to remove all of those things and getting back to the physical unit that you were trying to measure.

ADAMSON:
Okay. If you and I go down to Falls Park in downtown Sioux Falls and we each take a cell phone picture of the falls, and I'm standing over here, you're standing over there. They're both really good pictures, and we both recognize it as Falls Park, but they're going to be a little bit different. Now, that's not that big a deal, right, if we're just taking photos for fun?

ANDERSON:
Yeah, if you just want to put them on the wall.

ADAMSON:
Yeah.

ANDERSON:
Both of them could be pretty to look at. If you're trying to use those two images to estimate the amount of water coming over the falls or something, which if they're taken from different angles, you could do some of that.

ADAMSON:
Or a week apart or a month apart, something like that. What if you want to compare, yeah, and see what changed.

ANDERSON:
Yeah. And it'd be different how close you were to those falls, right, if you're closer to them, the water takes up more of the image, you know. Where if you're farther away, it looks like they're a long ways away. So you need to kind of correct for those distances, you know, and how much water is actually physically present in there.

ADAMSON:
Yeah, we're not just taking photos for fun.

ANDERSON:
Well, it is fun. But, yeah, we try to have a scientific purpose behind them.

ADAMSON:
And I've heard people say this around here that Landsat isn't just pretty pictures. It's scientific measurements.

ANDERSON:
Scientifically pretty pictures.

ADAMSON:
Okay. There's two general areas of Cal/Val, calibration/validation: radiometric and geometric. Let's tackle geometric first. What's that one all about?

ANDESON:
So when you're looking at a digital image. All that imagery is composed of squares, or pixels, in the image, right. So that's where you get all the different colors, the different shapes, and everything is built up of these different squares. So for Landsat, each square is 30 meters. But geometric calibration is trying to put those pixels, or those squares, in the right location. So we can all think of road intersections, right, where two roads meet. That makes a nice plus sign, a nice cross right between them. We want that intersection, or that cross, to be at the right location, the right latitude and longitude, when you click on that pixel within the imagery.

ADAMSON:
Okay, and throughout all of the Landsat images throughout the archive, those pixels are in the same spot?

ANDERSON:
Yep. Every single Landsat image that we acquire, whether it's from Landsat 1 to Landsat 9 and also going into the future, that road intersection should be on the same spot in every single image as soon as you open them up.

ADAMSON:
That's the geometric accuracy.

ANDERSON:
That is the geometric accuracy--and stability.

ADAMSON:
And stability! Yes, over time, over the decades, it has stayed in that same spot. It's frankly pretty amazing that we can do that. But it's not just that. We've got radiometric calibration as well. What's radiometry?

ANDERSON:
So if geometry is the location of those squares, or those pixels, radiometry is the brightness, or the value, of those pixels. Right, so when you're looking at an image, you can tell if an object is blue, green, red, or a combination of those colors, right. If it's brighter or if it's darker. That is the radiometry. The brightness of the data. And when we're talking about Landsat, we're talking about scientific accuracy. And it's to make sure that the value that you click on that one, it tells you this is the amount of reflectance, or this is the amount of radiance, that that number is scientifically correct.

ADAMSON:
And we're looking at brightness, or radiance, in each of those pixels in several different wavelengths of light.

ANDERSON:
Yep.

ADAMSON:
Can you explain how that works a little bit more, too?

ANDERSON:
You know, when you look at the sun, it looks kind of yellow or orangish? When astronauts look at the sun above the atmosphere, the sun is white. So why is the sky blue? It's because of that white light from the sun gets absorbed and scattered by the atmosphere, and then redirected in a bunch of different directions. And the color of the light that gets absorbed is the blue light. So essentially, the white light from the sun gets pulled apart by the atmosphere and scattered in a bunch of different directions. That makes the whole sky blue, but it makes the sun look more yellowish or a little less blue than that. So that is the different colors of light. So as the blue gets scattered out in the atmosphere, the rest of the colors usually get down to the surface and then bounce off the surface and come back to the sensor. So you can see those different wavelengths, those different colors. And then with Landsat, you can see beyond what the naked eye can see. So the part of the electromagnetic spectrum that we refer to, what we can see, is the visible portion of it. But yeah, we can see beyond that into the shortwave infrared and even into the thermal infrared, which is things that you can't detect with your naked eye.

ADAMSON:
That's what Landsat can see though, or detect for us.

ANDERSON:
Yeah.

ADAMSON:
I feel like when we're talking about the electromagnetic spectrum and how light behaves, it's something I should have remembered from eighth grade science.

ANDERSON:
Yeah. Whenever you did that example, or that experiment, when you were a kid, where you hold a prism up into the light and it's white light coming in, and then it separates it into the nice rainbow coming out of it. Yep. That's what we're talking about.

ADAMSON:
That's what we're taking advantage of to be able to take these measurements. Talk about how others rely on Landsat's accuracy for their work.

ANDERSON:
I don't think we invented it, but I do like to use it a lot. People say that Landsat is the gold standard. And that is really because how trustworthy the data is, how repeatable the data is, and how long we've been doing it. Most people know, or maybe most people don't know, but we definitely like to advertise the fact that we have a 50-year, now it's a 50-plus-year archive, going back continuously collected of the Earth's surface that has been calibrated the entire time. Cross calibrated. So it should be consistent from 1972 all the way up to today. People can trust that. And the fidelity of the data that we have today is also very useful to folks.

ADAMSON:
Do you want to talk about the JACIE meeting that's coming up? What do those letters stand for?

ANDERSON:
Yeah. So JACIE--I mentioned part of the EROS Cal/Val Center of Excellence; Landsat Cal/Val is one piece and JACIE is another one of the big pieces. That stands for the Joint Agency Commercial Imagery Evaluation.

ADAMSON:
Okay.

ANDERSON:
The USGS manages it and really leads a lot of it. But the partners are NASA, NOAA, USDA, NGA, and the NRO. So we have kind of the U.S. civil space side of things, the U.S. intel side of things, and then the USDA is a very big user of satellite remote sensing data. And so these agencies get together, we have an annual workshop, but we also meet with them several other times throughout the year. And commercial imagery is the focus of that group. So commercial imagery, there's more and more data providers in the commercial realm. They usually offer data at a higher resolution than Landsat or other global survey missions. They also offer more temporal resolution. So they're taking images more often. Sometimes these are true global survey images taken every day. Or they can be pointable, so there have more nimble or agile spacecraft to be able to point at different targets on the Earth and more rapidly to get you more acquisitions. But it's a commercial mission. And so that gets into how much time can they spend on calibration and validation. How much work can they put into that versus how fast they have to produce a camera, produce a sensor, get it up on orbit, start taking imagery so they can start selling imagery. When you do it that quickly, you don't necessarily know what the output or what the products of it should be, but that's where they are reliant on Landsat to cross calibrate and to point to their data. So JACIE is all interested in commercial data quality, evaluating these different commercial providers, and looking at how trustworthy or how good their data is. If they give you the same answer as Landsat, a lot of people out there would say, all right, I'm doing pretty good. I'm getting the same or close to the same answer as Landsat. If they're wildly different than Landsat, you might need to do a little bit more work to figure out why that is.

ADAMSON:
And this is your opportunity to communicate what you're doing, too, and being--you're being transparent about the kind of work that you do.

ANDERSON:
Yep, from the government side of things, from Landsat side of things, we're all open. We want to be trustworthy. And so we try to be as open, as forthright, as trustworthy as possible, and communicate that to a large audience. Communicate that to the commercial data providers. And then if they can show that they're close to or giving the same answer as Landsat, they can kind of use our trustworthiness and just say, we got the same answers as these guys. We don't have to describe everything that we did, right, kind of the secret sauce. But we got the same answer as Landsat. So trust us because you can trust them.

ADAMSON:
Okay. Landsats 8 and 9 are in orbit right now. And there's some calibration you can do with the actual data. Is there calibration that you also do before launch?

ANDERSON:
So as I said, we're always interested in truth. And one of the ways to try to get to that truth is you have to know what you're taking a picture of. And that is more difficult when you're on orbit and you're looking at the Earth. That's what we want to do. We want to observe the Earth. But you're not sure what you're looking at all the time. When you have the camera or the instrument on the ground in the lab, you can put a target in front of that, that you do know what the answer is. You know what the sensor is supposed to be giving you as an output. And one of the things that we use for that, they're called, spectralon panels or reference panels or just white panels. And so when you look at them, they're white. And anything that is white means it's giving us a uniform response. The same answer through all the different pieces of the electromagnetic spectrum. So on the blue channels, the green channels, the red channels out into the shortwave infrared channels. So they give us the same answer in every single channel. If you put one of these panels in front of the camera, you should get the same answer for all those different channels out. That's one of the tests that we do pre-launch before we put it on orbit. That is what we would call absolute calibration. There's lots of different types of calibration. So absolute calibration is, I want the sensor to give me the answer I expect it to give me. The same radiance unit, the same reflectance units. Then there's alignment to make sure, this gets into geometry, where I'm looking at wants to be where I am looking at. So you can put targets in front of the camera there that have different, kind of, locations on them, and you can actually figure out where you're looking at or if you're looking at a slight angle at something and we can correct for those. There's calibrations on what you would call the radiance response curve. And so if you put in a low signal level, you should get a low answer out, versus a high signal level, then you should get a very bright signal out of it. There's different things on how sharp the imagery is to make sure that it's not blurry. So you calibrate the spatial aspect of it. And if you put in a really sharp, fine line, white on one side, black on another side, that's the image that you should get out. You shouldn't get kind of a blurred line in between these two areas.

ADAMSON:
Kind of like being at the eye doctor.

ANDERSON:
A little bit. Yeah, that is essentially what we're doing. We're trying to understand what the visual response of the camera is just as an eye doctor understands the visual response of your eyes.

ADAMSON:
You're taking these measurements in the lab in a controlled setting before it gets up there into space, so that you know exactly what to expect.

ANDERSON:
Before you put it on a rocket and you launch into space, and it shakes it all up,--

ADAMSON:
--and then you have something to compare it to. When you start getting real data from orbit, you can compare it to that data that you took on the ground or in the lab?

ANDERSON:
Yep. Yeah, and that's, gets into the validation part of Cal/Val, where we take essentially those same white reflectance, reference panels and we'll take them out into the field. And so we'll have different sets of measurement sensors, that we can measure those same white panels out in the field. And then we'll go measure a large, field area. It might be vegetation, it might be sand or whatever it is. But we can take that measurement of the white panel and say, all right, this is giving us the same answer that we expected it to be in the lab. Then we can transfer that through the same sensors, the same measurement devices we just used to these large areas on the Earth's surface. And we can do that at the same time that the satellite is passing over that target on the Earth. And we can say, all right, the white panel gave us this answer, this large area gave us this answer, and the satellite gave us this answer. And they should all kind of be able to line up and give us the same answer through all those different, different measurements.

ADAMSON:
That's really cool. What are some of the favorite out in the field locations to go to do this?

ANDERSON:
Oh, this is getting a little debatable, I would say. So for, on the calibration side of things, we always like really boring targets, targets that are the same covering a large area, right? Large homogeneous regions. Also locations that don't change over time. We want the same answer time after time after time. Those are not the targets that scientists like to look at.

ADAMSON:
They're not so interesting.

ANDERSON:
Yeah, they're not--scientists are interested in actually detecting changes--

ADAMSON:
Yeah.

ANDERSON:
--of areas. So for calibration, we like large homogeneous deserts. They're really bright. And they give us a good signal to measure. They don't change over time. They cover a large area. So you get a lot of different samples of the same expected answer. But scientists want to look at vegetation, and we know the vegetation gets greener when it's wet, it starts to turn brown when it's dry. And that is what scientists are interested in. So we've been expanding the locations that we do these measurements at over the last couple of years. We've been doing more measurements over vegetative targets. Just out back of the EROS Center we actually have a vegetative target range that every time Landsat passes over, we send a team out there and they make measurements of the field.

ADAMSON:
When you say every single time, like, really, every single time?

ANDERSON:
Every eight days.

ADAMSON:
Every eight days is what that comes down to then. Wow. Okay, cool. Talk about what I understand to be known as the test sites catalog. And this might be similar to that; there are some test sites, and I think these are around the world that are used for this type of measurement, too.

ANDERSON:
Exactly. Those are the locations that we do these type of measurements at. So most of those, well, it's divided into radiometric, geometric, and spatial test sites. So the radiometric are those large homogeneous deserts. Typically there are some vegetative targets in there, but those are sites that, usually, there's been a field team out there and made measurements of them. So we know what the answer is. So anytime you fly over it, you say, all right, I know what the answer should be. And you can look at the imagery from your sensor and say, well, did it agree or did not agree? And then there's the geometric ones. So those are usually include some features on the ground that you can actually detect in the imagery and say, well, is this feature in the right location of where I know it to be? Okay. So we might have had survey teams out there making measurements of these features in the exact location of them. And then spatial gets into how blurry the image is. So there are some manmade targets out there where they've kind of put in these white and black checkerboards. You know, large black and white checkerboards.

ADAMSON:
30-meter checkerboards? I mean, they have to be pretty big, don't they?

ANDERSON:
Some of them are about that big. In order to do Landsat, you need a little bit larger than 30 meters, so there are none for Landsat resolution yet. These are some of the more commercial high-resolution things. If you want to use Landsat, we use bridge targets for those, we call them. So these are long, straight bridges, usually over a body of water. So the body of water gives you this uniform black background. And then the road is this nice straight linear feature, usually of concrete, which is a lot brighter and gives you a higher reflectance off of it. And then we can actually look at how blurry is that road. Right. Is it just a straight line within it or is it a little bit blurred. Also is it straight. Right, if it's a little wobble in it then there's an issue with your imagery.

ADAMSON:
Yeah. You're pretty certain that a bridge is going to stay in the same spot, unless someone decides to add a lane to that highway on the bridge or something like that.

ANDERSON:
Yeah, we try to monitor them if there's any construction activities going on.

ADAMSON:
Is there any use of AI, artificial intelligence, in the Cal/Val work?

ANDERSON:
Well, that's a good question. The whole world is very interested in AI, or artificial intelligence, now.

ADAMSON:
That's why we're asking.

ANDERSON:
Yep. Some of that, you know, it seems kind of contradictory or to go against calibration and validation. We're concerned about the truth or concerned about trustworthiness. And one of the big questions out there about AI is, well, what is it doing, and can you trust it? So how Cal/Val looks at using it is we crunch through a lot of data.

00:21:19.378 --> 00:21:22.481
AI is great at crunching through data, but we use it in more of a kind of a tipping, and a cueing, aspect, or a way. We'll use AI on something to sort of say, hey, we think there's something here to look at. You know, there's a correlation here, there's a relationship here. And then we will go and investigate that. So we don't kind of just take the AI and run with it. We say, oh, yeah, we maybe overlooked something there, and AI is able to point that out to us, and then we'll go investigate it, understand why that's there, and then explain why it's there, and then we may correct for it after that. So AI is very useful to point us in the right direction.

ADAMSON:
Okay, it can help you find some of these subtle differences that might be going on in the sensor?

ANDERSON:
Yeah. So as part of calibration and making sure we're getting to the truth or getting to the right answer is also making sure we get the right answer time after time after time. So we're concerned about stability and nonvariability of things. So whenever our answer changes a little bit, we need to understand why that changed. So AI can help us point to those times when it did change. And then, so it tells us, all right, go look at this time period or the specific place on the earth. And then we try to understand what was going on there and what caused this change. Was it a real change? Was it something on the surface of the Earth that changed, so we got a different response to it? Or was it something in the actual sensor or the camera itself that changed that we should not be seeing in the data. Okay. It's another tool to help you out anyway. Now, researchers using Landsat also have that kind of trust. That's our entire goal. The Landsat Science Team, right, is kind of a group that the USGS tries to work with, the kind of leading experts in remote sensing science applications. We always like questions. We like people to really dig into what we're doing. But we also liked a lot of people just say, you know, that was pretty boring, but we trust you guys. We're glad someone is looking at these things so we don't have to. And when the Landsat Science Team says how much they trust the data, that is really, I guess, the feedback we're looking for, right. They're just saying, yep, you guys did a good job. We trust you. We never have to think about it.

ADAMSON:
That's the goal. They don't have to think about-- This is--you spend your life working on something that you hope no one else has to think about.

ANDERSON:
Exactly. If we ever hear someone talking about Cal/Val, that's really not a good thing. It means that they're probably questioning the trustworthiness of the data. We always want to stay out of the conversation, and just, they assume that the data is good and then they can focus on their science application.

ADAMSON:
This podcast is the exception. We're just trying to highlight the work you're doing. We're not questioning.

ANDERSON:
I appreciate that.

ADAMSON:
I'd like to thank Cody Anderson for joining us for this episode of Eyes on Earth. Now we know how Landsat got to be such a trusted source for accurate Earth data. Lots of careful work on calibration and validation. And thank you, listeners. Check out our social media accounts to watch for all future episodes. You can also subscribe to us on Apple and YouTube podcasts.

VARIOUS VOICES:
This podcast, this podcast, this podcast, this podcast, this podcast is a product of the U.S. Geological Survey Department of Interior.