Eyes on Earth Episode 31 – Landsat 7 Flight Operations

Steve Young: 

Hello, everyone. Welcome to another episode of Eyes on Earth. Our podcast focuses on our ever-changing planet and on the people here at EROS and across the globe who use remote sensing to monitor and study the health of earth. I'm your host, Steve Young. Just recently, in mid-July, the flight operations team charged with keeping the Landsat 7 satellite running smoothly achieved a major milestone. They have gone 8 straight years now without an operator error. Think about that. When you are operating a machine that costs hundreds of millions of dollars and flies at more than 17,000 miles per hour, avoiding any kind or operator error for 8 years is a reason to celebrate. Here to discuss all of that, is Guy Thayer with the Aerospace Corporation. Guy is the Landsat 7 Systems Flight Manager, who with his flight operations team keeps track of Landsat 7 for the US Geological Survey. Thanks for joining us, Guy.

Guy Thayer: 

Nice to be here!

YOUNG: 

Let's start off by talking about 8 straight years of no operator errors flying Landsat 7. Tell us why that is a big deal. 

THAYER: 

Well, it's a really big deal because the operator errors are pretty easy to make. So having this amount of time go by and this many activities happen and not have an error is a truly large achievement. 

YOUNG: 

I have this image of people sitting at a console moving around joy sticks to maneuver this multimillion-dollar machine through space. Is that how it works?

THAYER: 

No, (laughter). It's mostly done through files and commands that we send. But those files have to be created and we're talking about errors, that's a place we could easily make an error. So, no, not a joy stick. It is funny that you mention that, because if you look at some of the pictures from the MOC, you'll see one sitting on the desk that we use for tours. Just for laughs.

YOUNG: 

When you talk about files, do you send up hundreds or thousands of commands every day?

THAYER: 

The long and short of it is, we set up two different loads, we call them, command loads every day, 365 days a year. One tells the satellite what to do and when to do it. And the other one is the ephemerous load that tells it where it's going to be during that period of time. The operators have to go in and create that load every morning and then throughout the day they observe the satellite and make sure that it is operating from that command load. Making sure it is doing what it's supposed to be doing. 

YOUNG: 

Does operating Landsat mean simply keeping it in its orbit, or what else do these operators have to do to keep it performing the way it should?

THAYER: 

Obviously, keeping it in its orbit is a big part of the job. Typically, we would have to send commands to raise the orbit of the spacecraft as it encounters drag, but lately we haven't had to do very much of that in the last couple of years. The bigger work is the everyday work of creating those loads and putting them up on the spacecraft. The main part of the job is reviewing the information we get from the spacecraft and making sure that it is healthy. 

YOUNG: 

By healthy, you mean for example....give us some examples of what kind of health concerns a satellite would have. 

THAYER: 

Power is a big deal to us, obviously. We have to generate and store power so we can operate the spacecraft. So, we need to closely monitor the output of the solar array, and the health of the batteries, that we can monitor by watching the temperature and the pressure of the batteries. We look at the attitude of the spacecraft to make sure that it is working within specifications. And we look at, for example, temperatures of gyros and how much power they are drawing to make sure that they are working properly. Unlike your car for instance, you can hear squeals and knocks and what have you, right? But with the spacecraft, we have to ascertain what's going on by looking at telemetry, by looking at data points.

YOUNG: 

How many people does it take to operate a satellite? And do they all do the same thing or do they have different roles?

THAYER: 

Actually, we have several different roles throughout the Mission Operations center. Up in the front we have our flight dynamicist. He figures out where we've been. Calculates where we've been based on the information he receives from the down link sites throughout the world. He builds a file which tells us where we're going to be in the future. Which is very important so that we know when to acquire the satellite as it is coming within range of the satellite dish on earth and it also tells the satellite when and where to point the instrument and where to turn the down link capability on, the radios on to beam that information back down to earth. He builds that file and gives it to the daily planner. He needs to build the file that tells the instrument when to turn on and off and other house keeping things. Then we have 2 people, which are the command controller and the spacecraft analyst, the CC and the SA. That's where the rubber meets the road so to speak. Those are the people who actually initiate those command loads, send them up and then they watch for 8 hours a day. They are the people in charge of making sure that the spacecraft is safe by looking at the telemetry as it comes down in real time. Also, if we need to send other commands, those are the people who are again, actually touching the keyboard to send commands directly to the spacecraft. Beyond that, we have what are sometimes referred to as the offline engineers, but the spacecraft engineers in the back look at long term and short term trends and create the long term plan for this spacecraft. We've also got a couple of layers of management as well as we have some IT folks because after all this is a bunch of computers talking to a computer in space, and so we have 2 IT specialists and then of course, me. All told, we have about 14 people. We will have anywhere from 2 to 4 people operating the spacecraft every day. Everybody else has been working from home. We do have capability that was developed by this team. We call it "lights out capability". The acronym is called LOOFA and it monitors the spacecraft when we go home in the afternoon until we get there in the morning.

YOUNG: 

Tell me, in operating a satellite, what are the major challenges you face in doing that with Landsat 7? What kind of challenges come up that really acquire your attention?

THAYER: 

It is an older satellite. Most things that break tend to break early. We've kind of settled into a rhythm. So, what are the actual challenges? Challenges, the regular challenges, that every satellite operator faces, I don't want to call it monotony, but it is maintaining focus on keeping the thing running, doing many similar things every day. So, complacency is actually one of the challenges that you have. We've been very, very lucky, knocking on wood, that we haven't had very many major challenges in the last several years. I mean the solar array slowly degrades. And in our case our angle to the sun is slipping based on the fact we are no longer doing inclination maneuvers to maintain our inclination because we don't have enough fuel to do so. So, that is a challenge, trying to figure out going forward if we're going to be able to generate enough power. So, other than that, it's just making sure nothing hits the spacecraft and staying on top of all those telemetry points we talked about earlier.

YOUNG: 

How often do you have to worry about something hitting the spacecraft?

THAYER: 

Not very often right now. Remember, I mentioned a little while ago about the fact that the drag environment is fairly benign right now? As it happens, when the drag environment, and it is cyclical, when it is higher, things are falling through our orbit on a more regular basis. We've only had to perform a few risk mitigation maneuvers on Landsat 7. And we haven't performed one in about a year and a half. 

YOUNG: 

If you get an alarm, and a light goes off or something that would suggest that there's an issue going on, how long do you have to figure that out before it becomes a major problem?

THAYER: 

That is a big "it depends" question. What the issue is will determine how long we have to react. For example, let's say the satellite went into safe hold, which, knocking on wood again, we've never done. It is safe, and it is stable. We can sit there almost indefinitely and not have to work too quickly which would make us happy because we don't like to do anything quickly. But, if there are other things for example if the solar array were caught in the wrong angle and were stuck, we would have to act very quickly within a couple of hours to start getting that thing fixed. 

YOUNG: 

You've been fairly fortunate it sounds like over the last 8 years. Does that mean you don't have moments that get your blood pumping?

THAYER: 

We absolutely do. In the last 8 years, we've had a couple of different things happen. For example, we had a clock generator that is an esoteric piece of equipment that meters the clock output. Again, it's an esoteric piece of equipment, we never think of it, it's buried in there, it's not something that we can command. One day, it just kind of switched from the primary piece of equipment to the redundant piece of equipment. For the life of us we never could figure out, even though we had some subject matter experts dive into it, we never could quite figure out why. And that is jarring, and when something like that happens, and you don't know why it happened, that gets your blood going because then you start worrying about the cascading effect of "What if this? What if that?" If you don't know what caused something to happen, it makes you very nervous about what's happening on the spacecraft. But, other than that, we've been pretty darn lucky. There's a couple of what we'll call fairly routine anomalies. Things that we've seen before that we know how to fix, but other than that we've been very lucky in the last 8 years.

YOUNG: 

You've got enough experience on that team that it sounds like to me it doesn't very often happen that an anomaly occurs that baffles you. You've got enough experience to know what the problems are 99% of the time, 100% of the time?

THAYER: 

Darn near all the time. You mention experience and that is one of the reasons why we've been able to have this 8-year streak happen. Three of the people that we have on the team and have been here since prior to launch. Who have been here like 4 years before launch, our lead engineer. And then we hired 4 others in the first year of operation. So, we've got 7 people who have been here for 20 years, which is just amazing. There's one story I like to tell that tells you a lot about the team and the experience. We had an anomaly in the middle of the afternoon on a weekend. The lead engineer got the call and he's on the softball field, no soccer field watching his daughter play They read off to him the mnemonic. They said "Hey, we're getting this crazy thing and we don't know what it is." He said, "I'll be right there." Then, like in about half an hour, he was in the office, he walked right to his cubicle, pulled a piece of paper off the wall and pointed to it and said, "This is what it is." And that was, I believe that was the clock gen. It takes quite a bit to rattle this crew.

YOUNG: 

And if that gentleman hadn't been around, hadn't been available, you might of panicked?

THAYER: 

That's right. And even more amazing is he hadn't opened, he hadn't looked at that piece of paper since launch. But he knew that someday, he may need it and it was right at hand. 

YOUNG: 

Tell me Guy, just briefly, the people that work out of flight operations team. I take it they're not like me, they are not former journalists, they are not math majors, I mean, what's the educational background of somebody that is going to operate satellites? 

THAYER: 

Actually, we have had math majors. Math majors and physicists make fine engineers. But most of them are astro, aero engineers. A lot of people in the space business tend to be electrical engineers. That's generally the makeup. Aero, astro engineers and electrical engineers. I mean in my experience, folks that I have worked with over the years.

YOUNG: 

Let me ask you finally, have you ever heard of operator errors that have occurred with other satellite systems.

THAYER: 

Well, let me tell you about one. It was easy for me to remember because it was my first day representing USGS with Landsat 5 actually. It was literally my first day on the job. The command load for that was a 2-part command load. You send up what you want the spacecraft to do and then you would have to send up another file that told the spacecraft when to perform those things. The dilemma was the operator somehow, mistakenly sent up 2 of the command file without the time stamp file. Now, normally that wouldn't cause any problem at all, right? It would be "gobbly-gook", it wouldn't make any sense to the spacecraft. But, it just so happens that the satellite gods were not smiling on us that day and one of the commands that corelated with the command load, the satellite read it as an actual command. The next time the spacecraft was supposed to come into contact with the ground system, we didn't get contact. If you want to know what gets your blood pumping, that's what gets your going. When it comes over the hill and you don't see it. That operator error actually put the spacecraft into a tumble. We figured it out quickly. Thank goodness. We got things right within a day but that's how easy it can go downhill quick.

YOUNG: 

I would imagine that if you hadn't figured it out quickly and a satellite would have been lost, you might have gone into the restaurant business.

THAYER: 

That first day might have been my last day. Thankfully it worked out. That was in 2011, so it's worked out ok so far.

YOUNG: 

We've been talking to Guy Thayer the Flight Systems Manager for Landsat 7 about his team's recent achievement of going 8 straight years without an operator error when it comes to flying that important satellite system.

THAYER: 

Thank you very, much and we are very, very proud of the people we've got.

YOUNG: 

We hope you come back for the next episode of Eyes on Earth. This podcast is a product of the U.S. Geological Survey, Department of the Interior. Thanks for joining us.