518: Nothing We Can Do About Frogs
Transcript from 518: Nothing We Can Do About Frogs with James Cameron, Christopher White, and Elecia White.
EW (00:00:06):
Welcome to Embedded. I am Elecia White, alongside Christopher White. Our guest this week is returning guest, James Cameron, who we talk to about One Laptop Per Child and Forth. This time we are going to talk about telescopes, and whatever that bird in the background is.
JC (00:00:27):
<laugh>
CW (00:00:29):
Hi, James and extra guest. <laugh> Welcome to the show again.
JC (00:00:34):
They are frogs. They just started. As soon as you started recording, they just started.
CW (00:00:41):
Nothing we can do about frogs.
JC (00:00:45):
They are about six meters away in a drain pipe. So yeah, I do not know how to fix that. We have never been able to fix frogs. They are in chorus now, so it is probably hormonal.
CW (00:00:58):
We are proceeding despite the frogs.
EW (00:01:01):
It is like we have got all the show titles all set up.
CW (00:01:02):
Yep.
EW (00:01:02):
James, could you tell us about yourself as if we met, I do not know, on the Embedded Slack channel?
JC (00:01:12):
Okay. Yep, sure. I am hard to describe, because I do not often describe myself. But I think the best one is scientist with a business degree. I had childhood exposure to music, physics, electronics, and computing. Science seemed to be the way to get things done. It seemed to be the way I operated myself. That is the way I always thought through problems.
(00:01:40):
When I was looking for something to do as a job, computer programming was the bee's knees. It was really well thought of and therefore well paid. And so that is what I started doing. So computer science is what I have focused on. But not with a qualification, because I had done all that by the time I was in middle school. The next thing to do was work out how to make money from it, and hence the business degree.
EW (00:02:18):
We want to ask you lightning round questions. You are familiar with the show. Are you ready?
JC (00:02:26):
Yep. Go ahead.
CW (00:02:26):
Are the frogs ready? I do not hear them, so I do not think they are ready. Have you seen a total eclipse?
JC (00:02:35):
Yes. Yes, I have. Of various bodies. Mostly an eclipse of the sun.
EW (00:02:43):
Have you seen the green flash?
JC (00:02:45):
No.
CW (00:02:47):
Have you seen the zodiacal light?
JC (00:02:50):
Yes. Yes, I have. But someone had to point it out to me. I was not actually trying to see it. It was just sort of, "Oh, look, there it is!" "What is that?" "Zodiacal light." "What is that? Oh, right. Okay."
EW (00:03:01):
Have you seen a platypus in real life?
JC (00:03:04):
Yes, I have. I was walking in bushland and came across a little rivulet of water. There was this weird thing swimming in it, and it just did not seem to swim right. So I focused on it visually for a while and tried to work out what it was. Eventually I was rewarded with a large nose, which was what is in the books. So, yes. It was surprising.
CW (00:03:32):
Have you ever touched a kangaroo?
JC (00:03:36):
When I was young and visiting an animal petting zoo, yes. And when I have had to remove one from the road after it is dead, yes. But other than that, no. You tend not to get close to them.
CW (00:03:54):
Yes, I have seen their upper bodies. They seem very-
EW (00:03:56):
Boxy.
CW (00:03:56):
Boxy.
EW (00:03:56):
Like, good at boxing.
JC (00:04:00):
They have a radius of comfort. The radius of comfort for a kangaroo here where we are, is about 50 meters. Which is to say if you are within 50 meters, they will move away.
EW (00:04:14):
Hmm. Okay.
JC (00:04:14):
Up at the observatory where I worked, the radius of comfort is about ten meters. That is unfortunate, because you get closer to them. If you are driving, it means that they are unlikely to jump away, and so they might jump into you, as you are driving. That is not nice. But yeah, they are not friendly little critters. They have got their own agenda.
(00:04:44):
And the males when they are wanting to mate are particularly irritating, because they are chasing after the females at 40 kilometers per hour as they go past. The female is running away, the male is chasing them. That seems to be their thing, what they like to do. And the males in that state are without thought and strategy. Well, they have thought and strategy, but they are concentrating on something else. And so-
CW (00:05:20):
It does not involve motor vehicles.
JC (00:05:22):
Yeah. They do not notice you. And that is on foot especially, where that is dangerous.
EW (00:05:28):
Favorite Australian animal?
JC (00:05:31):
At the moment it is echidna. We have numerous echidna around here. Very spiky ant eating type animal. They dig up bugs and ants. They make holes all over the place. Not as many holes as-
EW (00:05:49):
Echidnas also have cloacae.
JC (00:05:52):
Yeah. Well, I do not go into that detail.
EW (00:05:54):
Sorry.
JC (00:05:54):
But, yes.
CW (00:05:58):
It is a reference.
EW (00:05:58):
It is a reference. It will not make sense to anybody, but Christopher.
CW (00:06:02):
Moving on from cloacae.
JC (00:06:04):
If you get close to an echidna, like within one meter, that is when they can actually see and sense you. So their range of comfort is one meter. So you get right up to echidna before they even do anything about it.
(00:06:16):
Their reaction is to try and dig a hole in the ground right where they are. Or if they know they cannot dig there, then run as fast as they can for 95 centimeters and then dig a hole. They then have all their spikes pointing up and their soft body parts in the hole. They do not actually hide in the hole. They sort of use the hole to protect half of themselves. They are weird.
(00:06:44):
And when they are in their mating frenzy, they are a train of echidnas traveling across the landscape, one after the other. They are so funny.
CW (00:06:59):
I feel like we should wrap up lighting around, but we have many more.
JC (00:07:01):
Sorry.
EW (00:07:03):
Do the next one.
CW (00:07:05):
There are three.
EW (00:07:06):
The mount question.
CW (00:07:07):
Oh, okay. Equatorial, altazimuthal or fixed?
JC (00:07:13):
I do not mind. I mean, you can mix and match. Where I worked, the building was azimuthal and the telescope was equatorial.
CW (00:07:22):
Okay.
JC (00:07:22):
Exactly. It causes interesting results. But they are just different ways of compensating for the rotation of things. I prefer my chairs to be azimuth.
CW (00:07:40):
Yes.
JC (00:07:42):
So I do not slide off them.
EW (00:07:46):
So tell me about this telescope. Forget lightning round. We want a real answer this time.
CW (00:07:52):
Those were real answers.
EW (00:07:53):
They were.
JC (00:07:55):
Sure. So one day I was looking for something to do, and heard that the team up at the telescope needed someone to do things for them. So I popped up there and they said, "Come join us."
(00:08:09):
The telescope that I am talking about is the Anglo-Australian Telescope at Coonabarabran, northwest of Sydney in Australia. It is the largest optical telescope in Australia, as far as I know.
(00:08:24):
It is not the largest telescope. There are radio telescopes that are much bigger. We use the whole of Australia when we link up those radio telescopes. It makes a very large aperture.
(00:08:36):
But it is the largest optical telescope. It is also quite old, being built in 1974 or so, and has been maintained since.
CW (00:08:48):
Hey!
JC (00:08:48):
Sorry?
CW (00:08:48):
I was also built in 1974, <laugh> so not feeling quite old yet.
JC (00:08:55):
My apologies. But you have got automatic systems-
CW (00:08:59):
That is true.
JC (00:08:59):
Which regenerate your components. Whereas this telescope, we used humans to maintain the components there.
(00:09:08):
When I first encountered that telescope, I was a tourist and just visiting. See the visitors' center and climb the stairs into the viewing gallery. Gaze with awe at this gigantic thing. Then go away and go to the next tourist trap.
(00:09:31):
But when we moved up into this area, it is the white thing on the mountains over there. You can see it as you are driving along. It is a very recognizable white dome. The density of settlement out here is very low. The population density is very low.
(00:09:52):
So anything mounted on a mountain is going to look very strange. And it does look very strange. It is out of place in that respect. We are all used to it now, but it is weird to see a building on top of a mountain. So it attracts all sorts of people.
CW (00:10:14):
How big is the scope itself? How many meters is the aperture?
JC (00:10:19):
I think it is 2.7 meters or something. Something like that. It is a very big mirror. It is one of those telescopes where it has a primary mirror facing the target, and the light is then concentrated into a secondary mirror, which then reflects the light back down into a hole in the primary mirror. Then you put your instrument down the bottom below the primary mirror. That is one of the modes it has of operating.
(00:10:48):
Another mode is that you put the instrument up the top of the telescope facing downwards. Then the starlight is only reflected off the primary mirror, before it reaches the instrument. That gives it a much wider field of view. Which is great if you are looking at galaxies, lots of them at once. But terrible if you want to look at a single star.
(00:11:08):
So the first mode, where it is using a primary and secondary mirror, and the instrument is at the bottom, that is when it is being used to look at a single target, like a galaxy or a star.
EW (00:11:22):
I want to pull a couple of stats from Wikipedia, "3.9 meter equatorial mounted telescope".
JC (00:11:31):
That is it. 3.9. Sorry.
EW (00:11:34):
One of the "most scientifically productive four meter class optical telescopes in the world."
JC (00:11:38):
Yep.
EW (00:11:38):
It has a "seven story circular concrete building, topped with a 36 meter rotating steel dome."
CW (00:11:51):
It is huge!
EW (00:11:54):
"The top of the dome is 50 meters above ground level."
JC (00:11:57):
Yep.
EW (00:11:58):
And the "moving mass of the dome is 260 tonnes."
CW (00:12:07):
That is incredible. I would- If you asked me how much- Well, the dome itself probably weighs a lot, because it is a giant dome. But. Yeah, a four-meter telescope is big, but I guess I just do not have any reference for this.
(00:12:20):
I work with small amateur telescopes and like, "Oh, I have had an eight inch telescope. Four meters, that is bigger, much bigger. But it is not 260 tonnes of support bigger. But I guess it is."
EW (00:12:30):
It really is.
JC (00:12:33):
Well, firstly, imagine taking your telescope and placing a dome around it. How big would that dome be?
CW (00:12:37):
Yeah, that is true. Several feet, yes.
JC (00:12:40):
It is already going to be several meters, because you need to be able to walk around the telescope. You need to be able to hold the dome up. You need the dome to rotate to face the target. So you need a tracking system to rotate on, and all that sort of stuff.
(00:12:55):
So the size of the dome itself is not terribly surprising. It is the size of the telescope which actually set the size of the dome.
(00:13:03):
These days, telescopes are made slightly differently, in that it tends to be cheaper to make the whole thing capable of living outdoors, and just have bits and pieces open up. Rather than have the entire thing squirreled away inside the dome. So the way in which things are done has changed.
(00:13:27):
But yes, it is very big and very heavy, and has some really good motors which make it rotate. So yeah, it is big.
EW (00:13:39):
Well, and it has to do- The steel dome is partially because you have high winds where you are.
JC (00:13:46):
Yes. Definitely. The reason we have high winds is that mostly the area around these mountains is very flat, and these mountains protrude into the airflow. We often get a condition, and we have it today, where you look at the mountains and there are clouds downwind from the mountains, that have been formed by the wind passing over the mountains.
(00:14:11):
So you get local weather, local rainfall around the mountains, as a result of the mountains being there. Yes, when you put something on top of the mountain and you are dealing with very high speed winds, the thing had to protect itself.
(00:14:29):
We did have, for example, an engineered maximum wind speed that we were allowed to keep operating with. If the wind speed got above that, we had to shut the dome. And if the wind speed got above something like double that, I imagine we would evacuate. It is one of the things you have to deal with when you put something on top of a mountain, is you get high winds.
EW (00:14:57):
So as a developer for a telescope, I would think you would have all of these different control problems. But the telescope was built in 1974. What exactly do you need to do? Like, what is a day in the life?
JC (00:15:18):
When you have a fault with the telescope- Everyone arrives in the morning. We have our eight o'clock meeting, and we get the list of faults that have been detected by the operator overnight. Some of the faults will involve the control systems. We need to know if the fault is in the control system or in the hardware. Or if it is a false report.
(00:15:45):
So we need to be able to maintain the software of the control system to some extent, or at least to be able to probe the software to understand it. And so sometimes we would get a fault and we would have to write a bit of code, in order to probe the understanding of how the control system works.
(00:16:03):
Or, we would have to read the code of the control system, to figure out how to tweak what is going on. Or we would have to go find the documentation in the archives, and try and work out how it is meant to work.
(00:16:17):
Every now and then there would be a new system to develop. Or a system to replace, because we just cannot get the parts anymore to make it run, and we need to replace the control system with something else. So there is an ongoing maintenance and there is also the daily fault finding. So yes, there are things that have to be done.
EW (00:16:40):
The fault finding. When I think about moving 260 tonnes, I think one of the faults may be overcurrent.
JC (00:16:49):
Yes.
EW (00:16:52):
What kind of faults are we talking about here?
JC (00:16:54):
Oh, there are an awful range of faults.
CW (00:16:56):
<laugh>
JC (00:16:56):
Everything you can think of goes wrong. Because this is a one of a kind thing. You cannot take a list of faults at other telescopes and make any sense of it enough to be able to predict what sort of faults would happen at this telescope, for example. There are just not enough telescopes in the world to be able to, of this scale, to be able to do that kind of analysis.
(00:17:17):
But things that go wrong, everything from a part wearing out, to electrostatic discharge. To lubrication was insufficient and so the current was too high and it tripped, and the operator did not have the qualification to reset the trip. Or something like that.
(00:17:42):
I do not know. The faults, we would typically get two or three a day. We would have to work through them and figure out how to fix it. Preferably so that the fault does not come back the next day.
(00:17:56):
Every now and then we would have a fault keep coming back, because the overnight operator-
EW (00:18:04):
Just keeps pushing "Go." Do it anyway.
JC (00:18:07):
Trying to use the telescope. Yeah.
CW (00:18:09):
Trying to do science here, man.
JC (00:18:13):
That is right. There are all sorts of things like false alarms, like the alarm goes off saying there is rain, but there is clearly no rain, no clouds. Why is that happening? That sort of thing. Or, trips of circuit breakers, that we have never had tripped before. We have to go and find the circuit breaker and reset it. That sort of thing.
(00:18:35):
There are all sorts of instrument faults as well, not just telescope and dome faults. There are ways in which the instruments can fail to thrive. Like they might be the wrong temperature. Or, the supply of liquid nitrogen might have fallen too low. Or, cannot turn it on. Or, cannot find the key to get in the room. All sorts of things go wrong.
(00:19:04):
It is not for lack of attention to detail or design. It is because it is a really complex system, with lots of subsystems that all inter-react in weird ways. So yeah, it is weird.
EW (00:19:24):
It has built up over decades. Would it be simpler to just start over?
CW (00:19:32):
Well, I was going to ask, is this the Ship of Theseus? How many of the original systems when it became operational, were there by the time you left?
JC (00:19:40):
Grandfather's axe. Yeah, quite a few of them. Very little was actually replaced.
CW (00:19:47):
Really?
JC (00:19:48):
When I was there. Over the years that I was there, if you count it as a percentages of total number of subsystems, mostly maintained. So yeah, that is-
CW (00:20:01):
So to Elecia's question of-
JC (00:20:02):
That is kind of normal. To answer the question, "Would it be better to knock it down and start again?" people do make that kind of calculation. The people in charge of the funding, they work out how much it would cost to replace it, how much it would cost to do something comparable, and whether they would be better served to do that rather than to proceed as they are.
(00:20:29):
Almost always they find it is cheaper to proceed as they are. Cheaper in that they can get more science done if they keep the thing running, than if they stopped for a while and built a new one. If they stopped for a while and built a new one, the next question is, "Would they build it there?"
CW (00:20:49):
Right.
JC (00:20:50):
Because there might be reasons not to build there now. The light pollution is different to how it was in the 1960s when the decision was made. The pollution from the satellite clusters, constellations, is different.
(00:21:08):
And also the science. What science do you need to do? What kind of field of view do you need? What kind of analysis of the light do you need to apply to it, to be able to make the research results that generate the science papers? That changes over time as well.
(00:21:27):
You mentioned before, a four meter class telescope. The classifications are, I think, arbitrary. There is, I think, one other telescope, which is the same fundamental design. I do not know if it is still in use. But there are other bigger telescopes being used for different things.
(00:21:49):
So this telescope is sort of in a niche, scientifically. It is used for mostly galactic survey and for spectrographic analysis of stellar sources. It is not used for planetary science, much. It is not used for looking at asteroids, well, much.
CW (00:22:13):
Not intentionally, necessarily, but sometimes does pop up anyway.
JC (00:22:18):
I was operating it once and we were asked to look at asteroids. So that was novel and unusual, because it was not imaging of the asteroids in terms of looking at the shape and size and how they are rotating. But it was chemical analysis of the reflected light using the spectrograph, compared against sunlight.
(00:22:40):
You look at the subtraction between the two spectra, and you can work out the absorption spectra of the surface. From that, you can work out what chemicals are present. If you compare those with the results of the last time you did it, you can see if anything interesting has boiled off. Or if someone has been up there mining and you had not noticed that.
CW (00:23:04):
So this was built in the '70s, and you said most of the control systems, or a fair portion of them, remain. What were the computing systems and control systems? How were those built then?
EW (00:23:17):
And have you replaced them with a Raspberry Pi?
CW (00:23:19):
Is this an IBM mainframe? Or was this totally bespoke?
JC (00:23:27):
Raspberry Pis got a bad reputation, because of the microSD card tendency to fail. While microSD cards were doing that for certain years of production, it was those years that the telescope first started using Raspberry Pis. So they had a bad experience. The modern microSD cards do not do it as badly anymore, and there are ways to make sure your Raspberry Pi stays live.
(00:23:55):
But yes, the really old control system that operated the telescope drive, which points where you want to see the target, that system has been replaced two or three times. It is currently an Intel based Linux system. Previously, it was a VAX based VMS system. And before that was something else.
(00:24:22):
But each time it was replaced, it grew new features. It retained the same pointing accuracy so that the science could continue But its main purpose as a subsystem is to point the telescope at the target and to keep it pointed there, as the most irritating thing in the world happens, which is the earth rotates.
(00:24:46):
As the earth is rotating, your telescope slides off the target. So therefore you have to keep the telescope moving at exactly the same rate as the earth rotating. If you do not keep it at that rate, then your pictures become blurry.
CW (00:25:01):
Not only that, you got to make sure that thing is aligned correctly with the celestial pull.
JC (00:25:07):
Yeah, that is right. That is one of the advantages of the equatorial mount, is that only one motor is running, mostly, when it is tracking like that. Because it is one motor which compensates for the rotation of the earth.
(00:25:22):
Whereas if you are using an altazimuth mount, you are running two motors continuously, at different rates. So mechanically, that was an advantage back then. But these days you would build altazimuth because it is easier.
EW (00:25:41):
Because it is easier to build?
JC (00:25:43):
Yeah.
CW (00:25:44):
And we have other technology- When you do altazimuth, even though you can track an object, the sky still rotates over time some. So as you take images, you have to compensate for that by rotating the entire sensor to keep- Yeah. So there is other stuff you have to do.
JC (00:26:01):
There are downsides as well. An altazimuth telescope cannot look directly upwards. Because if it does and you are tracking a star at the same time, there is a point in time where you have to rapidly rotate the telescope, so you can start following the star down the other horizon.
EW (00:26:19):
Gimbal lock.
CW (00:26:20):
Pretty much.
JC (00:26:21):
It is called gimbal- Yeah, that is right. We have that happen with the dome. It is rather slow and subtle to begin with. Everyone is sitting in the control room. The astronomers are reading their articles and doing their exposures. They have invariably picked a star which is going to go right overhead, because that minimizes the air mass, maximizes the photons that are collected by the bucket. So the spectrograph results are the best.
(00:26:52):
So as the star goes directly overhead, what happens is the dome zooms, goes faster and faster trying to keep up. It always does. There is enough gap in the aperture so that the dome does keep up. But it gets so noisy. At some point everyone looks around and say, "What is that noise?" And, "Oh yes, the target is right overhead." That is because the dome is trying to catch up with the telescope. Then it gradually slows down and calms down.
(00:27:28):
For the particularly young astronomers who have never seen this happen, I would take them outside in the dark of the telescope, and let them see with their small amount of light, the dome racing past.
(00:27:46):
So yeah, it is one of the side effects of having that kind of mount, combining with another kind of mount. There are other limits as well, like the telescope cannot point too far down, otherwise it would fall off its mount. We do not want that to happen, and so therefore there are limits.
EW (00:28:07):
You mentioned the Dark Energy Survey? Or the Two Degree Field?
JC (00:28:15):
That has been mentioned, yes.
EW (00:28:17):
This is- Again, Wikipedia knowledge here. "A robotic optical fiber positioner for obtaining spectroscopy." Basically tell me how this works.
CW (00:28:34):
<laugh>
JC (00:28:37):
Yep, sure. So it is a big plate. It is several hand spans in diameter, very flat. It has been machined to be very flat. It has a metal which has been chosen to have a very low coefficient of expansion as temperature changes. Therefore, as the temperature changes inside the dome, it does not change shape and size.
(00:29:05):
There are these 400 fibers that are in retractors around the edge of the plate. What you do is you pick out the end of the fiber, which has a magnet and a mirror at 45 degrees. You put the magnet onto the plate.
(00:29:23):
That collects the starlight at that point on the plate, because you focus the image of the stars onto the plate, using the telescope. And then the starlight from that one star or that one galaxy, goes down the fiber to the spectrograph, where it is presented to the spectrograph for analysis.
(00:29:44):
To put out 400 of these things by hand has been done, but is very boring and takes too long. So there is a positioner, an XY table and a gripper, which picks these magnets up from the edge and puts them onto the plate, in the position according to where we think the starlight will appear.
EW (00:30:09):
Okay. I want to see if I understand. You have 400 fibers. Where naively you might make a pin screen where you just have 400 in a grid, that does not make sense, because stars are not in a grid.
(00:30:27):
Stars are in, let us say constellations, even though that is not really right. Instead of putting your 400 optical fibers, single one optical fibers, in a grid, you are going to put them in the constellation pattern with great precision.
CW (00:30:46):
Do you want one fiber per target?
EW (00:30:48):
One fiber per star.
JC (00:30:50):
Yeah. And you decline to put the fibers out where you do not want the light collected. So if there is a nearby star which is obscuring your view of all these other galaxies-
EW (00:31:02):
Because it is too bright.
JC (00:31:04):
You do not put fibers- Yes, too bright. You do not put fibers there, so you are blocking the light. Therefore you get to analyze the light from the things you are interested in.
(00:31:13):
Of course, this takes a survey view of the sky first. The people who prepare these lists of targets for the positioner, they have gone through a lengthy analysis process to work out what targets they are interested in, and what shading they may require.
(00:31:32):
All that is done by the astronomers long before they get to the telescope. They give us the data file. They upload it into the Linux system, which handles the positioner. It then positions the fibers into all those places.
(00:31:47):
And yes, you can be up at the prime focus axis, which is where we lean the telescope over so we can service this instrument during the night. And you can look at the plate of fibers and you can see the constellation pattern.
(00:32:04):
But since it is only a two degree field of view, it is probably not going to be a constellation you recognize. Because two degrees in the sky is not the shape and size constellations that we have named.
CW (00:32:18):
Four full moons.
EW (00:32:19):
So two degrees is four full moons?
CW (00:32:22):
Yeah.
EW (00:32:23):
And full moon is a half a degree.
CW (00:32:25):
I think so.
JC (00:32:25):
At the moment, it is very bright.
CW (00:32:28):
Yes.
EW (00:32:29):
Right. Well, we are at perihelion now.
JC (00:32:36):
Yes. For us down here in Australia, the moon is in the northern sky, which seems weird and rare. It does that occasionally. The Earth-Moon system geometries and patterns are frankly strange. But yes, at the moment it is leaning over quite low. So as it rises, it is rising quite low.
(00:33:02):
We are in the middle of summer. We are about to enter into a week-long heatwave, so we know about it.
EW (00:33:08):
Okay. So I would not recognize the constellation, unless it was some tiny constellation. Maybe the Pleiades I might recognize? I would not, because I do not, but Chris would.
JC (00:33:20):
Yes, that is right. I have been up there trying to sort out a tangle, and recognize the pattern of what it is. But on the other hand, I have probably been primed by seeing it on the computer screens down the control room. But yes, you get to see where the astronomers have chosen to select the light using this instrument.
(00:33:41):
Then the light goes down those 400 fibers into the spectrograph. They go into a long row of fiber terminations. So the spectrograph is looking at this vertical slit containing fibers. Each of those fibers, the computer knows where they are up on the plate, and so therefore can identify which target they are.
(00:34:07):
The spectrograph processes these 400 stars all at once. As a result, you get to do a spectrographic analysis of 400 stars each time you use this. So 20 minutes of exposure and there you have it.
(00:34:24):
During that time, the plate at the other side of the instrument is being prepared by the positioner, while it is doing the exposure. Then when we are finished exposing, the thing is tumbled while we move the telescope to the new target. So a new set of stars is then collected.
(00:34:42):
It is a very efficient way of collecting spectra of stars and galaxies, in terms of the number of spectra you can collect per hour.
EW (00:34:54):
But the downside is you can only look at things you expect to see.
JC (00:35:00):
Yes. You have to have done a survey first, so you know where the things are. Or you would be using the databases to work out where the things are. There are certain things that you cannot look at at the same time. You will tend to select all the stars of a certain brightness. So the exposure in the spectrograph is appropriate and nominal.
(00:35:17):
Because if you accidentally collect a very bright nearby star, while trying to image very distant galaxies, you will just have a lot of light coming out of one pixel in the spectrograph and that will disturb your results.
EW (00:35:33):
So there are a lot of things you have mentioned here. There is the control system of controlling the whole building. There is the keeping it cool, which has to have a control system too.
(00:35:45):
There is the robotic positioning of these optical fibers, based on what astronomers give you, which probably is never in exactly the format you want. There is all this data collection. And then of course, eventually there is going to be this pile of data that needs to be turned into a scientific paper, depending on what they were looking for.
(00:36:07):
What parts of this were you involved with when you worked there?
JC (00:36:14):
For the break fix, pretty much anything that involves software, or control systems with a software component.
(00:36:19):
For everything- Being an operator of the telescope overnight, I was responsible for everything. But not necessarily having to fix it. I would just have to find a workaround. So if the rain alarm is going off when it should not, then I would just ignore it, or I would go outside to make sure it is not raining.
(00:36:39):
If the internet was down, then I would find a workaround. If there is a power failure, I would make sure the generator started. If the dome does not open, I would kick a few things and get help from the technician on duty. And if that still did not work, we would have to abandon the night.
(00:36:57):
If the dome does not close, that is a fire risk. Because you do not want the sunlight shining on the mirror, because it can reflect off and cause fires. So you make sure everything is closed down on the telescope, and you point the dome away from the sun. And you tell the morning crew that, "Sorry, we could not close the dome."
(00:37:17):
If that 2dF instrument faults in a way where it tangles the fibers up, then you have to get up there with a set of pliers and move the magnets around them. Put them in their parking positions, and then get the software to survey them.
(00:37:32):
Sometimes there would be things where we are not sure why it is happening, and we have got three hours spare because there is cloud. So I will read the software which drives the control system, read the source code before I put in the fault report, to see if I can figure out what is actually going on.
(00:37:48):
So it is a lot of live production DevOps kinds of things during the night. But when you are on day shift, it is a matter of what the fault is, which subsystems are involved. Which ones of those are software, hand them off to the software technicians. Which one of those are mechanical, hand them off to the mechanical technicians.
(00:38:10):
We go our separate ways, do our separate things. They put the oil in. I do the upgrades or analyze the data. Then hopefully we get it working again. But yes, a very large collection of very complex subsystems with all sorts of interesting relationships, makes for some exciting work.
EW (00:38:32):
It is funny how sometimes people think that engineering is just sitting with your butt in the chair, and typing at your computer. It does not sound like that has been true for you.
JC (00:38:43):
No. At One Laptop Per Child, we had this philosophy of dog fooding. Where if we are making a laptop for children, we really need to make sure we can use them ourselves. That way we find all the faults. Because there is nothing like finding a fault, when you are trying to use a laptop for your own use.
(00:39:05):
The same thing when operating this telescope. The software faults which were particularly annoying to me as a telescope operator, were the ones that I would tend to concentrate on and fix.
(00:39:18):
An example is there is an acquisition and guide unit underneath the telescope, where you used to put the glass plates, back when we did that kind of photography in the 70s. The image of the stars is focused down there, and that is where you put the smaller instruments, rather than 2dF.
(00:39:39):
But this XY table down there can position probe cameras, which are used to guide the telescope. You have one of the cameras looking at a star off the center of the field of view, and guide the telescope based on the movements of that star.
(00:39:56):
But this XY table has a control system all of its own right. It was occasionally not finishing the job, when you ask it to move a camera. You tell it, "Move the camera over here," and it would move only one of the co-ordinates. The X co-ordinate, instead of X and Y.
(00:40:16):
So I got curious one day and said, "This is costing us a lot of irritation while observing, because you try to do something and it does not always complete." And found faults in the software, where interrupts were occurring and they should not have been.
(00:40:33):
Once I had fixed that fault, which took a few nights of rainfall where we could not do anything else, the acquisition and guide unit motor control was now reliable. In that 100% of the time when you ask it to move, it would move. Whereas before it was only 75%.
(00:40:56):
I cannot remember the question now.
EW (00:41:01):
I think you answered it just fine.
JC (00:41:03):
Good.
CW (00:41:03):
So this is a big organization.
EW (00:41:08):
It is not just you.
CW (00:41:09):
Yeah. It is not just you. You mentioned other software technicians, mechanical technicians. I am sure there are many operators who are ostensibly operating, and then there are the astronomers behind.
(00:41:19):
One of the things you mentioned about the fiber thing was efficiency. Like being able to reconfigure while something else was happening, so that it is ready to go as quickly as possible.
(00:41:31):
I imagine there are not a lot of big observatory telescopes in the world, and there are a lot of astronomers. So there is probably a big backlog. There are a lot of people who want to get time on these instruments.
(00:41:44):
So having downtime is probably- You have got people yelling at you, right? Like, "Oh, this schedule. I have got to observe this thing. If I do not get it this week, then it is going to be out of view, or unfavorable." How much does that pressure come into the work?
EW (00:42:00):
How much do you really enjoy rainstorms? <laugh>
CW (00:42:04):
Yeah.
JC (00:42:04):
Yeah. Well, the astronomers compete for time on the telescope. They put in their draft papers, if you like, what they plan to do with it. And a team of scientists works out the priority of which these observation plans should go ahead and when.
(00:42:28):
Then they have to be fitted into what the moon is doing, because some observations cannot be taken when the moon is bright, or when the moon is behind the target, those sort of things. Then there is the funding. Some of these observations are funded by certain sources and some by others.
(00:42:47):
That generates a schedule of observations. So an observer who wants to use the telescope for four nights to look at a cluster of galaxies, might only need only one night to get that done. But they have booked in four nights, because they really, really need to get that done.
(00:43:07):
The team doing the scientific decision making has agreed with them that they really need to get it done. So booking it over four nights makes it more likely that they would get it done. They would have secondary observing tasks that they complete within that time, so other less important targets.
(00:43:25):
For those nights, we would notice they would be quite calm and collecting very boring targets for six hours, until their target of preference rises high enough to minimize the air mass for the science. Then it is everyone is excited, and point the telescope and capture this thing, and eagerly wait for the results.
(00:43:52):
Then, "Okay, that is done." They will have dinner while they would do the last exposure, and then go back to boring targets again. So it can change during the night.
(00:44:04):
We had six telescope operators. So every six weeks you would get a week on. Everyone would get a turn one by one, some of them more frequently than others, according to their contract.
(00:44:23):
But from the point of view of the astronomers, they have booked their time. If they do not get their time because it is wet, the agreement they have with the science committee may mean that they will get time reserved to them in the next schedule.
(00:44:38):
So sometimes we would have astronomers come back to complete the work that they had hoped to do before. But they have had to come back a month later when the moon is back to how it was. Or, the next semester when the schedule is rewritten. Or, there would be reserve nights where that sort of thing might happen, if the director so desires.
(00:45:01):
Yeah, there is a whole complex of allocation and scheduling that happens, long before the telescope operator gets to meet the astronomer on the night.
EW (00:45:12):
Listener Bailey asked a question that works well here. "Did you ever get to look at something you wanted?"
JC (00:45:19):
Yes. That was fun, actually getting to look at something I want. Because you are in charge of this gigantic telescope, and your job is to make sure the astronomers get what they want to look at. But every now and then the astronomers would stop. They would not want to look any further, because something was stopping them from looking.
CW (00:45:38):
They have seen the horrors of the universe. <laugh>
EW (00:45:40):
<laugh>
JC (00:45:40):
Well, they are usually over that by the time we get them.
CW (00:45:47):
Oh, right. <laugh>
JC (00:45:49):
It is when the atmosphere is particularly noisy, when the light is not arriving cleanly. It is called when the "seeing" is really bad. Or when the clouds are so prevalent that they cannot get an exposure in, without also collecting moonlight or reflected light from cities or something like that. So therefore the science they would do is hindered by the conditions. Or if it is just plain raining, or it is cloudy.
(00:46:17):
But situations where I can keep the dome open, and the only thing stopping us from doing something is that the clouds are not right. If the astronomer agrees, then we might swing the telescope to something else that we can look at for some other reason.
(00:46:38):
Like for example, if there was a fault with the telescope, which said that it was unwise to run it in the southwest sky for some reason, because of the dome catching or something like that. Then we would give that a swing.
(00:46:54):
We would pick an interesting target down there and run down there, while we are waiting for the clouds to stop being all furry and wobbly as they go past. So yes, every now and then that sort of thing happened.
(00:47:07):
There were one or two cases where the astronomer just bowed out, because the seeing was too bad. They would go off and sleep, which they really need to do. I would be left with a telescope and so look at the moon or a planet or-
(00:47:23):
There are also other things that have to be done, like updating the mathematical pointing model. Which means looking at 20 or so stars all in a row, and making sure that they are exactly where they are meant to be. That lets you make sure the telescope is going to find the star the next time. There is that sort of job to do as well.
(00:47:44):
Along your way, you might see, "Oh, look, there is something over there," have a look at that. At one stage I knew where Voyager was, and so pointed the telescope in that direction. Entirely could not see it.
CW (00:47:58):
<laugh>
EW (00:47:58):
It would be so small.
CW (00:48:00):
Oh, damn.
JC (00:48:02):
Exactly. There is no way to see it, but to know that you have got a picture of the sky, totally black, where Voyager would be if it was big enough to see, that was fun.
(00:48:15):
There are other objects around like Vesta, and unusual objects that are worth looking at sometimes. But it is not a telescope that is really good for looking at things that are close by.
CW (00:48:32):
Planets and such.
JC (00:48:34):
Like you can look at a mountain or two on the moon, because the field of view is so narrow. It is for looking at very distant objects. And it is a spectrograph telescope. It has got a bunch of spectrographs behind it.
(00:48:46):
So the only cameras we have for looking at things with, for looking at things as a human, are the guide cameras. They only really need to approximate the center of a star, in order to guide the telescope to keep it on track.
(00:49:03):
They are not very good cameras for this sort of stellar photography sort of thing, that you can do with mobile phones and smaller telescopes. So it is not as rewarding as getting your own telescope and pointing it at the sky.
(00:49:18):
But it is really good at seeing very distant objects. That is the sort of class of telescope it is. It is not a fault. That is what it is designed to be. It is designed to collect spectra.
CW (00:49:33):
While this is an old telescope, it is not such an old observatory that it was a telescope with optical viewing, with an eye piece that you could go look at. This is all camera based stuff, from the get-go.
JC (00:49:47):
We do have eye pieces.
CW (00:49:47):
Oh!
JC (00:49:47):
But I must admit we have not used them frequently. I am not even sure if they all work. It is easier to use the guide cameras. Because the problem with using the eye pieces is they are in the telescope, and you have got to get into the middle of the cage.
CW (00:50:00):
Climb up. Yeah.
(00:50:02):
You cannot do that alone, because you need to have someone to get you out again afterwards. With safety and human health these days, you need to not do that if you can at all avoid it.
(00:50:14):
Back in the 70s and 80s, the astronomers would ride the telescope in the focal positions with their stack of photographic plates and collect all their targets that way, communicating with the operator. You would wonder how they would get out of the thing if they needed to. They would bring a rope ladder in case the drive failed. I never found out that.
(00:50:39):
I never wanted to be in the telescope. It like being in a cage. It is actually called a "cage," for that reason. You cannot easily get in and out if it is not in its parking position.
EW (00:50:51):
William asked if the proliferation of objects by the commercialization of space has impacted your work. Do you have to adapt to the increased orbital traffic? Is this going to become an issue more so in the future?
JC (00:51:08):
Living on the mountain as a resident, you had to secure your own internet access, which was always mobile phone service. When Starlink arrived, we were very happy. Because we could each get our little Starlink kits and have full internet access, instead of having to walk up to the telescope and using the Wi-Fi there.
(00:51:30):
So in one respect, it was great. In another respect, it gives you something to look at when you see the trains of Starlink satellites going over.
EW (00:51:41):
They are pretty.
JC (00:51:43):
They can be pretty. They appear on the All Sky Camera, which is available on the internet. You can actually look at what the sky looks like at the observatory. There is a link to it somewhere. I can provide that.
(00:51:56):
The Sky Camera also captures all the satellites as they go over a particular point in the southwest about an hour before dawn, when their solar panels reflect the sun. There is a glinting effect that appears as well. We noticed that.
(00:52:14):
We have never had to, in my experience there of many nights observing, never had to point somewhere else because of one of these satellites. Because the satellites only obscure the field of view for a half second or so, and our exposures are typically ten to 20 minutes long.
(00:52:34):
So if you are getting the wrong photons for a couple of seconds out of 20 minutes, it does not really matter. You can work out mathematically what the effect of that noise is, and you can compensate for it by adding a few more minutes of exposure.
(00:52:51):
If we were a telescope that was trying to take astrophotography, photographs of wonderful looking things in space, yes, it is a problem. But with the field of view and the type of spectrographs we use, it was not a problem.
(00:53:07):
It is a problem for astronomy in general. The astronomers are, of course, rightfully upset about it and that they are losing their dark skies. It is also happening in radio astronomy as well, where many of these sources going overhead are now emitting in fairly wide bands of radio frequency. So they have got problems too.
(00:53:32):
But for this particular telescope, apart from knowing that it is a problem for everyone else, and apart from being able to see the things as they go over, we did not really have anything where we needed to avoid a target.
EW (00:53:51):
Christopher, you have a telescope, a small telescope.
CW (00:53:56):
I have a telescope in the sense of I have a Hot Wheels car compared to a semi, to compare these two, yes.
EW (00:54:08):
It used to be, when you had a different older telescope, you needed it to be as dark as possible. You were really annoyed by the neighbor's light. But now you set it out and cars drive by. You take hours and hours of exposure and then stack it. So you are not necessarily affected by the Starlink lights either.
CW (00:54:33):
That dictates how many of those I have to throw out.
JC (00:54:37):
Yes. It sets the number of hours.
EW (00:54:40):
Is this spectrograph sampling similar to your stacking?
CW (00:54:45):
I do not think so, because imagine I am just looking at individual star. I do not care about the whole sky. If I am taking a picture of Andromeda, that is a big picture of the sky. I want to see the thing, and I do not want 16 lines going through it.
EW (00:54:57):
Which is what the satellites would make. They would make straight lights.
CW (00:55:01):
Now imagine I just want to look at 400 stars in that view, or galaxies which are pinpoint sources. But I just put the fibers over them to look at individual ones or something. The chances that those lines cross one of those is much smaller, than distorting my entire image. I think.
JC (00:55:22):
Remember also that when we are observing at a big telescope, we are typically interested in full darkness, so after twilight is finished. And when the target is almost directly overhead. So most of the satellite constellations are then invisible. If they are glowing at all, they are glowing because of reflected light from cities, not from the sun.
(00:55:47):
So you really notice it in the evening, if you are trying to do astrophotography before you are going to bed. Because you have still got the sun up there illuminating the satellites. But much later in the night, it is much less of a problem.
(00:56:00):
One of our spectrographs, Veloce, is for looking at single targets. It measures the rotation of those targets very accurately, using spectrographic analysis. For that one, I would imagine that we would not want to use it too early in the night, because there is too much reflected light from the sky itself, let alone reflected light from satellites.
(00:56:28):
But even then, the exposures are sufficiently long for most of the science targets, that even if you do have a satellite going across it, probably would not notice. And you would do another exposure if you were not sure.
CW (00:56:44):
Yeah.
EW (00:56:47):
Okay. I have two more big questions. Christopher, do you have any before I go on?
CW (00:56:52):
No, go ahead.
EW (00:56:54):
What is a starbug and how can I get one? That is one question.
JC (00:56:58):
Well, I always thought the Starbug was the little shuttle craft from "Red Dwarf," the British science fiction comedy. But when I was up there, I learned that starbug was a re-engineering of the 2dF target mirrors. Which instead of using a positioner to position the mirrors on the plate, the mirror would position itself by being vibrated into position.
(00:57:28):
We had people who had to go and clean those things once a week, but I never actually got to see it myself. That was at a telescope down the road at the same observatory.
EW (00:57:45):
Starbugs just popped out at me as something that sounded like a lot of fun.
(00:57:49):
You have been an engineer, or been in engineering roles for quite a while. You mentioned you have a business degree. But you have been doing development for a few years now.
JC (00:58:09):
Yep.
EW (00:58:09):
What advice would you give to folks starting their engineering careers? Or considering engineering careers?
JC (00:58:18):
Firstly, it is rare to be asked for advice. Secondly, there are insufficient cases of my giving advice and seeing the outcome that I can determine if my advice is reliable or useful. Thirdly, it is against my interest to give advice, because I would prefer to be the only one on the planet who can do this. That way, the price that I can charge is the highest.
(00:58:44):
But in general, engineering is best done repeatedly, and in as many parts of the engineering sequence as possible. So while I started in financial software, I moved to building complex systems out of existing products, when I was at Digital. And working in areas of engineering, which I had not been experienced with before, but utilized the skills.
(00:59:22):
I just kept increasing the skills that I had. Those charts of skills to learn are a good guide as well. The other thing I learned from experience is that as I was already very good at engineering before I even entered the workforce, because I had an engineering father, I was able to concentrate on a different degree. A degree that would serve me better. That is why I got a business degree.
(00:59:56):
Since then, I have worked in every part of a engineering. Sustaining at One Laptop Per Child. QA at One Laptop Per Child. Support when I was at HP.
(01:00:09):
Support of software is funded by the engineering group. When an engineering group gets the software wrong in some way and raises the support costs, they very quickly learn what they have done wrong and they issue a release to reduce their support costs. We would very gladly show them how they have done it wrong, because we are all engineers too.
(01:00:36):
So yeah, get another degree. If you fixate on engineering, what you are doing is selecting your potential employers. You are reducing the field. I never thought that was a good idea. Because I am working in Australia, where there is so little population compared to other countries, that my situation is different.
(01:01:05):
Where most of your listeners are in highly populated areas, you may need to use different strategies or specialization and labeling, and hence get an engineering degree. But once you got your degree, go get another one. In music or something else, and then combine them.
(01:01:25):
Because someone with a combined degree is in a smaller population of potential candidates. So when you find that perfect job which combines those degrees, off you go and get it.
(01:01:38):
The other thing I learned was not to overly optimize my performance, because the rewards are not necessarily useful. And so I- How can I say this? I would tend toward picking roles which have an engineering component, rather than being purely engineering.
(01:02:04):
Yeah, that is what I have done with my career. I have not planned my career. It is just something that happened.
EW (01:02:15):
I think that is true for a lot of people. I have had plans at different times, and pretty much none of them have panned out. I am perfectly happy with that.
JC (01:02:25):
Yeah. Planning is a wonderful way of failing.
EW (01:02:28):
That is a wonderful way of taking stock. Just because you plan, it does not mean that is where you are going to end up. But at least you have an idea of the possibilities. Still, my plans really have not worked.
(01:02:39):
Oh, one more question about telescopes, sort of. Simon wanted to know if the Aussie wildlife is as dangerous for the telescopes, as it is for the humans. Do you have any good animal stories that involve the telescopes?
CW (01:02:55):
Oh no.
JC (01:02:55):
Yes. Yes. One of our technicians was bitten by a brown snake, as he was driving to the site, I think. He has written about that.
EW (01:03:10):
Wait, wait. In the car? He was-
JC (01:03:12):
Yeah, he got out of the car, and the snake was on the ground next to where he had parked his car. It bit him immediately.
EW (01:03:21):
Brown snakes are bad?
CW (01:03:22):
I would assume any snake in-
EW (01:03:24):
In Australia is. Yeah.
JC (01:03:26):
The Australian brown snake is the worst. We also had brown snakes living in the car park at the telescope. We have signs at the doors reminding us as we leave, not to go that way. We have had some of the holes sealed up.
CW (01:03:44):
That is good.
JC (01:03:44):
So that the snakes would go somewhere else. Another big one is ladybugs.
CW (01:03:49):
Excuse me?
EW (01:03:51):
Ladybugs.
JC (01:03:51):
Ladybugs.
EW (01:03:51):
They are so bad. What?
CW (01:03:54):
The venomous ladybugs in Australia that I do not know about?
EW (01:03:57):
That makes sense.
CW (01:03:57):
<laugh>
EW (01:03:57):
Venomous ladybugs.
JC (01:04:00):
No, I do not think these are venomous, unless you eat higher than your LD50 for them. But they get into everything. They have been buzzing around this national park next door, and see this white thing on the hill and think it must be the moon. So they head in that direction. They get into all the nooks and crannies of the equipment.
CW (01:04:20):
<laugh>
JC (01:04:23):
One of our telescopes on the mountain had such an infestation of ladybugs, that it was get out the broom and the vacuum cleaner and try and remove them. And there is on the 2dF instrument- The way it is assembled onto the telescope, is that these instruments get craned on and off the telescope at different times.
(01:04:44):
There is this couple of big cylinders about a meter or two long and half a meter diameter, which you use to roll up the hundreds of meters of optical fiber bundle, so that you can take the instrument off the telescope.
(01:05:00):
So just to explain, we pull the fiber out of the building and wind it onto these two spindles on the telescope. Then take that bit of the telescope off the telescope and set it aside on the floor. Then put something else on the telescope. It is part of an instrument change, which happens every week or so.
(01:05:21):
And in the cylinders, someone has written "Ladybug Storage," as a joke. In that it is one of the things which would collect the ladybugs during observing. You might have to do something about it, if there are too many of them, or they give off a stink.
(01:05:47):
The kangaroos are always a menace. Because when you are driving past them, if they have not done their liftoff to get away from threat, then the direction that they are going to jump is going to be random. They do not always pick the direction you expect.
(01:06:07):
In a given herd of ten kangaroos, they almost never pick the same direction. Unless one of the others has already picked their direction, in which case the others may follow. So if you arrive at a herd of kangaroos too quickly, they will jump in all directions. One of those directions might be into your car. So, yeah, that is where life at the telescope can be irritating.
CW (01:06:33):
Got to hope the kangaroos and the brown snakes never join forces.
JC (01:06:37):
No, they do not seem to be very happy with each other. Kangaroos will very much stay clear of the brown snakes. The birds can sometimes warn you about the snakes. Birds will react if you get too close to their nest, and they will persuade you to leave.
(01:06:53):
But the same alarm call happens in a different way, when you get close to a snake. Which is weird. They are trying to protect you, as well as protect their nest.
EW (01:07:05):
Okay. I did not realize ladybugs went towards the moon. But can you imagine being a moth? You get to the telescope and you are like, "I did it. I got to the moon. I did it!"
CW (01:07:16):
<laugh>
JC (01:07:19):
Now crawl into a hole somewhere. That hole might be important.
EW (01:07:25):
James, it has been wonderful to talk to you. Do you have any thoughts you would like to leave us with?
JC (01:07:30):
Keep engineering. It is how we make the world.
EW (01:07:35):
Our guest has been James Cameron, developer of many things.
CW (01:07:40):
Thanks, James.
JC (01:07:40):
Thank you.
EW (01:07:43):
Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listeners Slack group for their many questions. And of course, thank you for listening. You can always contact us at show@embedded.fm or hit the contact link on embedded.fm.
(01:07:57):
There will be many links in the show notes, and of course links to being able to support us. We really appreciate it if you do.
(01:08:06):
Now a quote to leave you with. Hmm. Victor Hugo. "We see past time in a telescope and present time in a microscope. Hence the apparent enormities of the present."