The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
[FELICE FRANKEL] We're lucky to have my friend, Brian Hayes,
with us today.
Brian is a writer-- a very wonderful writer, in fact,
but he also takes photographs and he
is very good at explaining what exactly is going on technically
when you make images.
And I thought it would be interesting
for he and myself to just take a little more closer look at what
happens.
So I brought Brian into the studio
to have a conversation with me and this is what
you'll be listening to now.
[BRIAN HAYES] Hi, Felice.
It's great to be here with you.
[FF] Maybe the place to start is discussing
pixels and the sensor that's inside the camera that
records the image.
[BH] We know that the sensor is an array
of little square pixels and we know
that the picture that comes out of the camera in the end when
look at on the screen or whatever,
that that also is an array of little square pixels.
And you can't help yourself just imagining
that there's a simple one-to-one mapping, that each
of those little squares in the sensor
corresponds to one of the squares in the picture.
And it isn't so.
It just isn't so and it never will be.
There's a computer graphics person
named Alvy Ray Smith who wrote the most
petulant academic paper that I've ever seen.
The title is "A Pixel is Not a Little Square,
A Pixel is Not a Little Square, A Pixel
is Not a Little Square."
[FF] I think we understand what he's trying to say.
Right.
[BH] He's quite firm about this.
And of course, he also has to admit later on the paper
that a pixel is a little square.
[FF] OK.
[BH] Pixels are what happened to photography
when it became digitized, but they're not strictly
an invention.
They existed on film, we just weren't aware of them
in quite the same way.
And if you translate this discussion back 50 years,
the question was then a matter of not
how many pixels do you want, but how fine grained
do you want your film?
[FF] You bet.
[BH] And the effect on the finished image
was very similar.
If you chose a film with large grains of silver,
you got very fast exposures, tended to get high contrast,
but you also got a lot of noise and random fluctuations,
especially in dark areas of the photograph where
it looked mottled.
Now we have a much more regular geometric structure
with, in spite of what Alvy Ray Smith says,
little squares, pixels.
And we care about counting those pixels,
how many millions of them form an image.
But the question of how much information is in the image,
how closely can you look at it before you begin
to see the structure of the image
rather than the subject of the image, those issues
have been there from the very beginning.
[FF] So, for example, just comparing it
to film, although most of our students
possibly don't even know what the word means,
they have seen a little example of how
we see the difference between setting the ISO number at 200
as compared to 6,400.
That they will have seen that there
is a lot of grain and noise, or appears to be grainy--
that's the word, although that might not be accurate.
[BH] I don't think that the difference is worth
bothering with.
[FF] Yeah.
[BH] It is grainy.
It's noisy.
I think that the clearest way to follow what's happening there
and why that noise happens and why
ISO matters is to think about the basic physics of what
you're doing when you make a photograph.
What you're doing is counting photons, the quanta of light.
So you expose a piece of film or the sensor in a digital camera
to the varying brightness in a scene.
It's focused on the plane of the film or the plane of the sensor
and what you're doing is in each small area, the square pixels
on the sensor or the chemical film grains,
in each small area you're counting how many photons
strike that spot within the brief moment
that the shutter is open.
And there's a statistical issue here.
When a photon hits the silicon sensor,
it's supposed to not lose an electron
and it's actually the electrons that you wind up counting.
When it hits the film, it's supposed
to catalyze a chemical reaction that produces
a visible grain of silver.
Both of those things are somewhat unreliable.
It's-- in part, this goes back to the deep physics of quantum
mechanics, that some of these things are probabilistic
and you can't, even if you try, determine exactly what's going
to happen.
But for a lot of reasons, when, say, 1,000 photons hit
the film or the sensor, you may register some other number
other than the exact 1,000.
You might count 900, you might actually
count more than there were photons
because some electrons are just spontaneously--
[FF] Yeah.
[BH] And sometimes the chemical reaction in the film
happens even without light.
And there is a simple rule of thumb, a mathematical rule
for explaining why it's so much better to have a lot of photons
than just a few.
The error that you get in counting
turns out to be roughly the square root
of the number of photons.
And so if you have 10,000 photons that you're
trying to count, well, the square root of 10,000
is 100 and that's about 1% error.
If you get 9,900 instead of 10,000 you're 1% off.
And when you translate that into an amount of light
or the gray level on an image, 1%
is probably something you're not even
going to be able to notice.
On the other hand, if there were only 100 photons
that you were counting, the square root of 100 is 10
and so you can have an error of 10% and something--
some pixel might be 10% brighter or darker
than it's supposed to be and you will notice this.
And in fact, it looks like snow on the image.
It looks like random fluctuations
darker and lighter.
And that's true whether you're looking at a digital image
or a film image.
So you want to capture as many photons as you can
and that's a complicated function of how
long the shutter is open, how wide the aperture of the lens
has been set, and how long or how sensitive the film is
to light or how sensitive the sensor is.
And the ISO dial is what determines that sensitivity.
In the case of film, there was no dial.
You went to the store and bought a film
with an ISO rating of whatever you chose.
With a digital camera you can adjust it was you wish.
[FF] Almost.
On my camera I cannot get-- I used to use Kodachrome 25.
[BH] I was an Ektar 25.
[FF] You're Ektar.
OK.
And I can't go down to 25 on my camera.
[BH] That's true.
I would say, however, that the whole strange historical
accident that we have digital cameras with a setting that's
labeled ISO is a little bit deceptive.
Setting something to ISO 26,000 on a digital camera
doesn't match, in any obvious direct way, what
the setting would be on a film.
And so it's with a good digital camera,
the noise level that you get out of it at a low ISO setting
is the equal of anything that you could have
gotten on your Kodachrome 25.
[FF] So in the course, we talk about
that because we have control over everything
we make in this particular course,
because it's in a controlled situation,
we are suggesting to set the ISO at the lowest number
so that we do not pick up a lot of noise.
[BH] Other things being equal, that's surely the--
[FF] Let's try to keep it simple for-- yeah.
[BH] Well, yeah.
[FF] So that's how we're taking pictures in our course.
[BH] Yes.
If you're taking pictures of something
that holds still for you--
[FF] Yes.
[BH] --and especially if you have a tripod so that
the camera also holds still--
[FF] This is the only way we're taking pictures in this course.
[BH] There would be no reason not to use the lowest ISO --
[FF] And we can add more light if we need to.
[BH] Yes.
[FF] So we have control.
[BH] Yeah.
[FF] This is very specific kind of photography.
[BH] And you have a luxurious situation.
[FF] Indeed.
[BH] There are occasions when it's appropriate,
I think, to use a higher ISO setting.
But what you described, that kind
of photography, of course-- why not?
[FF] Why not?
So shall we talk about exactly what happens to that pixel?
[BH] Sure.
I think the complexity of it is the kind of complexity
that you don't really need to understand in detail.
You certainly don't need it in order to make good pictures.
There have been generations of brilliant photographers
who knew nothing about the technology of cameras
and certainly about the chemistry of film.
However, I think it's something that
is useful to have in the back of your head.
And it may change your attitude toward the images that come out
of the camera and how you think of them
and what you dare to do to them.
There's a natural tendency, since we
know that inside the camera there's
this postage stamp sized slab of silicon that is the sensor that
receives the light, the lens focuses an image
onto that sensor.
And we know that's made up of these little squares called
pixels in a rectangular array like seats in a theater
or something, you have rows and columns full of them.
And the picture that comes out of the camera
is also a rectangular array of colored squares in rows
and columns, and so it's just natural to imagine that there
is a simple one-to-one correspondence
between the pixels on the sensor and the pixels on the image.
And if only it were that simple.
In fact, if you think about it a little more,
you realize that it can't work quite that way
because the square pixels on the sensor.
I've learned recently that the professionals who
make these things, they call them
sensels on the sensor and pixels in the picture.
But each of the picture elements on the sensor
has to respond to any light that it receives of any color.
And in order to take a color photograph,
you need to have separate signals for at least
three color channels.
Each sensor element, then, is actually
made up of four sensor elements, each of which
has a color filter pasted on top of it
like the kind of celluloid filters
that you would use over a spotlight
or something to change the lighting,
except of course, they're very small.
These are arranged in a pattern so
that each group of four sensor elements
has two green and a red and a blue filter.
So right there if you have, say, 1,000 or a million pixels
in the finished image-- 1,000 by 1,000,
say-- you actually have 2,000 by 2,000 on the sensor.
There are four million elements rather than just one million.
So we're already at that point not having
a one-to-one correspondence between the sensor
and the picture.
And in fact, it's not even close to that.
There is no simple ratio.
There are more sensor elements on the sensor than there
are in the finished picture and no pixel in the image
actually corresponds to any one sensor element.
There's an algorithm that runs on the little computer
inside the camera that looks at a whole neighborhood of sensor
elements, and by averaging and filtering
the outputs of a large number of elements
it assigns a particular color to each pixel in the finished
image.
[FF] So that each manufacturer probably has their own formula.
[BH] Very much so.
And they don't tell us what they are.
[FF] Of course not.
Is it fair to talk about, in this conversation about pixels,
is this a place where we could talk
about the difference between RAW, TIFF and JPEG?
[BH] Sure.
[FF] It would be good to have an idea about the differences.
[BH] Sure.
So each camera has a choice of outputs.
Typically, you can get either RAW or JPEG from most cameras.
And the RAW format is not the same from company to company,
I guess.
But they're all, apart from details of file formats
and things like that, they're all very similar.
It's a collection of numbers that
represent color values in a very direct way.
If you looked at the numbers and assigned colors
to each possible number, and you took a bunch of crayons
and a big piece of graph paper with squares on it,
you could actually reconstruct the whole image
by painting the appropriate color
in each square based on just looking
at the numbers in the RAW file.
You wouldn't really want to do this,
but it's there in principle.
Among the other common formats, the TIFF
is actually very much the same.
It has the same information in it,
it's just been rewritten in a different order, basically.
[FF] Good.
I have conversations with other photographers who are--
they cannot believe that I don't take RAW images.
[BH] Well, there's more to that.
If you take a RAW image and you upload it to a computer
in that form and you use the appropriate piece of software,
then, to convert it to a TIFF, you
should have the same set of numbers in the new file
as you did when it came out of the camera as a raw file.
On the other hand, if you take a JPEG
and then convert that to a TIFF, the numbers
will not match up one for one.
And the reason for that is that the JPEG format doesn't,
in fact, store this big array of rows and columns
of color values.
Instead, it takes blocks of pixels and analyzes
the set of colors found within each block and then
in order to store the-- well, not the same information,
but substantially the same information-- in a smaller
amount of data, it encodes the information
into a different form.
You might, for example, look at the whole block of pixels--
I think they're eight by eight is
the standard way of doing this.
You take those 64 pixels, an eight by eight square,
and you ask, what's the average color in that whole block?
And so that's a first approximation
of how you might represent the image.
And then you ask, so, what's the average color
on the right half, and what's the average color
on the left half, and on the top half and the bottom half?
And you store that information also.
And there's a little more to it than this, but not much more,
actually.
What you wind up with is a way of representing
the information that is more compact, but it's not exact.
And when you then reconvert it to, say, a TIFF file,
you're going to wind up with something
that's a little bit different.
And certainly, the purists among us
would argue that there is damage done
when you go to a compressed file format like JPEG.
Unquestionably, there are changes made.
That is, you have altered the information that is stored.
And the argument is that these days, with ample disk
space and essentially the ability
to keep every image you take in its full form,
why risk losing anything?
That's a good argument, and I think that, for example,
if you have the ability to keep both a RAW version and a JPEG
that that's probably a good idea.
It leaves your options open.
You can always go back to the original form
and work with it if it turns out that a particular JPEG isn't
looking the way you want to.
However, I have to say that this is
based purely on personal experience and not
scientific investigation.
I've found many cameras whose JPEG processor
does a better job of sharpening and color
correcting the image than I can do myself.
[FF] I'm so happy you said that because I have,
anecdotally, made comparisons between my images,
either TIFF or JPEG, enlarged them like crazy on the screen,
and I don't see the differences.
[BH] Yes.
Yes.
Cameras are very good, I'm afraid.
[FF] They are, yeah.
And it's a lot of smart people who've
worked very hard to make it come out well.
Not to say that you can never beat them,
but you've got to work pretty hard to do better.
And it's also worth keeping in mind,
if only as a way of keeping your sanity in a world where lots
of people want to tell you what to do,
it's worth keeping in mind that a "RAW' file,
in spite of its name, is not "raw"-- it's "cooked."
The camera has already done a lot of processing
and it has to.
What would truly be a "raw" file-- that
is the output of the sensor directly
without any processing at all-- it would hardly
look like a photograph.
The dynamic range would be all wrong,
the colors would be wildly incorrect.
The camera doesn't see the way our eye sees
and it needs to be fixed before we even recognize it
as a photographic image.
This was also true in the age of film.
If you look at, say, a color negative,
a piece of color negative film that's
been processed but not printed, the entire image
has got a rosy salmon colored tone to it
that's wildly far from the correct color.
And that needs to be corrected.
In the old days, this happened chemically.
It was the film that you were printing
to from the color negative was designed explicitly
to correct for that distortion.
And in fact, if you use that film
to print from a negative that didn't have the orange cast
to it, you would get something that was terribly wrong
in the other direction.
It's just never been the case that there
was a "raw" format that was truly
some kind of pristine representation of nature.
It's all "cooked" formats.
And given that, it's sort of liberating to know that there's
already a whole lot of processing
going on before the picture ever comes out of the camera.
And so maybe you shouldn't be too afraid of turning
a dial or two.
The conception of a camera as some sort
of objective recording device that captures reality
without distortion has never been true.
And even at a very fundamental level,
when you look at the physics and technology
of how the machine works.
[FF] OK.
So we talk very briefly about trying
to get some really wonderful images for cover submissions,
and we do talk about the submission should be 300 dots
per inch because that's what the art directors usually ask for.
And if it's a full bleed cover, it's 8 and 1/2 by 11,
but sometimes your images is not that size
so we briefly touch on putting it into software
and increasing the file size.
There's a lot that goes on when you click that button,
I'm assuming.
What do you think about all that?
[BH] Increasing the file size is really very easy.
As I say, you can just press a button.
Increasing the resolution of the image is not so easy.
In some precise sense, you're not adding any information
to an image when you simply increase
the resolution in Photoshop, say, by making it
into a larger file.
If you look at it in the simplest terms, if you were at,
say 150 dots per inch, you could take each pixel
and divide it into four smaller pixels.
Each square becomes four tinier squares.
Each of those four squares is identical.
And you then have a file that's four times as large
in terms of megabytes and has twice the resolution
and it's now 300 dots per inch.
If you look at the metadata in Photoshop or something,
that's what it will tell you.
But it has no more information than the original one.
[FF] OK.
But what we have touched on with the flatbed scanner
is when we tell the scanner to scan at,
let's say, 72 DPI and then we tell it later
to scan at 300 DPI, are we not capturing more information when
we scan the image?
[BH] Absolutely.
Yes, you are.
You are dealing there with a physical process
that-- again, if we look at it in the simplest terms,
and this isn't terribly realistic-- but
the sensor in the scanner, when you
scan at reduced resolution, at, say, 150 dots per inch,
it's going to use two sensor elements together as if they
were one and simply record half as much information.
Then when you set the scanner for 300 dots per inch,
it reads each of those pairs of sensors as two separate sensors
and records twice as much information,
or really four times as much because it's in two dimensions.
The scanner is moving across the surface,
it's not just one line.
When you do this in software, however, you
have no way of going back to the original object
that you were scanning or photographing
and recording more information about it.
All you have is what's in the original file.
Now, this seems very gloomy.
It seems as if you have no hope of actually increasing
the resolution of an image.
Once you've made it, if you can't go back to the original
and recreate it from life, then you'll
never have more dots per inch than what
you captured to begin with.
It's not quite that bad.
The statement that you can never put more information into it
than was there to begin with, it's
true in some mathematical sense in that
if you have, say, an image of totally random dots
that are scattered across the surface
and you try to increase the resolution in order
to get more dots per inch, it's not going to work.
All you'll do is make the dots you have a little bit fuzzier.
But we don't generally take pictures
of random arrays of dots.
We take pictures of objects that have macroscopic form, that
have sharp edges, they have areas
of one color and another color, they
have smooth gradients of color.
And all of that is structure at a much higher level
than individual pixels.
And it is possible, if you're clever
or if your software is clever, to find
ways to boost the resolution of an image
that if the image cooperates, if it's a picture of something
that has reasonable structure to it, which
could be anything from a human face, to a plant,
to a technological artifact.
Because reasonably large areas of color or smooth gradients
or sharp edges and that these are the things
that you're interested in, it actually comes out pretty well.
I've done it myself.
[FF] Me, too.
[BH] And in most cases, the result is successful.
There is a quasi ethical question
here of do you disclose to the art director
you're sending this to that, in fact,
this was a 150 dot per inch file that you have manipulated--
there's that terrible word-- in order to boost the resolution?
I'm not an ethicist.
I leave that to other people to decide.
It's not alteration of the content, in my view.
It's not like you're air brushing out something that
wasn't supposed to be there.
But obviously, it is a deliberate act
to make the picture look better.
I would also say that another problem here
is that the art director who insists that the file has
to be 300 DPI is being rather foolishly rigid about this.
There are images that you can print on a magazine cover that
are 150 DPI that work out fine.
But changing the world in that direction
is much harder than fixing the photograph.
[FF] Especially when the art director basically
has to first talk to the editor, who generally knows
nothing about image making.
[BH] As a former editor, I will let you have your say.
[FF] Thank you.
But no, I would say for cover submissions
where we're talking about we're bringing
the notion of aesthetics and communication,
creating a finer image.
[BH] I entirely agree.
There are various standards that get applied.
Journalism has one set of rules that's really quite harsh.
Photojournalists working for a newspaper or an agency
like Magnum, they suffer severe penalties
if they're caught making changes to images.
[FF] Like moving Egyptian tombs.
[BH] Yeah, right.
[FF] Do you remember that?
[BH] I certainly do.
[FF] This was a long time ago.
[BH] In that case, if I remember correctly,
it wasn't the photographer, it was the art director
who "improved."
[FF] They moved the pyramids so that the typeface
could be dropped more properly as a full cover design.
That was a long time ago.
It's dating us, Brian, I'm afraid.
But those questions still arise.
[BH] Of course they do, yes.
And it's easier to do it in the photograph
than it is to move the pyramid itself.
[FF] As of today, yes, I would have to say that.
So we are going to be discussing this in week five,
how far can we go?
[BH] It's a fascinating issue with a very long history.
[FF] Yeah.
Oh, yeah.
[BH] And it is-- I think it's something that individuals,
they need to make their own ethical choices,
but obviously if you're working in a social setting
where there are other people making the rules,
you also need to be aware of that.
[FF] There's a whole issue as we talk
about the whole notion of manipulation and how far
can one go.
And certainly in science imaging,
there are very serious restrictions,
and there should be.
It isn't all about the data, but the question
is defining what the data is.
The fact that you're making an image with a tool,
you're already changing the data.
We have to remember, and we'll go back to this later,
but making a picture is making a representation.
It is not the thing, it is a re- presentation
and there is some sort of decision-making involved.
The thing that I'm trying to emphasize in the course
is that we have to create a literacy, in a way,
of what makes for a good picture.
And you and I might disagree on that, and we have, I think.
And I hope that students come to their own decisions
about what is better than that or less than that
and what is good.
But I think we have to start somewhere,
and the "somewhere" for us is to create an image that
well communicates what it is that this device or Petri dish
or thing that we're photographing-- what picture
can best communicate the structure?
We're generally interested in structure.
And if we are talking about process,
for example, as we do when we make photo illustrations,
we have examples later on that the students will
see how I've used bits and pieces of photographs,
put it all together for an illustration.
So it's not a documentary image and there's a place for that
in science.
But in the end, it's about getting the finest quality
to communicate the science.
That is what we're doing here in this course.
And by the way, we will be discussing using
tablets and mobile devices.
Very tangentially, but there's a place for that.
Absolutely.
When you really, really need to take a picture quickly and send
it to your colleague to show them that you have evidence
of what you've produced.
But as far as making the finest quality image,
we still are going to-- at this point--
stick to the camera and the 105 lens.
That's what we're sticking to.
[BH] I wouldn't try to argue otherwise.
I think the way people will think
of photography and of pictures in years
to come may be rather different from our view.
It's certainly changed within my lifetime, when photographs
were physical objects that you put in scrapbooks
or filed away.
We don't think of them in those terms anymore.
And I think they may change their nature again
because they are so ubiquitous.
But that doesn't really have much to do
with the kinds of photography that
are involved in, say, scientific documentation
or scientific illustration.
In art photography, you never know what's going to happen.
And there are lots of artists, I think,
who are now working with cell phones and such,
and even with the plastic lens Lomo cameras and things
that deliberately distort the view of the world.
So I think there's lots of latitude
for different approaches to this.
But for serious documentary work or for trying
to create the most vivid image that you possibly can
of something that holds still for you
while you point the camera at it, clearly
you're going to use the best tool that you can come up with.
Even mediocre ones today, I think,
are considerably better than the best ones of a few decades ago.
They're probably going to get better still.