Lecture 6: Multicore Programming

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

JULIAN SHUN: Today, we're going to talk about multicore programming. And as I was just informed by Charles, it's 2018. I had 2017 on the slide.

So first, congratulations to all of you. You turned in the first project's data. Here's a plot showing the tiers that different groups reached for the beta. And this is in sorted order. And we set the beta cutoff to be tier 45. The final cutoff is tier 48. So the final cutoff we did set a little bit aggressively, but keep in mind that you don't necessarily have to get to the final cutoff in order to get an A on this project.

So we're going to talk about multicore processing today. That's going to be the topic of the next project after you finish the first project. So in a multicore processor, we have a whole bunch of cores that are all placed on the same chip, and they have access to shared memory. They usually also have some sort of private cache, and then a shared last level cache, so L3, in this case. And then they all have access the same memory controller, which goes out to main memory. And then they also have access to I/O.

But for a very long time, chips only had a single core on them. So why do we have multicore processors nowadays? Why did semiconductor vendors start producing chips that had multiple processor cores on them?

So the answer is because of two things. So first, there's Moore's Law, which says that we get more transistors every year. So the number of transistors that you can fit on a chip doubles approximately every two years.

And secondly, there's the end of scaling of clock frequency. So for a very long time, we could just keep increasing the frequency of the single core on the chip. But at around 2004 to 2005, that was no longer the case. We couldn't scale the clock frequency anymore.

So here's a plot showing both the number of transistors you could fit on the chip over time, as well as the clock frequency of the processors over time. And notice that the y-axis is in log scale here. And the blue line is basically Moore's Law, which says that the number of transistors you can fit on a chip doubles approximately every two years.

And that's been growing pretty steadily. So this plot goes up to 2010, but in fact, it's been growing even up until the present. And it will continue to grow for a couple more years before Moore's Law ends.

However, if you look at the clock frequency line, you see that it was growing quite steadily until about the early 2000s, and then at that point, it flattened out. So at that point, we couldn't increase the clock frequencies anymore, and the clock speed was bounded at about four gigahertz.

So nowadays, if you go buy a processor, it's usually still bounded by around 4 gigahertz. It's usually a little bit less than 4 gigahertz, because it doesn't really make sense to push it all the way. But you might find some processors that are around 4 gigahertz nowadays.

So what happened at around 2004 to 2005? Does anyone know? So Moore's Law basically says that we can fit more transistors on a chip because the transistors become smaller. And when the transistors become smaller, you can reduce the voltage that's needed to operate the transistors.

And as a result, you can increase the clock frequency while maintaining the same power density. And that's what manufacturers did until about 2004 to 2005. They just kept increasing the clock frequency to take advantage of Moore's law. But it turns out that once transistors become small enough, and the voltage used to operate them becomes small enough, there's something called leakage current.

So there's current that leaks, and we're unable to keep reducing the voltage while still having reliable switching. And if you can't reduce the voltage anymore, then you can't increase the clock frequency if you want to keep the same power density.

So here's a plot from Intel back in 2004 when they first started producing multicore processors. And this is plotting the power density versus time. And again, the y-axis is in log scale here. So the green data points are actual data points, and the orange ones are projected. And they projected what the power density would be if we kept increasing the clock frequency at a trend of about 25% to 30% per year, which is what happened up until around 2004.

And because we couldn't reduce the voltage anymore, the power density will go up. And you can see that eventually, it reaches the power density of a nuclear reactor, which is pretty hot. And then it reaches the power density of a rocket nozzle, and eventually you get to the power density of the sun's surface.

So if you have a chip that has a power density equal to the sun's surface-- well, you don't actually really have a chip anymore. So basically if you get into this orange region, you basically have a fire, and you can't really do anything interesting, in terms of performance engineering, at that point.

So to solve this problem, semiconductor vendors didn't increased the clock frequency anymore, but we still had Moore's Law giving us more and more transistors every year. So what they decided to do with these extra transistors was to put them into multiple cores, and then put multiple cores on the same chip.

So we can see that, starting at around 2004, the number of cores per chip becomes more than one. And each generation of Moore's Law will potentially double the number of cores that you can fit on a chip, because it's doubling the number of transistors. And we've seen this trend up until about today. And again, it's going to continue for a couple more years before Moore's Law ends. So that's why we have chips with multiple cores today.

So today, we're going to look at multicore processing. So I first want to introduce the abstract multicore architecture. So this is a very simplified version, but I can fit it on this slide, and it's a good example for illustration. So here, we have a whole bunch of processors. They each have a cache, so that's indicated with the dollar sign. And usually they have a private cache as well as a shared cache, so a shared last level cache, like the L3 cache. And then they're all connected to the network.

And then, through the network, they can connect to the main memory. They can all access the same shared memory. And then usually there's a separate network for the I/O as well, even though I've drawn them as a single network here, so they can access the I/O interface. And potentially, the network will also connect to other multiprocessors on the same system. And this abstract multicore architecture is known as a chip multiprocessor, or CMP. So that's the architecture that we'll be looking at today.

So here's an outline of today's lecture. So first, I'm going to go over some hardware challenges with shared memory multicore machines. So we're going to look at the cache coherence protocol. And then after looking at hardware, we're going to look at some software solutions to write parallel programs on these multicore machines to take advantage of the extra cores.

And we're going to look at several concurrency platforms listed here. We're going to look at Pthreads. This is basically a low-level API for accessing, or for running your code in parallel. And if you program on Microsoft products, the Win API threads is pretty similar. Then there's Intel Threading Building Blocks, which is a library solution to concurrency. And then there are two linguistic solutions that we'll be looking at-- OpenMP and Cilk Plus. And Cilk Plus is actually the concurrency platform that we'll be using for most of this class.

So let's look at how caches work. So let's say that we have a value in memory at some location, and that value is-- let's say that value is x equals 3. If one processor says, we want to load x, what happens is that processor reads this value from a main memory, brings it into its own cache, and then it also reads the value, loads it into one of its registers. And it keeps this value in cache so that if it wants to access this value again in the near future, it doesn't have to go all the way out to main memory. It can just look at the value in its cache.

Now, what happens if another processor wants to load x? Well, it just does the same thing. It reads the value from main memory, brings it into its cache, and then also loads it into one of the registers. And then same thing with another processor.

It turns out that you don't actually always have to go out to main memory to get the value. If the value resides in one of the other processor's caches, you can also get the value through the other processor's cache. And sometimes that's cheaper than going all the way out to main memory.

So the second processor now loads x again. And it's in cache, so it doesn't have to go to main memory or anybody else's cache. So what happens now if we want to store x, if we want to set the value of x to something else?

So let's say this processor wants to set x equal to 5. So it's going to write x equals 5 and store that result in its own cache. So that's all well and good. Now what happens when the first processor wants to load x? Well, it seems that the value of x is in its own cache, so it's just going to read the value of x there, and it gets a value of 3.

So what's the problem there? Yes?

AUDIENCE: The path is stale.

JULIAN SHUN: Yeah. So the problem is that the value of x in the first processor's cache is stale, because another processor updated it. So now this value of x in the first processor's cache is invalid.

So that's the problem. And one of the main challenges of multicore hardware is to try to solve this problem of cache coherence-- making sure that the values in different processors' caches are consistent across updates.

So one basic protocol for solving this problem is known as the MSI protocol. And in this protocol, each cache line is labeled with a state. So there are three possible states-- M, S, and I. And this is done on the granularity of cache lines.

Because it turns out that storing this information is relatively expensive, so you don't want to store it for every memory location. So they do it on a per cache line basis. Does anyone know what the size of a cache line is, on the machines that we're using? Yeah?

AUDIENCE: 64 bytes.

JULIAN SHUN: Yeah, so it's 64 bytes. And that's typically what you see today on most Intel and AMD machines. There's some architectures that have different cache lines, like 128 bytes. But for our class, the machines that we're using will have 64 byte cache lines. It's important to remember that so that when you're doing back-of-the-envelope calculations, you can get accurate estimates.

So the three states in the MSI protocol are M, S, and I. So M stands for modified. And when a cache block is in the modified state, that means no other caches can contain this block in the M or the S states.

The S state means that the block is shared, so other caches can also have this block in shared state.

And then finally, I mean the cache block is invalid. So that's essentially the same as the cache block not being in the cache.

And to solve the problem of cache coherency, when one cache modifies a location, it has to inform all the other caches that their values are now stale, because this cache modified the value. So it's going to invalidate all of the other copies of that cache line in other caches by changing their state from S to I.

So let's see how this works. So let's say that the second processor wants to store y equals 5. So previously, a value of y was 17, and it was in shared state. The cache line containing y equals 17 was in shared state.

So now, when I do y equals 5, I'm going to set the second processor's cache-- that cache line-- to modified state. And then I'm going to invalidate the cache line in all of the other caches that contain that cache line. So now the first cache and the fourth cache each have a state of I for y equals 17, because that value is stale.

Is there any questions? Yes?

AUDIENCE: If we already have to tell the other things to switch to invalid, why not just tell them the value of y?

JULIAN SHUN: Yeah, so there are actually some protocols that do that. So this is just the most basic protocol. So this protocol doesn't do it. But there are some that are used in practice that actually do do that. So it's a good point. But I just want to present the most basic protocol for now. Sorry.

And then, when you load a value, you can first check whether your cache line is in M or S state. And if it is an M or S state, then you can just read that value directly. But if it's in the I state, or if it's not there, then you have to fetch that block from either another processor's cache or fetch it from main memory.

So it turns out that there are many other protocols out there. There's something known as MESI, the messy protocol. There's also MOESI and many other different protocols. And some of them are proprietary. And they all do different things. And it turns out that all of these protocols are quite complicated, and it's very hard to get these protocols right. And in fact, one of the most earliest successes of formal verification was improving some of these cache [INAUDIBLE] protocols to be correct. Yes, question?

AUDIENCE: What happens if two processors try to modify one value at the same time

JULIAN SHUN: Yeah, so if two processors try to modify the value, one of them has to happen first. So the hardware is going to take care of that. So the first one that actually modifies it will invalidate all the other copies, and then the second one that modifies the value will again invalidate all of the other copies.

And when you do that-- when a lot of processors try to modify the same value, you get something known as an invalidation storm. So you have a bunch of invalidation messages going throughout the hardware. And that can lead to a big performance bottleneck. Because each processor, when it modifies its value, it has to inform all the other processors.

And if all the processors are modifying the same value, you get this sort of quadratic behavior. The hardware is still going to guarantee that one of their processors is going to end up writing the value there. But you should be aware of this performance issue when you're writing parallel code. Yes?

AUDIENCE: So all of this protocol stuff happens in hardware?

JULIAN SHUN: Yes, so this is all implemented in hardware. So if you take a computer architecture class, you'll learn much more about these protocols and all of their variants. So for our purposes, we don't actually need to understand all the details of the hardware. We just need to understand what it's doing at a high level so we can understand when we have a performance bottleneck and why we have a performance bottleneck. So that's why I'm just introducing the most basic protocol here. Any other questions?

So I talked a little bit about the shared memory hardware. Let's now look at some concurrency platforms. So these are the four platforms that we'll be looking at today.

So first, what is a concurrency platform? Well, writing parallel programs is very difficult. It's very hard to get these programs to be correct. And if you want to optimize their performance, it becomes even harder. So it's very painful and error-prone. And a concurrency platform abstracts processor cores and handles synchronization and communication protocols. And it also performs load balancing for you. So it makes your lives much easier. And so today we're going to talk about some of these different concurrency platforms.

So to illustrate these concurrency platforms, I'm going to do the Fibonacci numbers example. So does anybody not know what Fibonacci is? So good. Everybody knows what Fibonacci is. So it's a sequence where each number is the sum of the previous two numbers. And the recurrence is shown in this brown box here.

The sequence is named after Leonardo di Pisa, who was also known as Fibonacci, which is a contraction of Bonacci, son of Bonaccio. So that's where the name Fibonacci came from. And in Fibonacce's 1202 book, Liber Abaci, he introduced the sequence-- the Fibonacci sequence-- to Western mathematics, although it had been previously known to Indian mathematicians for several centuries. But this is what we call the sequence nowadays-- Fibonacci numbers.

So here's a Fibonacci program. Has anyone seen this algorithm before? A couple of people. Probably more, but people didn't raise their hands.

So it's a recursive program. So it basically implements the recurrence from the previous slide. So if n is less than 2, we just return n. Otherwise, we compute fib of n minus 1, store that value in x, fib of n minus 2, store that value in y, and then return the sum of x and y.

So I do want to make a disclaimer to the algorithms police that this is actually a very bad algorithm. So this algorithm takes exponential time, and there's actually much better ways to compute the end Fibonacci number. There's a linear time algorithm, which just computes the Fibonacci numbers from bottom up.

This algorithm here is actually redoing a lot of the work, because it's computing Fibonacci numbers multiple times. Whereas if you do a linear scan from the smallest numbers up, you only have to compute each one once. And there's actually an even better algorithm that takes logarithmic time, and it's based on squaring matrices. So has anyone seen that algorithm before? So a couple of people. So if you're interested in learning more about this algorithm, I encourage you to look at your favorite textbook, Introduction to Algorithms by Cormen, Leiserson, Rivest, and Stein.

So even though this--

[LAUGHTER]

Yes. So even though this is a pretty bad algorithm, it's still a good educational example, because I can fit it on one slide and illustrate all the concepts of parallelism that we want to cover today.

So here's the execution tree for fib of 4. So we see that fib of 4 is going to call fib of 3 and fib of 2. Fib of 3 is going to call fib of 2, fib of 1, and so on. And you can see that repeated computations here. So fib of 2 is being computed twice, and so on. And if you have a much larger tree-- say you ran this on fib of 40-- then you'll have many more overlapping computations.

It turns out that the two recursive calls can actually be parallelized, because they're completely independent calculations. So the key idea for parallelization is to simultaneously execute the two recursive sub-calls to fib. And in fact, you can do this recursively. So the two sub-calls to fib of 3 can also be executed in parallel, and the two sub-calls of fib of 2 can also be executed in parallel, and so on. So you have all of these calls that can be executed in parallel. So that's the key idea for extracting parallelism from this algorithm.

So let's now look at how we can use Pthreads to implement this simple Fibonacci algorithm. So Pthreads is a standard API for threading, and it's supported on all Unix-based machines. And if you're programming using Microsoft products, then the equivalent is Win API threads.

And Pthreads is actually standard in ANSI and IEEE, so there's this number here that specifies the standard. But nowadays, we just call it Pthreads. And it's basically a do-it-yourself concurrency platform. So it's like the assembly language of parallel programming. It's built as a library of functions with special non-C semantics. Because if you're just writing code in C, you can't really say which parts of the code should be executed in parallel. So Pthreads provides you a library of functions that allow you to specify concurrency in your program.

And each thread implements an abstraction of a processor, and these threads are then multiplexed onto the actual machine resources. So the number of threads that you create doesn't necessarily have to match the number of processors you have on your machine. So if you have more threads than the number of processors you have, then they'll just be multiplexing. So you can actually run a Pthreads program on a single core even though you have multiple threads in the program. They would just be time-sharing.

All the threads communicate through shared memory, so they all have access to the same view of the memory. And the library functions that Pthreads provides mask the protocols involved in interthread coordination, so you don't have to do it yourself. Because it turns out that this is quite difficult to do correctly by hand.

So now I want to look at the key Pthread functions. So the first Pthread is pthread_create. And this takes four arguments. So the first argument is this pthread_t type. This is basically going to store an identifier for the new thread that pthread_create will create so that we can use that thread in our computations.

pthread_attr_t-- this set some thread attributes, and for our purposes, we can just set it to null and use the default attributes.

The third argument is this function that's going to be executed after we create the thread. So we're going to need to define this function that we want the thread to execute.

And then finally, we have this void *arg argument, which stores the arguments that are going to be passed to the function that we're going to be executing.

And then pthread_create also returns an error status, returns an integer specifying whether the thread creation was successful or not.

And then there's another function called pthread_join. pthread_join basically says that we want to block at this part of our code until this specified thread finishes. So it takes as argument pthread_t.

So this thread identifier, and these thread identifiers, were created when we called pthread_create. It also has a second argument, status, which is going to store the status of the terminating thread. And then pthread_join also returns an error status. So essentially what this does is it says to wait until this thread finishes before we continue on in our program.

So any questions so far?

So here's what the implementation of Fibonacci looks like using Pthreads. So on the left, we see the original program that we had, the fib function there. That's just the sequential code. And then we have all this other stuff to enable it to run in parallel. So first, we have this struct on the left, thread_args. This struct here is used to store the arguments that are passed to the function that the thread is going to execute.

And then we have this thread_func. What that does is it reads the input argument from this thread_args struct, and then it sets that to i, and then it calls fib of i. And that gives you the output, and then we store the result into the output of the struct. And then that also just returns null.

And then over on the right hand side, we have the main function that will actually call the fib function on the left. So we initialize a whole bunch of variables that we need to execute these threads. And then we first check if n is less than 30. If n is less than 30, it turns out that it's actually not worth creating threads to execute this program in parallel, because of the overhead of thread creation. So if n is less than 30, we'll just execute the program sequentially.

And this idea is known as coarsening. So you saw a similar example a couple of lectures ago when we did coarsening for sorting. But this is in the context of a parallel programming. So here, because there are some overheads to running a function in parallel, if the input size is small enough, sometimes you want to just execute it sequentially.

And then we're going to-- so let me just walk through this code, since I have an animation. So the next thing it's going to do is it's going to marshal the input argument to the thread so it's going to store the input argument n minus 1 in this args struct.

And then we're going to call pthread_create with a thread variable. For thread_args, we're just going to use null. And then we're going to pass the thread_func that we defined on the left. And then we're going to pass the args structure. And inside this args structure, the input is set to n minus 1, which we did on the previous line.

And then pthread_create is going to give a return value. So if the Pthread creation was successful, then the status is going to be null, and we can continue. And when we continue, we're going to execute, now, fib of n minus 2 and store the result of that into our result variable. And this is done at the same time that fib of n minus 1 is executing. Because we created this Pthread, and we told it to call this thread_func function that we defined on the left. So both fib of n minus 1 and fib of n minus 2 are executing in parallel now.

And then we have this pthread_join, which says we're going to wait until the thread that we've created finishes before we move on, because we need to know the result of both of the sub-calls before we can finish this function. And once that's done-- well, we first check the status to see if it was successful. And if so, then we add the outputs of the argument's struct to the result. So args.output will store the output of fib of n minus 1.

So that's the Pthreads code. Any questions on how this works? Yeah?

AUDIENCE: I have a question about the thread function. So it looks like you passed a void pointer, but then you cast it to something else every time you use that--

JULIAN SHUN: Yeah, so this is because the pthread_create function takes as input a void star pointer. Because it's actually a generic function, so it doesn't know what the data type is. It has to work for all data types, and that's why we need to cast it to avoid star. When we pass it to pthread_create and then inside the thread_func, we actually do know what type of pointer that is, so then we cast it.

So does this code seem very parallel? So how many parallel calls am I doing here? Yeah?

AUDIENCE: Just one.

JULIAN SHUN: Yeah, so I'm only creating one thread. So I'm executing two things in parallel. So if I ran this code on four processors, what's the maximum speed-up I could get?

AUDIENCE: [INAUDIBLE].

JULIAN SHUN: So the maximum speed-up I can get is just two, because I'm only running two things in parallel. So this doesn't recursively create threads. It only creates one thread at the top level.

And if you wanted to make it so that this code actually recursively created threads, it would actually become much more complicated. And that's one of the disadvantages of implementing this code in Pthreads. So we'll look at other solutions that will make this task much easier.

So some of the issues with Pthreads are shown on this slide here. So there's a high overhead to creating a thread. So creating a thread typically takes over 10 to the 4th cycles. And this leads to very coarse-grained concurrency, because your tasks have to do a lot of work in order to amortize the costs of creating that thread.

There are something called thread pulls, which can help. And the idea here is to create a whole bunch of threads at the same time to amortize the costs of thread creation. And then when you need a thread, you just take one from the thread pull. So the thread pull contains threads that are just waiting to do work.

There's also a scalability issue with this code that I showed on the previous slide. The Fibonacci code gets, at most, 1.5x speed-up for two cores. Why is it 1.5 here? Does anyone know? Yeah?

AUDIENCE: You have the asymmetry in the size of the two calls.

JULIAN SHUN: Yeah, so it turns out that the two calls that I'm executing in parallel-- they're not doing the same amount of work. So one is computing fib of n minus 1, one is computing fib of n minus 2. And does anyone know what the ratio between these two values is? Yeah, so it's the golden ratio. It's about 1.6. It turns out that if you can get a speed-up of 1.6, then that's great. But there are some overheads, so this code will get about a 1.5 speed up.

And if you want to run this to take advantage of more cores, then you need to rewrite this code, and it becomes more complicated.

Third, there's the issue of modularity. So if you look at this code here, you see that the Fibonacci logic is not nicely encapsulated within one function. We have that logic in the fib function on the left, but then we also have some of the fib logic on the right in our main function. And this makes this code not modular.

And if we want to build programs on top of this, it makes it very hard to maintain, if we want to just change the logic of the Fibonacci function a little bit, because now we have to change it in multiple places instead of just having everything in one place. So it's not a good idea to write code that's not modular, so please don't do that in your projects.

And then finally, the code becomes complicated because you have to actually move these arguments around. That's known as argument marshaling. And then you have to engage in error-prone protocols in order to do load balancing. So if you recall here, we have to actually place the argument n minus 1 into args.input and we have to extract the value out of args.output. So that makes the code very messy.

So why do I say shades of 1958 here? Does anyone know what happened in 1958? Who was around in 1958? Just Charles? So there was a first something in 1958. What was it?

So turns out in 1958, we had the first compiler. And this was the Fortran compiler. And before we had Fortran compiler, programmers were writing things in assembly. And when you write things in assembly, you have to do argument marshaling, because you have to place things into the appropriate registers before calling a function, and also move things around when you return from a function.

And the nice thing about the first compiler is that it actually did all of this argument marshaling for you. So now you can just pass arguments to a function, and the compiler will generate code that will do the argument marshaling for us.

So having you do this in Pthreads is similar to having to write code in assembly, because you have to actually manually marshal these arguments. So hopefully, there are better ways to do this. And indeed, we'll look at some other solutions that will make it easier on the programmer.

Any questions before I continue?

So we looked at Pthreads. Next, let's look at Threading Building Blocks. So Threading Building Blocks is a library solution. It was developed by Intel. And it's implemented as a C++ library that runs on top of native threads. So the underlying implementation uses threads, but the programmer doesn't deal with threads. Instead, the programmer specifies tasks, and these tasks are automatically load-balanced across the threads using a work-stealing algorithm inspired by research at MIT-- Charles Leiserson's research.

And the focus of Intel TBB is on performance. And as we'll see, the code written using TBB is simpler than what you would have to write if you used Pthreads. So let's look at how we can implement Fibonacci using TBB.

So in TBB, we have to create these tasks. So in the Fibonacci code, we create this fib task class. And inside the task, we have to define this execute function. So the execute function is the function that performs a computation when we start the task. And this is where we define the Fibonacci logic. This task also takes as input these arguments parameter, n and sum. So n is the input here and sum is the output.

And in TBB, we can easily create a recursive program that extracts more parallelism. And here, what we're doing is we're recursively creating two child tasks, a and b. That's the syntax for creating the tasks. And here, we can just pass the arguments to FibTask instead of marshaling the arguments ourselves.

And then what we have here is a set_ref_count. And this basically is the number of tasks that we have to wait for plus one, so plus one for ourselves. And in this case, we created two children tasks, and we have ourselves, so that's 2 plus 1.

And then after that, we start task b using the spawn(b) call. And then we do spawn_and_wait_for_all with a as the argument. In this place, he says, we're going to start task a, and then also wait for both a and b to finish before we proceed. So this spawn_and_wait_for_all call is going to look at the ref count that we set above and wait for that many tasks to finish before it continues.

And after both and a and b have completed, then we can just sum up the results and store that into the sum variable. And here, these tasks are created recursively. So unlike the Pthreads implementation that was only creating one thread at the top level, here, we're actually recursively creating more and more tasks. So we can actually get more parallelism from this code and scale to more processors.

We also need this main function just to start up the program. So what we do here is we create a root task, which just computes fib of n, and then we call spawn_root_and_wait(a). So a is the task for the root. And then it will just run the root task.

So that's what Fibonacci looks like in TBB. So this is much simpler than the Pthreads implementation. And it also gets better performance, because we can extract more parallelism from the computation. Any questions?

So TBB also has many other features in addition to tasks. So TBB provides many C++ templates to express common patterns, and you can use these templates on different data types. So they have a parallel_for, which is used to express loop parallelism. So you can loop over a bunch of iterations in parallel.

They also have a parallel_reduce for data aggregation. For example, if you want to sum together a whole bunch of values, you can use a parallel_reduce to do that in parallel. They also have pipeline and filter. That's used for software pipelining.

TBB provides many concurrent container classes, which allow multiple threads to safely access and update the items in a container concurrently. So for example, they have hash tables, trees, priority cues, and so on. And you can just use these out of the box, and they'll work in parallel. You can do concurrent updates and reads to these data structures.

TBB also has a variety of mutual exclusion library functions, such as locks and atomic operations. So there are a lot of features of TBB, which is why it's one of the more popular concurrency platforms. And because of all of these features, you don't have to implement many of these things by yourself, and still get pretty good performance.

So TBB was a library solution to the concurrency problem. Now we're going to look at two linguistic solutions-- OpenMP and Cilk. So let's start with OpenMP.

So OpenMP is a specification by an industry consortium. And there are several compilers available that support OpenMP, both open source and proprietary. So nowadays, GCC, ICC, and Clang all support OpenMP, as well as Visual Studio. And OpenMP is-- it provides linguistic extensions to C and C++, as well as Fortran, in the form of compiler pragmas.

So you use these compiler pragmas in your code to specify which pieces of code should run in parallel. And OpenMP also runs on top of native threads, but the programmer isn't exposed to these threads. OpenMP supports loop parallelism, so you can do parallel for loops. They have task parallelism as well as pipeline parallelism.

So let's look at how we can implement Fibonacci in OpenMP. So this is the entire code. So I want you to compare this to the Pthreads implementation that we saw 10 minutes ago. So this code is much cleaner than the Pthreads implementation, and it also performs better. So let's see how this code works.

So we have these compiler pragmas, or compiler directives. And the compiler pragma for creating a parallel task is omp task. So we're going to create an OpenMP task for fib of n minus 1 as well as fib of n minus 2. There's also this shared pragma, which specifies that the two variables in the arguments are shared across different threads. So you also have to specify whether variables are private or shared.

And then the pragma omp wait just says we're going to wait for the preceding task to complete before we continue. So here, it's going to wait for fib of n minus 1 and fib of n minus 2 to finish before we return the result, which is what we want. And then after that, we just return x plus y. So that's the entire code.

And OpenMP also provides many other pragma directives, in addition to task. So we can use a parallel for to do loop parallelism. There's reduction. There's also directives for scheduling and data sharing. So you can specify how you want a particular loop to be scheduled. OpenMP has many different scheduling policies. They have static parallelism, dynamic parallelism, and so on. And then these scheduling directives also have different grain sizes. The data sharing directives are specifying whether variables are private or shared.

OpenMP also supplies a variety of synchronization constructs, such as barriers, atomic updates, mutual exclusion, or mutex locks. So OpenMP also has many features, and it's also one of the more popular solutions to writing parallel programs. As you saw in the previous example, the code is much simpler than if you were to write something using Pthreads or even TBB. This is a much simpler solution.

Any questions? Yeah?

AUDIENCE: So with every compiler directive, does it spawn a new [INAUDIBLE] on a different processor?

JULIAN SHUN: So this code here is actually independent of the number of processors. So there is actually a scheduling algorithm that will determine how the tasks get mapped to different processors. So if you spawn a new task, it doesn't necessarily put it on a different processor. And you can have more tasks than the number of processors available. So there's a scheduling algorithm that will take care of how these tasks get mapped to different processors, and that's hidden from the programmer. Although you can use these scheduling pragmas to give hints to the compiler how it should schedule it. Yeah?

AUDIENCE: What is the operating system [INAUDIBLE] scheduling [INAUDIBLE]?

JULIAN SHUN: Underneath, this is implemented using Pthreads, which has to make operating system calls to, basically, directly talk to the processor cores and do multiplexing and so forth. So the operating system is involved at a very low level.

So the last concurrency platform that we'll be looking at today is Cilk. We're going to look at Cilk Plus, actually. And the Cilk part of Cilk Plus is a small set of linguistic extensions to C and C++ that support fork-join parallelism. So for example, the Fibonacci example uses fork-join parallelism, so you can use Cilk to implement that. And the Plus part of Cilk Plus supports vector parallelism, which you had experience working with in your homeworks.

So Cilk Plus was initially developed by Cilk Arts, which was an MIT spin-off. And Cilk Arts was acquired by Intel in July 2009. And the Cilk Plus implementation was based on the award-winning Cilk multi-threaded language that was developed two decades ago here at MIT by Charles Leiserson's research group.

And it features a provably efficient work-stealing scheduler. So this scheduler is provably efficient. You can actually prove theoretical bounds on it. And this allows you to implement theoretically efficient algorithms, which we'll talk more about in another lecture-- algorithm design. But it provides a provably efficient work-stealing scheduler. And Charles Leiserson has a very famous paper that has a proof of that this scheduler is optimal. So if you're interested in reading about this, you can talk to us offline.

Cilk Plus also provides a hyperobject library for parallelizing code with global variables. And you'll have a chance to play around with hyperobjects in homework 4.

The Cilk Plus ecosystem also includes useful programming tools, such as the Cilk Screen Race Detector. So this allows you to detect determinacy races in your program to help you isolate bugs and performance bottlenecks. It also has a scalability analyzer called Cilk View. And Cilk View will basically analyze the amount of work that your program is doing, as well as the maximum amount of parallelism that your code could possibly extract from the hardware.

So that's Intel Cilk Plus. But it turns out that we're not actually going to be using Intel Cilk Plus in this class. We're going to be using a better compiler. And this compiler is based on Tapir/LLVM. And it supports the Cilk subset of Cilk Plus. And Tapir/LLVM was actually recently developed at MIT by T. B. Schardl, who gave a lecture last week, William Moses, who's a grad student working with Charles, as well as Charles Leiserson. So talking a lot about Charles's work today.

And Tapir/LLVM generally produces better code, relative to its base compiler, than all other implementations of Cilk out there. So it's the best Cilk compiler that's available today. And they actually wrote a very nice paper on this last year, Charles Leiserson and his group. And that paper received the Best Paper Award at the annual Symposium on Principles and Practices of Parallel Programming, or PPoPP. So you should look at that paper as well.

So right now, Tapir/LLVM uses the Intel Cilk Plus runtime system, but I believe Charles's group has plans to implement a better runtime system. And Tapir/LLVM also supports more general features than existing Cilk compilers. So in addition to spawning functions, you can also spawn code blocks that are not separate functions, and this makes writing programs more flexible. You don't have to actually create a separate function if you want to execute a code block in parallel.

Any questions?

So this is the Cilk code for Fibonacci. So it's also pretty simple. It looks very similar to the sequential program, except we have these cilk_spawn and cilk_synv statements in the code. So what do these statements do?

So cilk_spawn says that the named child function, which is the function that is right after this cilk_spawn statement, may execute in parallel with the parent caller. The parent caller is the function that is calling cilk_spawn. So this says that fib of n minus 1 can execute in parallel with the function that called it. And then this function is then going to call fib of n minus 2. And fib of n minus 2 and fib of n minus 1 now can be executing in parallel.

And then cilk_sync says that control cannot pass this point until all of the spawn children have returned. So this is going to wait for fib of n minus 1 to return before we go to the return statement where we add up x and y.

So one important thing to note is that the Cilk keywords grant permission for parallel execution, but they don't actually force or command parallel execution. So even though I said cilk_spawn here, the runtime system doesn't necessarily have to run fib of n minus 1 in parallel with fib of n minus 2. I'm just saying that I could run these two things in parallel, and it's up to the runtime system to decide whether or not to run these things in parallel, based on its scheduling policy.

So let's look at another example of Cilk. So let's look at loop parallelism. So here we want to do a matrix transpose, and we want to do this in-place. So the idea here is we want to basically swap the elements below the diagonal to its mirror image above the diagonal.

And here's some code to do this. So we have a cilk_for. So this is basically a parallel for loop. It goes from i equals 1 to n minus 1. And then the inner for loop goes from j equals 0 up to i minus 1. And then we just swap a of i j with a of j i, using these three statements inside the body of the for loop. So to execute a for loop in parallel, you just have to add cilk underscore to the for keyword. And that's as simple as it gets.

So this code is actually going to run in parallel and get pretty good speed-up for this particular problem. And internally, Cilk for loops are transformed into nested cilk_spawn and cilk_sync calls. So the compiler is going to get rid of the cilk_for and change it into cilk_spawn and cilk_sync.

So it's going to recursively divide the iteration space into half, and then it's going to spawn off one half and then execute the other half in parallel with that, and then recursively do that until the iteration range becomes small enough, at which point it doesn't make sense to execute it in parallel anymore, so we just execute that range sequentially.

So that's loop parallelism in Cilk. Any questions? Yes?

AUDIENCE: How does it know [INAUDIBLE] something weird, can it still do that?

JULIAN SHUN: Yeah, so the compiler can actually figure out what the iteration space is. So you don't necessarily have to be incrementing by 1. You can do something else. You just have to guarantee that all of the iterations are independent. So if you have a determinacy race across the different iterations of your cilk_for loop, then your result might not necessarily be correct. So you have to make sure that the iterations are, indeed, independent.

Yes?

AUDIENCE: Can you nest cilk_fors?

JULIAN SHUN: Yes, so you can nest cilk_fors. But it turns out that, for this example, usually, you already have enough parallelism in the outer loop for large enough values of n, so it doesn't make sense to put a cilk_for loop inside, because using a cilk_for loop adds some additional overheads. But you can actually do nested cilk_for loops. And in some cases, it does make sense, especially if there's not enough parallelism in the outermost for loop. So good question.

Yes?

AUDIENCE: What does the assembly code look like for the parallel code?

JULIAN SHUN: So it has a bunch of calls to the Cilk runtime system. I don't know all the details, because I haven't looked at this recently. But I think you can actually generate the assembly code using a flag in the Clang compiler. So that's a good exercise.

AUDIENCE: Yeah, you probably want to look at the LLVM IR, rather than the assembly, to begin with, to understand what's going on. It has three instructions that are not in the standard LLVM, which were added to support parallelism. Those things, when it's lowered into assembly, each of those instructions becomes a bunch of assembly language instructions. So you don't want to mess with looking at it in the assembler until you see what it looks like in the LLVM first.

JULIAN SHUN: So good question. Any other questions about this code here? OK, so let's look at another example. So let's say we had this for loop where, on each iteration i, we're just incrementing a variable sum by i. So this is essentially going to compute the summation of everything from i equals 0 up to n minus 1, and then print out the result.

So one straightforward way to try to parallelize this code is to just change the for to cilk_for. So does this code work? Who thinks that this code doesn't work? Or doesn't compute the correct result? So about half of you. And who thinks this code does work? So a couple people. And I guess the rest of the people don't care.

So it turns out that it's not actually necessarily going to give you the right answer. Because the cilk_for loop says you can execute these iterations in parallel, but they're all updating the same shared variable sum here. So you have what's called a determinacy race, where multiple processors can be writing to the same memory location. We'll talk much more about determinacy races in the next lecture. But for this example, it's not necessarily going to work if you run it on more than one processor.

And Cilk actually has a nice way to deal with this. So in Cilk, we have something known as a reducer. This is one example of a hyperobject, which I mentioned earlier. And with a reducer, what you have to do is, instead of declaring the sum variable just has an unsigned long data type, what you do is you use this macro called CILK_C_REDUCER_OPADD, which specifies we want to create a reducer with the addition function.

Then we have the variable name sum, the data type unsigned long, and then the initial value 0. And then we have a macro to register this reducer, so a CILK_C_REGISTER_REDUCER. And then now, inside this cilk_for loop, we can increment the sum, or REDUCER_VIEW, of sum, which is another macro, by i.

And you can actually execute this in parallel, and it will give you the same answer that you would get if you ran this sequentially. So the reducer will take care of this determinacy race for you. And at the end, when you print out this result, you'll see that the sum is equal to the sum that you expect. And then after you finish using the reducer, you use this other macro called CILK_C_UNREGISTER_REDUCER(sum) that tells the system that you're done using this reducer.

So this is one way to deal with this problem when you want to do a reduction. And it turns out that there are many other interesting reduction operators that you might want to use. And in general, you can create reduces for monoids. And monoids are algebraic structures that have an associative binary operation as well as an identity element. So the addition operator is a monoid, because it's associative, it's binary, and the identity element is 0.

Cilk also has several other predefined reducers, including multiplication, min, max, and, or, xor, et cetera. So these are all monoids. And you can also define your own reducer. So in fact, in the next homework, you'll have the opportunity to play around with reducers and write a reducer for lists.

So that's reducers. Another nice thing about Cilk is that there's always a valid serial interpretation of the program. So the serial elision of a Cilk program is always a legal interpretation. And for the Cilk source code on the left, the serial elision is basically the code you get if you get rid of the cilk_spawn and cilk_sync statements. And this looks just like the sequential code.

And remember that the Cilk keywords grant permission for parallel execution, but they don't necessarily command parallel execution. So if you ran this Cilk code using a single core, it wouldn't actually create these parallel tasks, and you would get the same answer as the sequential program. And this is-- in the serial edition-- is also a correct interpretation.

So unlike other solutions, such as TBB and Pthreads, it's actually difficult, in those environments, to get a program that does what the sequential program does. Because they're actually doing a lot of additional work to set up these parallel calls and create these argument structures and other scheduling constructs. Whereas in Cilk, it's very easy just to get this serial elision. You just define cilk_spawn and cilk_sync to be null. You also define cilk_for to be for. And then this gives you a valid sequential program.

So when you're debugging code, and you might first want to check if the sequential elision of your Cilk program is correct, and you can easily do that by using these macros. Or actually, there's actually a compiler flag that will do that for you and give you the equivalent C program. So this is a nice way to debug, because you don't have to start with the parallel program. You can first check if this serial program is correct before you go on to debug the parallel program.

Questions? Yes?

AUDIENCE: So does cilk_for-- does each iteration of the cilk_for its own task that the scheduler decides if it wants to execute in parallel, or if it executes in parallel, do all of the iterations execute in parallel?

JULIAN SHUN: So it turns out that by default, it groups a bunch of iterations together into a single task, because it doesn't make sense to break it down into such small chunks, due to the overheads of parallelism. But there's actually a setting you can do to change the grain size of the for loop. So you could actually make it so that each iteration is its own task. And then, as you the scheduler will decide how to map these different task onto different processors, or even if it wants to execute any of these tasks in parallel. So good question.

So the idea in Cilk is to allow the programmer to express logical parallelism in an application. So the programmer just has to identify which pieces of the code could be executed in parallel, but doesn't necessarily have to determine which pieces of code should be executed in parallel.

And then Cilk has a runtime scheduler that will automatically map the executing program onto the available processor cores' runtime. And it does this dynamically using a work-stealing scheduling algorithm. And the work-stealing scheduler is used to balance the tasks evenly across the different processors. And we'll talk more about the work-stealing scheduler in a future lecture.

But I want to emphasize that unlike the other concurrency platforms that we looked at today, Cilk's work-stealing scheduling algorithm is theoretically efficient, whereas the OpenMP and TBB schedulers are not theoretically efficient. So this is a nice property, because it will guarantee you that the algorithms you write on top of Cilk will also be theoretically efficient.

So here's a high-level illustration of the Cilk ecosystem. It's a very simplified view, but I did this to fit it on a single slide. So what you do is you take the Cilk source code, you pass it to your favorite Cilk compiler-- the Tapir/LLVM compiler-- and this gives you a binary that you can run on multiple processors. And then you pass a program input to the binary, you run it on however many processors you have, and then this allows you to benchmark the parallel performance of your program.

You can also do serial testing. And to do this, you just obtain a serial elision of the Cilk program, and you pass it to an ordinary C or C++ compiler. It generates a binary that can only run on a single processor, and you run your suite of serial regression tests on this single threaded binary. And this will allow you to benchmark the performance of your serial code and also debug any issues that might have arised when you were running this program sequentially.

Another way to do this is you can actually just compile the original Cilk code but run it on a single processor. So there's a command line argument that tells the runtime system how many processors you want to use. And if you set that parameter to 1, then it will only use a single processor. And this allows you to benchmark the single threaded performance of your code as well. And the parallel program executing on a single core should behave exactly the same way as the execution of this serial elision. So that's one of the advantages of using Cilk.

And because you can easily do serial testing using the Cilk platform, this allows you to separate out the serial correctness from the parallel correctness. As I said earlier, you can first debug the serial correctness, as well as any performance issues, before moving on to the parallel version. And another point I want to make is that because Cilk actually uses the serial program inside its task, it's actually good to optimize the serial program even when you're writing a parallel program, because optimizing the serial program for performance will also translate to better parallel performance.

Another nice feature of Cilk is that it has this tool called Cilksan, which stands for Cilk Sanitizer. And Cilksan will detect any determinacy races that you have in your code, which will significantly help you with debugging the correctness as well as the performance of your code.

So if you compile the Cilk code using the Cilksan flag, it will generate an instrumented binary that, when you run, it will find and localize all the determinacy races in your program. So it will tell you where the determinacy races occur, so that you can go inspect that part of your code and fix it if necessary. So this is a very useful tool for benchmarking your parallel programs.

Cilk also has another nice tool called Cilkscale. Cilkscale is a performance analyzer. It will analyze how much parallelism is available in your program as well as the total amount of work that it's doing. So again, you pass a flag to the compiler that will turn on Cilkscale, and it will generate a binary that is instrumented. And then when you run this code, it will give you a scalability report.

So you'll find these tools very useful when you're doing the next project. And we'll talk a little bit more about these two tools in the next lecture. And as I said, Cilkscale will analyze how well your program will scale to larger machines. So it will basically tell you the maximum number of processors that your code could possibly take advantage of.

Any questions? Yes?

AUDIENCE: What do you mean when you say runtime?

JULIAN SHUN: So I mean the scheduler-- the Cilk runtime scheduler that's scheduling the different tasks when you're running the program.

AUDIENCE: So that's included in the binary.

JULIAN SHUN: So it's linked from the binary. It's not stored in the same place. It's linked.

Other questions?

So let me summarize what we looked at today. So first, we saw that most processors today have multiple cores. And probably all of your laptops have more than one core on it. Who has a laptop that only has one core?

AUDIENCE: [INAUDIBLE].

JULIAN SHUN: When did you buy it? Probably a long time ago.

So nowadays, obtaining high performance on your machines requires you to write parallel programs. But parallel programming can be very hard, especially if you have the program directly on the processor cores and interact with the operating system yourself. So Cilk is very nice, because it abstracts the processor cores from the programmer, it handles synchronization and communication protocols, and it also performs provably good load-balancing.

And in the next project, you'll have a chance to play around with Cilk. You'll be implementing your own parallel screensaver, so that's a very fun project to do. And possibly, in one of the future lectures, we'll post some of the nicest screensaver that students developed for everyone to see. OK, so that's all.