A better question might be: ''Why is parallel programming perceived to be so difficult?'' To see the answer, let's go back to the year 1991. Paul McKenney was walking across the parking lot to Sequent's benchmarking center carrying six dual-80486 Sequent Symmetry CPU boards, when he suddenly realized that he was carrying several times the price of the house he had just purchased.F.1 This high cost of parallel systems meant that parallel programming was restricted to a privileged few who worked for an employer who either manufactured or could afford to purchase machines costing upwards of $100,000 -- in 1991 dollars US.
In contrast, in 2006, Paul finds himself typing these words on a dual-core x86 laptop. Unlike the dual-80486 CPU boards, this laptop also contains 2GB of main memory, a 60GB disk drive, a display, Ethernet, USB ports, wireless, and Bluetooth. And the laptop is more than an order of magnitude cheaper than even one of those dual-80486 CPU boards, even before taking inflation into account.
Parallel systems have truly arrived. They are no longer the sole domain of a privileged few, but something available to almost everyone.
The earlier restricted availability of parallel hardware is the real reason that parallel programming is considered so difficult. After all, it is quite difficult to learn to program even the simplest machine if you have no access to it. Since the age of rare and expensive parallel machines is for the most part behind us, the age during which parallel programming is perceived to be mind-crushingly difficult is coming to a close.F.2
Quick Quiz .2:
How could parallel programming ever be as easy
as sequential programming?
Answer:
It depends on the programming environment.
SQL [Int92] is an underappreciated success
story, as it permits programmers who know nothing about parallelism
to keep a large parallel system productively busy.
We can expect more variations on this theme as parallel
computers continue to become cheaper and more readily available.
For example, one possible contender in the scientific and
technical computing arena is MATLAB*P,
which is an attempt to automatically parallelize common
matrix operations.
Finally, on Linux and UNIX systems, consider the following shell command:
get_input | grep "interesting" | sort
This shell pipeline runs the get_input, grep, and sort processes in parallel. There, that wasn't so hard, now was it?
Quick Quiz .3:
Oh, really???
What about correctness, maintainability, robustness, and so on?
Answer:
These are important goals, but they are just as important for
sequential programs as they are for parallel programs.
Therefore, important though they are, they do not belong on
a list specific to parallel programming.
Quick Quiz .4:
And if correctness, maintainability, and robustness don't
make the list, why do productivity and generality?
Answer:
Given that parallel programming is perceived to be much harder
than is sequential programming, productivity is tantamount and
therefore must not be omitted.
Furthermore, high-productivity parallel-programming environments
such as SQL have been special purpose, hence generality must
also be added to the list.
Quick Quiz .5:
Given that parallel programs are much harder to prove
correct than are sequential programs, again, shouldn't
correctness really be on the list?
Answer:
From an engineering standpoint, the difficulty in proving
correctness, either formally or informally, would be important
insofar as it impacts the primary goal of productivity.
So, in cases where correctness proofs are important, they
are subsumed under the ``productivity'' rubric.
Quick Quiz .6:
What about just having fun?
Answer:
Having fun is important as well, but, unless you are a hobbyist,
would not normally be a primary goal.
On the other hand, if you are a hobbyist, go wild!
Quick Quiz .7:
Are there no cases where parallel programming is about something
other than performance?
Answer:
There are certainly cases where the problem to be solved is
inherently parallel, for example, Monte Carlo methods and
some numerical computations.
Even in these cases, however, there will be some amount of
extra work managing the parallelism.
Quick Quiz .8:
Why all this prattling on about non-technical issues???
And not just any non-technical issue, but productivity
of all things?
Who cares?
Answer:
If you are a pure hobbyist, perhaps you don't need to care.
But even pure hobbyists will often care about how much they
can get done, and how quickly.
After all, the most popular hobbyist tools are usually those
that are the best suited for the job, and an important part of
the definition of ``best suited'' involves productivity.
And if someone is paying you to write parallel code, they will
very likely care deeply about your productivity.
And if the person paying you cares about something, you would
be most wise to pay at least some attention to it!
Besides, if you really didn't care about productivity, you would be doing it by hand rather than using a computer!
Quick Quiz .9:
Given how cheap parallel hardware has become, how can anyone
afford to pay people to program it?
Answer:
There are a number of answers to this question:
Quick Quiz .10:
This is a ridiculously unachievable ideal!
Why not focus on something that is achievable in practice?
Answer:
This is eminently achievable.
The cellphone is a computer that can be used to make phone
calls and to send and receive text messages with little or
no programming or configuration on the part of the end user.
This might seem to be a trivial example at first glance, but if you consider it carefully you will see that it is both simple and profound. When we are willing to sacrifice generality, we can achieve truly astounding increases in productivity. Those who cling to generality will therefore fail to set the productivity bar high enough to succeed in production environments.
Quick Quiz .11:
What other bottlenecks might prevent additional CPUs from
providing additional performance?
Answer:
There are any number of potential bottlenecks:
Specific hardware systems may have any number of additional bottlenecks.
Quick Quiz .12:
What besides CPU cache capacity might require limiting the
number of concurrent threads?
Answer:
There are any number of potential limits on the number of
threads:
Specific applications and platforms may have any number of additional limiting factors.
Quick Quiz .13:
Are there any other obstacles to parallel programming?
Answer:
There are a great many other potential obstacles to parallel
programming.
Here are a few of them:
One can argue that many of these obstacles are non-technical in nature, but that does not make them any less real. In short, parallelization can be a large and complex effort. As with any large and complex effort, it makes sense to do your homework beforehand.
Quick Quiz .14:
Where are the answers to the Quick Quizzes found?
Answer:
In Appendix starting on
page
.
Hey, I thought I owed you an easy one!
Quick Quiz .15:
Some of the Quick Quiz questions seem to be from the viewpoint
of the reader rather than the author.
Is that really the intent?
Answer:
Indeed it is!
Many are modeled after Paul--just ask anyone who has had the
misfortune of being assigned to teach him.
Others are quite similar to actual questions that have been asked
during conference presentations and lectures covering the
material in this book.
Still others are from the viewpoint of the author.
Quick Quiz .16:
These Quick Quizzes just are not my cup of tea.
What do you recommend?
Answer:
There are a number of alternatives available to you:
Paul E. McKenney 2011-12-16