CS 15 Special segment
Winter 1995
Introduction to Parallel programming
David Kotz

Day 1: Introduction

• Motivation for parallel computing
	• need for high-performance computing (NASA video)
	• difficult to make current computer hardware go faster and faster
	• use computers in parallel to accomplish very large computations
	• often, this just means hooking many workstations to a network
	• some are designed and sold particularly for parallel computing

• What kinds of problems are parallel?   Parallelism is natural.
	• simulating particles flowing over airplane wing
	• take the class to Boston using 1 car... 1 carload at a time, or take many cars...
	• laundry: many loads at once, if you have many washers. also dry and wash 
	• computing grades as a weighted average of each student's homework grades
	• handing in homeworks: pass to the left, then up to the front

• What kinds of problems are not  parallel?
	• army way of speeding up pregnancy: 9 women for 1 month
	• driving to Boston: you have to drive to manchester, nashua, ... boston

• Demo: people as processors, (slips of paper with words) as data
	• parallel search for a particular word, vs. sequential search

• Data parallelism and task parallelism
	• data parallel: do the same  thing at the same time to many pieces of data
	• task parallel: do different  things at the same time to many pieces of data

• Data parallelism
	• think of it as having a virtual  processor (VP) for each data element
	• a controller, like a drill sargent, tells the processors what to do in each step
	• (another analogy is contradance caller)
	• all VPs are either active  or inactive;  only the actives do what controller says
	• (demo)  search for even-length words; active iff you match.  
Now, actives fold your paper in half.
Now, actives fold your paper in half.
Now, actives fold your paper in half.
(what are the inactives doing during this period?  inefficient).

• Parallel complexity and Work
	• how do we measure how fast we are going, and is it faster than sequential?
	• analyze the parallel algorithm: number of time steps, number of processors
	• sequential computing has only one processor, so we compute time complexity
only, and report it as time Ts(N) = O(N), etc. 
	• parallel computing has P processors each working for Tp(N) steps. 
	• certainly, faster if Tp(N) < Ts(N)
	• the speedup  is S(N) = Ts(N) / Tp(N)
	• how much work  is being done?  Wp(N) = P(N) * Tp(N)
	• how much work is done by the sequential algorithm?  Ws(N) = 1 * Ts(N)
	• if the work is the same in both cases, ie, Ts(N) = P(N) * Tp(N), then the speedup Ts(N)/Tp(N) =  P(N).  This is perfect speedup.

• Amdahl's law
	• ideally the parallel program will take 
		Tp = Ts/P (drop the (N) for now).   
	But this only happens when all of the sequential program's time Ts can be sped up with parallelism.   More likely, 
		Ts = Ta + Tb 
where Ta  cannot be parallelized, and Tb can be parallelized. Then we expect
		Tp = Ta + Tb/P
Note that we can add all the processors we like, ie, P -> infinity, and we're left with
		Tp = Ta
which means the best speedup is 
		S = Ts/Tp = Ts/Ta
Suppose that Ta = 10% of Ts, that is, 90% of the sequential time is parallelizable, and 10% is not.  That means that
		S = Ts / (0.10 * Ts) = 10
No matter how many processors we have, we can't get better than 10x speedup!!
So, we have to work very hard to parallelize (nearly) all of the program. 

• Handouts:
	- reading for next time: DAPPLE