CS 442/EECE 432 Introduction To Parallel Processing. Spring Semester, 2003, Tapy :30 PM 1:45 PM Tuesday/Thursday

(1)

CS 442/EECE 432 Introduction To Parallel Processing

Spring Semester, 2003, Tapy 219

12:30 PM – 1:45 PM Tuesday/Thursday

Lecturer: Professor Brian T. Smith, Galles Bldg, Room 2001, 277-8249 or 277-8337

Farris Engineering Center 330A, 277-4565

Introduction

The purpose of the course is to introduce you to the issues of parallel computing and the algorithms that are used to run various computational processes in parallel. The emphasis for this semester is on distributed memory parallelism (as opposed to data parallelism or shared memory parallelism, although these other programming paradigms are introduced and discussed to some extent). The course surveys parallel programming platforms, introduces the principles of parallel program design and the concepts of parallel programming models, presents performance models for these machines and how one measures performance (mainly scalability issues), and then studies the basic communication operations around which many parallel programs can based. Next, the course concentrates on the message passing interface (MPI), which has become the portable, standard message passing library for many production codes. This will be followed by a brief introduction of the shared address space tools, namely, POSIX threads and OpenMP programming techniques. The course ends with the study of algorithms and data representations in various important computational areas such as linear algebra (dense matrices), sorting, optimization, graph theory, and fast Fourier transforms. If time permits, material on dynamic programming and various other parallel programming approaches will be studied.

The objective is for you to become familiar with parallel processing and programming ideas and know when to use them and when it is not worth the effort. As pointed out in the textbook, parallel computing has in the past been used in the scientific and engineering computational areas such as fluid dynamics, weather forecasting, molecular dynamics, structural mechanics, signal processing, studies of black holes and fundamental particles in quantum physics and many other similar scientific applications. However, now new areas are taking advantage of parallel processing and quite likely will become the dominant parallel computational applications of the future. Many of these new areas involve data intensive applications with the need to manipulate and analyze massive amounts of data; these areas include transaction processing and information retrieval, multi-media services, data mining and analysis, computational biology, and nanotechnology.

Prerequisites And The Text Book

What is needed for this course are good programming and analysis skills. The programming languages that are used are C, C++, and Fortran (f77, f90, or f95). However, because concurrency in parallel programming is so different from your serial programming experience, the skill to understand what your programming is doing when not knowing exactly when it is doing it requires highly tuned reasoning skills. Unfortunately, the parallel debugging and parallel program analysis tools are very primitive so that you have to make up for this inadequacy.

The textbook used this semester is:

Introduction to Parallel Computing, Second Edition

Vipin Kumar, Ananth Grama, Anshul Gupta, George Karypis Benjamin/Cummings Publishing Company, 2003

ISBN 0-201-64865-2

(2)

understanding of the material. Other reference texts that you may find useful are: Introduction to Parallel Computing, First Edition, Second Printing Vipin Kumar, Ananth Grama, Anshul Gupta, George Karypis Benjamin/Cummings Publishing Company, 2003

ISBN 0-805-33170-0

Fundamentals of Parallel Processing Jordan and Alaghband, Prentice-Hall, 2003 Source Book Of Parallel Computing Dongarra et al, Morgan-Kaufmann, 2003

MPI – The Complete Reference, Vol. 1, 2nd Edition Snir et al., MIT Press, 1999

Computer Architecture – A Quantitative Approach, 3rd Edition Hennessy and Patterson, Morgan-Kaufmann, 2003

Modern Processor Design – Fundamentals Of Superscalar Processors, Beta Edition Shen and Lipasti, McGraw-Hill, 2003

For the most part, I will present material for approximately 60 minutes in each class and spend the remainder of the class working on some examples and exercises from the various places. During classes, I encourage active participation of the class, and in particular, will ask for participation while working out in-class examples and exercises.

I have noticed some typos and mistakes in the textbook, particularly the first edition. I have obtained from the Benjamin/Cummings web site and the authors web site a list of typos and corrections that will help you read the first edition of the text book. (For the second edition, the publisher promises the material will be posted by the end of January, 2003.) These will be posted on the class web site.

The Class Web Site

I have created a class web site on which I will post my lecture slides, handouts, homework exercises, class assignments, and a final project. In addition, I will post corrections to the textbook and other acquired material that I believe is of interest to the class. The web site is: http://www.ahpcc.unm.edu/~bsmith/CS442/CS442.html For example, in the directory …/CS442/Handouts/Postscript at this site, there are the published corrections in postscript format to the first and second printings of the first edition of the textbook. The other major directory you will use is: …/CS442/Lectures, with subdirectories Ppt, Postscript, Zip, and Pdf containing Microsoft PowerPoint 2000, postscript, zipped Microsoft Ppt files, and derived pdf versions of the lectures slides.

Grading

(3)

Office Hours

My office hours will be 1:45PM-3:30 PM Tuesday and Thursday each week at my CS Dept. Office, FEC

330A. My email is [email protected]. My CS office phone number is 277-4565. For help at other times, please contact the HPCERC (High Performance Computing, Education, and Research Center) receptionist at 277-8249 for an appointment.

These times are your opportunity to seek help from me on the material in this course, and are reserved in my schedule to help you with the material in this course. I strongly encourage you to take the opportunity to ask questions and to seek help when you do not understand the material; I do not bit e, and enjoy helping you understand this fascinating topic.

Topics, Quizzes, Homework, and Final Project Schedule

The following table provides the topic plan and schedule for the semester. Although I plan and intend to follow this schedule as closely as possible, it is possible the schedule may slip a class or two to accommodate unanticipated needs.

Week Date Topic Assignments or References

1 Jan. 21 Introduction: Why Parallel Computing? Sections 1.1-1.2.4 1 Jan. 23 Parallel Programming Platforms: Implicit

parallelism and memory performance limitations

Sections 2.1-2.2

2 Jan. 28 Parallel Programming Platforms: Physical organization and communication costs

Sections 2.3-2.4

2 Jan. 30 Parallel Programming Platforms: Routing mechanisms, and processor-processor mapping techniques

Sections 2.5-2.7

Homework Chapter 1 Due

3 Feb. 4 Parallel Algorithm Design: Preliminaries and decomposition techniques

3 Feb. 6 Parallel Algorithm Design: Tasks, interactions and load balancing mapping techniques

4 Feb. 11 Parallel Algorithm Design: Containing interaction overheads and parallel algorithm models

Section 3.5-3.6

4 Feb. 13 Basic Communication Operations: Broadcasts and reductions

5 Feb. 18 Basic Communication Operations: All reduce, prefix sum, scatter, gather and personalized communication

(4)

5 Feb. 20 Basic Communication Operations: Circular shift and speed improvements in

communication operations

6 Feb. 25 Quiz – Chapters 1-4

6 Feb. 27 Analytical Modeling: Overhead and performance metrics

Sections 5.1-5.2

7 Mar. 4 Analytical Modeling: Granularity and scalability

7 Mar. 6 Analytical Modeling: Min. execution time, min. cost, asymptotic analysis, and other scalability metrics

Sections 5.5-5.7

8 Mar.11 Programming Using MPI: Principles, building blocks, and MPI

8 Mar. 13 Programming Using MPI: Topologies, embedding, and overlapping communication with computation

Sections 6.4-6.5

Mar. 17-21 Spring Break

9 Mar. 25 Programming Using MPI: Collective

communication, groups, and communicators

Sections 6.6-6.7

9 Mar. 27 Programming Shared Address Space Platforms: Threads

10 Apr. 1 Programming Shared Address Space Platforms: Thread synchronization and control

10 Apr. 3 Programming Shared Address Space Platforms: OpenMP

Section 7.10

11 Apr. 8 Dense Matrix Algorithms: Matrix-vector and matrix-matrix multiplication

11 Apr. 10 Dense Matrix Algorithms: Solving linear systems in parallel

Section 8.3

12 Apr. 15 Quiz – Chapters 5-8

(5)

13 Apr. 22 Parallel Sorting: Bubble sort Section 9.3

13 Apr. 24 Graph Algorithms: Minimum spanning tree and single source shortest paths

Sections 10.1-10.3

14 Apr. 29 Graph Algorithms: all-pairs single source shortest paths and transitive closure

Sections 10.4-10.5

14 May 1 Search Algorithms: sequential searches and search overhead

Sections 11.1-11-3

15 May 6 Search Algorithms: parallel depth-first and best-first searches

Sections 11.4-11.5

15 May 8 Spare class to catch up on schedule or discuss final project

15 May 9, 5:00PM Final Project and all homework assignments Homework Chapter 11 Due

Assignments By Chapter

The assignments for each chapter are listed below. Please provide complete answers to each question, using some word processor of your choice.

Chapter Exercises Chapter Exercises