Architecture and interconnection of parallel computers; parallel programming models and applications; issues in high-performance computing; programming of parallel computers. Prerequisites: knowledge of programming in a high-level language; MATH 526 or 544 3.000 Credit hours 3.000 Lecture hours
Fundamentals and programming practices for parallel computing on parallel computing systems including general purpose multicore CPU, Graphic Processing Unit (GPU) manycores and high performance computing (HPC) clusters. Topics include:
The course content includes a significant amount of programming exercises to create performant code using C/C++ programming language on Linux environment.
In high performance computing (HPC) systems and enterprise systems, parallel and distributed computing have been the approaches for several decades to deliver the performance needed for large-scale scientific and engineering simulation, and for big data analysis and machine learning. Today, parallel computing capability are available in the computing devices we use daily. Multicores CPUs are widely used in laptop, desktop, smartpad and phone, and some of them have manycore GPUs. In this course, we will study the processor, memory and interconnection architectures of modern CPU, GPU and HPC clusters, learn to design high performance parallel algorithms, and develop parallel programs using OpenMP, CUDA and MPI programming models. The course content includes ~40% theory and fundamentals, and ~60% programming and exercises.
Following completion of the course, students should be able to:
There will be two 75-minute lectures per week, which constitutes 100% of the course lecture delivery. There will be no separate labs, and there are no plans for distance learning.
There are two exams and four assignments. The two exams are close books/notes. The contribution of each to your final grades are shown below. Questions with bonus points may be given.
| Letter | Percentage |
|---|---|
| A | 90-100 |
| B+ | 86-90 |
| B | 80-86 |
| C+ | 76-80 |
| C | 70-76 |
| D+ | 66-70 |
| D | 60-66 |
| F | 0-60 |
| Week | Date | Week date | Class | Topics and Lecture Notes | Resources | Assignment |
|---|---|---|---|---|---|---|
| 1 | 01/15 | Monday | MKL Jr. Day | |||
| 01/17 | Wednesday | 1 | Introduction | Sample Questions | ||
| 2 | 01/22 | Monday | 2 | Introduction | ||
| 01/24 | Wednesday | 3 | Review of C programming, Compiler, Makefile, Linux and SSH access | Linux and C, sum.c, axpy.c, mm.c, matvec.c | Assignment 1, Due 02/09 Friday | |
| 3 | 01/29 | Monday | 4 | OpenMP Parallel | omp_hello.c, mm_openmp.c, sum_openmp.c, Chapter 7.10 and the OpenMP book | |
| 01/31 | Wednesday | 5 | OpenMP Worksharing and Reduction | |||
| 4 | 02/05 | Monday | 6 | OpenMP Data Environment, Tasking and Synchronization | ||
| 02/07 | Wednesday | 7 | OpenMP Performance Optimization | |||
| 5 | 02/12 | Monday | 8 | Parallel Algorithm Design 01 | Chapter 3.1 to 3.2 | Assignment 2, Due 03/02 |
| 02/14 | Wednesday | 9 | Dense Matrix and Decomposition | matmul-decompose.c, Chapter 8.1 and 8.2.1 | ||
| 6 | 02/19 | Monday | 10 | Parallel Algorithm Design 02 | Chapter 3.2 to 3.5, and Chapter 8.3.1 | |
| 02/21 | Wednesday | 11 | Distributed Memory Systems | Chapter 6 | ||
| 7 | 02/26 | Monday | 12 | Message Passing and MPI | Chapter 6 | |
| 02/28 | Wednesday | 13 | Message Passing and MPI | MPI examples | ||
| 8 | 03/05 | Monday | 14 | MPI Exercises | Assignment 3, Due 03/30 | |
| 03/07 | Wednesday | 15 | Midterm Exam | Sample Questions | ||
| 9 | 03/12 | Monday | Spring Break | |||
| 03/14 | Wednesday | Spring Break | ||||
| 10 | 03/19 | Monday | 16 | Analytical Modeling of Parallel Program: Metrics and Analysis | Chapter 5.1 to 5.3 | |
| 03/21 | Wednesday | 17 | Analytical Modeling of Parallel Program: Metrics and Analysis | Chapter 5.4 to 5.7 | ||
| 11 | 03/26 | Monday | 18 | MPI Exercises and Assignment #3 | ||
| 03/28 | Wednesday | 19 | MPI Exercises and Assignment #3 | |||
| 12 | 04/02 | Monday | 20 | Analytical Modeling of Parallel Program: Scalability | Assignment 4, Due 04/30 | |
| 04/04 | Wednesday | 21 | Parallel Architecture: Thread Level Parallelism and Data Level Parallelism | |||
| 13 | 04/09 | Monday | 22 | Memory Hierarchy and Cache Coherence | Chapter 2.2 | |
| 04/11 | Wednesday | 23 | Manycore GPUs and CUDA | CUDA examples | ||
| 14 | 04/16 | Monday | 24 | CUDA Threading and Memory | ||
| 04/18 | Wednesday | 25 | CUDA Threading and Memory, and Assignment #4 | |||
| 15 | 04/23 | Monday | 26 | GPU/CUDA Review and Memory continued, and jacobi CUDA implementation in Assignment #4 | ||
| 04/25 | Wednesday | 27 | GPU/CUDA Streaming, Library and Tuning | |||
| 16 | 04/30 | Monday | 28 | Jacobi Kernel in Assignment #4, OpenMP/HOMP for accelerators, Wrapup in Chapel | OpenACC for accelerators | |
| 05/02 | Wednesday | Final Exam (9:00AM - 11:30AM) | Sample Questions |
We greatly appreciate National Science Foundation's XSEDE program and Pittsburgh Supercomputing Center for supporting students' access to Bridges Supercomputer.
The homework and project exercises are chiefly for your own benefit. You may collaborate and consult outside sources freely when doing the homework, but you must tell me who you are collaborating with. Please remember, however, that the best way to master the material is to try the exercises on your own, then submit your own work to get feedback on it. (You will get credit even if you do not solve the problem, as long as you make an honest effort.)
Each assignment has a due date and a cutoff date. Assignment due 11:55PM on the date, cut-off date is 72 hours after the due date/time. You have a total of 5 "slip days" throughout the semester that you can use at your discretion to turn in programming assignments past the posted due date. Slip days are used in whole day increments. Once your slip days are consumed, late programming assignments will be penalized at 50% per day. Assignment submission will not be accepted after the cutoff date.
Reading and lectures: The students are expected to read all assigned material before the lecture begins and review the material after the lecture.
There is no required attendance policy, but if for some reason you cannot attend class, you are responsible for any material covered during your absence. Late arrivals must enter the classroom quietly and discreetly.
Exams are given in class and are close-book/close-notes. No make-up exams will be given except under extreme circumstances (such as severe illness or death in the immediate family) in which case you must give me notice well before the exam if at all possible.
Examination work and assignment are expected to be the sole effort of the student submitting the work. If a student collaborates with anyone on the homework (whether or not a student), all collaborators must be listed at the top of the first page. Students are expected to follow the Code of Student Academic Responsibility. Every instance of a suspected violation will be reported. Students found guilty of violations of the Code will receive the grade of F for the course in addition to whatever disciplinary sanctions are applied by the University.
Students are expected to be aware of the university policy on use of computing resources, including the Student Guidelines for Responsible Computing, as well as the college and departmental policies on proper use of computing resources. Every instance of a suspected violation will be reported.
Any student with a documented disability should contact the Office of Student Disability Services at 803-777-6142 to make arrangements for appropriate accommodations.