PETSc Tutorial Exercises Intended for use with version 2.0.29 of PETSc *************** Getting Started *************** - Locate the PETSc on-line documentation, available via http://www.mcs.anl.gov/petsc/docs In particular, you may want to reference the on-line users manual and manual pages. - Set the following environmental variables: PETSC_DIR - location of PETSc installation PETSC_ARCH - machine on which you will use PETSc e.g., setenv PETSC_DIR /users/gropp/petsc-2.0.29 setenv PETSC_ARCH alpha At some sites, there may be commands such as module or soft to set these. - Explore some of the following example exercises. First, copy an example program and its corresponding makefile from the PETSc distribution into a working directory. For example, to use the "hello world" example listed below, do the following: % mkdir petsc_examples % cd petsc_examples % mkdir hello_world % cd hello_world % cp $PETSC_DIR/src/sys/examples/tutorials/ex2.c . % cp $PETSC_DIR/src/sys/examples/tutorials/makefile . Note that each of the examples categories should be placed in a separate working directory so that the makefile commands will function properly. - Compile an example program: debugging version: make BOPT=g , e.g., make BOPT=g ex2 optimized version: This is the letter O, not the digit zero make BOPT=O , e.g., make BOPT=O ex2 - Run an example program: mpirun -np , e.g., mpirun -np 2 ex2 ****************** Beginner Exercises ****************** - - - - - - Hello World - - - - - - * Objective: To run a simple program and learn the makefile and run commands Program: $PETSC_DIR/src/sys/examples/tutorials/ex2.c Compilation commands: debugging version: make BOPT=g ex2 optimized version: make BOPT=O ex2 Runtime commands: mpirun -np 1 ex2 mpirun -np 2 ex2 mpirun -np 4 ex2 - - - - - - - Vectors (Vec) - - - - - - - * Objectives: To demonstrate basic vector capabilities. To demonstrate creation/destruction of PETSc objects. Programs: $PETSC_DIR/src/vec/examples/tutorials/ex1.c or $PETSC_DIR/src/vec/examples/tutorials/ex1f.F Runtime commands: mpirun -np 2 ex1 or mpirun -np 2 ex1f * Objective: To demonstrate parallel vector layout Programs: $PETSC_DIR/src/vec/examples/tutorials/ex3.c or $PETSC_DIR/src/vec/examples/tutorials/ex3f.F Runtime commands: mpirun -np 2 ex3 -draw_pause -1 or mpirun -np 2 ex3f -draw_pause -1 where -draw_pause -1 : click the RIGHT mouse button to advance the program - - - - - - - - Matrices (Mat) - - - - - - - - * Objective: To demonstrate matrix assembly Program: $PETSC_DIR/src/sles/examples/tutorials/ex3.c Runtime commands: mpirun -np 2 ex3 -mat_view and mpirun -np 2 ex3 -mat_view_draw -draw_pause -1 where -mat_view : print matrix elements to screen -mat_view_draw : draw the matrix sparsity structure Comments: Note that this matrix assembly is done by 4x4 blocks, with each process contributing its own local parts; see calls to MatSetValues(). In this example, you may need to move one of the windows that pops up aside so that you can right-click the one underneath to continue the program's execution. - - - - - - - - - - - Linear Solvers (SLES) - - - - - - - - - - - SLES provides an easy-to-use interface to the combination of a Krylov subspace iterative solver (KSP) and a preconditioner (PC). See the SLES, KSP, and PC manual pages as well as the "Linear Equations Solvers" chapter of the PETSc users manual for details. * Objective: To demonstrate basic linear solver use Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c Runtime command: mpirun -np 2 ex23 -ksp_monitor where -ksp_monitor : print residual norm at each iteration Comments: Note that the program calls SLESView(), which displays information about the particular solvers used at runtime. * Objective: To demonstrate setting different preconditioners and Krylov methods at runtime Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c Runtime command: mpirun -np 2 ex23 -ksp_monitor -pc_type asm -ksp_type tfqmr -optionsleft where -ksp_type tfqmr : set Krylov method to transpose-free QMR -pc_type asm : set preconditioner to additive Schwarz -optionsleft : print information about the options specified at runtime Comments: Use the -help option for a complete list of solver options. * Objective: To produce a summary of program performance Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c Runtime command: mpirun -np 4 ex23 -n 1259 -log_summary where -n : set the problem dimension Comments: The performance summary output is wide, so use an xterm with 120 columns; either resize one or use "xterm -geometry 120x24". See the chapter "Hints for Performance Tuning" in the PETSc users manual for help in interpreting this output. Try using additional solver options as well, and note the effects on performance. For example, -ksp_rtol 1.e-10 : set convergence tolerance Use the -help option for a complete list of solver options. - - - - - - - - - - - - Nonlinear Solvers (SNES) - - - - - - - - - - - - * Objective: To demonstrate basic nonlinear solver use Programs: $PETSC_DIR/src/snes/examples/tutorials/ex5.c or $PETSC_DIR/src/snes/examples/tutorials/ex5f.F and ex5f.h or $PETSC_DIR/src/snes/examples/tutorials/ex5f90.F Runtime commands: mpirun -np 3 ex5 -mx 10 -my 12 -snes_monitor -snes_view or mpirun -np 3 ex5f -mx 10 -my 12 -snes_monitor -snes_view or mpirun -np 3 ex5f90 -mx 10 -my 12 -snes_monitor -snes_view where -snes_monitor : print residual norm at each iteration -snes_view : print information about the particular nonlinear solvers used at runtime -mx -my : set mesh dimensions Comments: By default a Netwon line search method is used. * Objective: To demonstrate setting different nonlinear solvers at runtime Programs: $PETSC_DIR/src/snes/examples/tutorials/ex5.c or $PETSC_DIR/src/snes/examples/tutorials/ex5f.F and ex5f.h or $PETSC_DIR/src/snes/examples/tutorials/ex5f90.F Runtime command: mpirun -np 3 ex5 -mx 10 -my 12 -snes_monitor -snes_view \ -snes_type tr -optionsleft (or similarly for ex5f and ex5f90) where -snes_type tr : set nonlinear solver to a Newton trust region method -optionsleft : print information about the options specified at runtime Comments: Use the -help option for a complete list of solver options. - - - - - - - - - - - - - Timestepping Solvers (TS) - - - - - - - - - - - - - * Objective: To demonstrate basic timestepping solver use Programs: $PETSC_DIR/src/ts/examples/tutorials/ex2.c and util2.c or $PETSC_DIR/src/ts/examples/tutorials/ex2f.F, util2.c, and ex2f.h Runtime command: mpirun -np 2 ex2 -ts_view or mpirun -np 2 ex2f -ts_view where -ts_view : print information about the particular timestepping solvers used at runtime Comments: The backward Euler method is set in this code by a call to TSSetType(). This example runs for 1000 time steps. * Objective: To demonstrate setting different timestepping solvers at runtime Programs: $PETSC_DIR/src/ts/examples/tutorials/ex2.c and util2.c or $PETSC_DIR/src/ts/examples/tutorials/ex2f.F, util2.c, and ex2f.h Runtime command: mpirun -np 2 ex2 -ts_view -ts_type euler or mpirun -np 2 ex2f -ts_view -ts_type euler where -ts_type euler : set timestepping solver to the Euler method Comments: Use the -help option for a complete list of solver options. ********************** Intermediate Exercises ********************** - - - - - - - - - - - - - - - - - - Mesh Management (DA and VecScatter) - - - - - - - - - - - - - - - - - - * Objective: To demonstrate use of distributed arrays (DA) to manage a parallel structured mesh computation. Program: $PETSC_DIR/src/snes/examples/tutorials/ex5.c Runtime command: mpirun -np 4 ex5 -mx 10 -my 11 -da_view_draw -snes_monitor \ -draw_pause -1 where -da_view_draw : draw DA configuration * Objective: To demonstrate use of vector scatters (VecScatter) to manage a parallel unstructured mesh computation. Program: $PETSC_DIR/src/snes/examples/tutorials/ex10d/ex10.c and adj.in Runtime command: mpirun -np 2 ex10 -snes_monitor - - - - - - - - - - - - - - - - - - Evaluating and Tuning Performance - - - - - - - - - - - - - - - - - - See the chapter "Hints for Performance Tuning" in the PETSc users manual. * Objective: To demonstrate verbose logging information Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c (and any other programs as well) Runtime command: mpirun -np 2 ex23 -log_info where -log_info : print verbose logging information Comments: Note that this command prints information about matrix assembly, such as the number of dynamic memory allocations used, maximum number of nonzeros per row, etc. This information can help with tuning the matrix assembly process. ****************************************************** Advanced Exercises ****************************************************** * Objective: Demonstrate various multigrid, additive Schwarz, and Neumann-Neumann preconditioner options for several test problems. - - - - - - - - - - - Linear Solvers (SLES) - - - - - - - - - - - * Objective: 3D Laplacian on a structured mesh Program: $PETSC_DIR/src/sles/examples/tutorials/ex22.c Multigrid: mpirun -np 1 ex22 -sles_view -ksp_monitor (Note: the default smoother is ILU(0)) mpirun -np 1 ex22 -sles_view -ksp_monitor \ -mg_levels_pc_type sor (use sor as a smoother) mpirun -np 4 ex22 -sles_view -ksp_monitor \ -mg_levels_pc_type sor -mg_levels_pc_sor_local_symmetric Try some additional multigrid options: - Use a different number of levels: -damg_nlevels - Use a W-cycle instead of the default V-cycle: -pc_mg_cycles 2 - Use a different scheme: -pc_mg_type - Use 2 pre and 2 post smoothing steps: -pc_mg_smoothup 2 -pc_mg_smoothdown 2 Also try different combinations of these. One-level additive Schwarz: (one domain per processor) mpirun -np 1 ex22 -sles_view -ksp_monitor -pc_type asm (Note: This has one subdomain, so is the same as ILU(0), the default subdomain solver) mpirun -np 4 ex22 -sles_view -ksp_monitor -pc_type asm (Note: default overlap of 1) mpirun -np 4 ex22 -sles_view -ksp_monitor -pc_type asm -sub_pc_type lu (Solve the subproblems exactly with LU) mpirun -np 4 ex22 -sles_view -ksp_monitor -pc_type asm -pc_asm_overlap 2 Two-level additive Schwarz: (grid ratio of 32 between levels) mpirun -np 4 ex22 -sles_view -ksp_monitor -damg_nlevels 2 -damg_ratio 32 * Objective: 2D Laplacian on an unstructured grid Program: $PETSC_DIR/src/contrib/oberman/laplacian_q1/* (copy everything in this directory) Compilation: make BOPT=g main (debugging version) or make BOPT=g main (optimized version) Default solver: block Jacobi with ILU(0) on each block mpirun -np 4 main -f gr10 (also try using larger grids: gr22, gr44, gr99) One-level additive Schwarz: mpirun -np 4 main -f gr10 -pc_type asm (Note: default is Restricted Additive Schwarz (RASM)) mpirun -np 4 main -f gr10 -pc_type asm -pc_asm_type basic (This is "standard" additive Schwarz, where the subdomain restriction and interpolation are the same.) Try varying the overlap, subdomain solver, etc. Mandel's balancing Neumann-Neumann preconditioner: mpirun -np 4 main -ksp_monitor -f gr10 -pc_type nn -mat_type is mpirun -np 8 main -ksp_monitor -f gr33n -pc_type nn -mat_type is (Note: This mesh has Dirichlet boundary conditions only on the left edge) mpirun -np 8 main -ksp_monitor -f gr33n -pc_type nn -mat_type is \ -turn_off_first_balancing -turn_off_second_balancing (Run with no coarse problem; note that this requires many more iterations.) - - - - - - - - - - - - Nonlinear Solvers (SNES) - - - - - - - - - - - - * Objective: Scalar, radiative transport PDE on a structured mesh Program: $PETSC_DIR/src/snes/examples/tutorials/ex18.c mpirun -np 2 ex18 -snes_view -snes_monitor The problem is run twice to preload the memory. Run the program with the different options suggested above for multigrid and additive Schwarz. * Objective: Multicomponent, driven cavity problem (with 4 degrees of freedom per node) on a structured mesh Program: $PETSC_DIR/src/snes/examples/tutorials/ex19.c mpirun -np 2 ex19 -snes_view -snes_monitor Run the program with the different options suggested above for multigrid and additive Schwarz. ****************************************************** Interfacing between PETSc and other software packages ****************************************************** See slide #7 for web page pointers to these software packages. To use one of these external tools with PETSc, the tool must have been enabled when the local PETSc installation was built (see the file $PETSC_DIR/bmake/<$PETSC_ARCH>/base.site). The following tools [may/may not] be available on your machine. - - - - - - - - - - - - - - - - - - - - - - - - - SPAI - sparse approximate inverse preconditioner - - - - - - - - - - - - - - - - - - - - - - - - - Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c Runtime commands: mpirun -np 1 ex23 -ksp_monitor -pc_type spai Comments: Use the -help option for a complete list of SPAI options. - - - - - - - - - - - - - - - - - - ParMETIS - parallel partitioner - - - - - - - - - - - - - - - - - - Program: $PETSC_DIR/src/dm/ao/examples/tutorials/ex2.c Runtime command: mpirun -np 2 ex2 -mat_partitioning_type parmetis - - - - - - - - - - - - PVODE - ODE integrator - - - - - - - - - - - - Programs: $PETSC_DIR/src/ts/examples/tutorials/ex2.c or $PETSC_DIR/src/ts/examples/tutorials/ex2f.F Runtime commands: mpirun -np 2 ex2 -ts_type pvode or mpirun -np 2 ex2f -ts_type pvode - - - - - - - - - - - - - - - - - - - - Overture - composite mesh PDE package - - - - - - - - - - - - - - - - - - - - Please speak with us if you are interested in the Overture/PETSc interface. - - - - - - - - - - - SAMRAI - AMR package - - - - - - - - - - - Please speak with us if you are interested in the SAMRAI/PETSc interface. - - - - - - - - - - - - - - - - - - - - - - TAO - large-scale optimization software - - - - - - - - - - - - - - - - - - - - - - See the TAO installation (TAO_DIR = /u3/sbenson/tao-0.0.2) and tutorial, including programs: unconstrained minimization: $TAO_DIR/src/unconstrained/examples/tutorials/eptorsion2.c $TAO_DIR/src/unconstrained/examples/tutorials/eptorsion2f.F bound constrained minimization: $TAO_DIR/src/bound/examples/tutorials/plate2.c $TAO_DIR/src/bound/examples/tutorials/plate2f.F