How parfeap uses PETSc?

[pjarz]

Hi,

I'm trying to understand how Parfeap collaborates with PETSc. After investigating parmanual and some code I think it works as follows (for simple problems):

1. Parfeap is run multiple time as seperate process.
2. Each parfeap process assembles matrix for their domain.
3. Each parfeap process inputs his matrix to PETSc matrix structure.
4. Each parfeap process calls KSPSolve so PETSc solves set of linear equations (PETSc does all parallel work here).
5. Each parfeap process gets the solution from his part of solution vector that was provided by PETSc.

Am I correct? Or there is something more?

[Prof. S. Govindjee]

That is pretty much correct.  The only thing missing is that for one or two items parfeap takes care of some message passing for smaller parallel computations.  Also for things like eigencomputations there are a few other technical details but for 95% of what
happens you have it correct.

[pjarz]

So in other words, there is no such thing in FEAP as domain decomposition, but more something like dividing mesh in several parts, each part is processed by separate CPU process, which puts their matrix into global matrix. Then PETSc solves this global matrix in their own way. Finally each process gets his part of solution from PETSc global solution vector?

[FEAP_Admin]

Domain decomposition happens at the time of partitioning the problem.  You start with a complete input file.  You start up parfeap/feap in serial mode, perform a nodal partition (i.e. a domain decomposition) which happens to be done using METIS, then you write out individual input files.
Next you start FEAP in parallel mode using MPIRUN.  These parallel instances of FEAP read the appropriate partitioned input file to perform the computation.

[pjarz]

I know how to run parFEAP. I'm trying to understand how it works inside. When I said domain decomposition I was thinking about methods such as Balancing domain decomposition method, BDDC, FETI etc. METIS can only divide mesh (actually for metis it's a graph) into several parts. So my question is: wheter FEAP uses some Domain decomposition methods? Right now it looks to me that FEAP divides mesh (via METIS) in several parts and each process assembles his matrix and puts this matirx into global matrix Kmat. This global matrix Kmat is solved by PETSc and PETSc handles all paralelism during solution. After solution, each process takes his part of the solution from global solution vector. Am I right?

[FEAP_Admin]

Basically yes.  Once the domain has been partitioned we do not perform any time of load balancing, say on the fly.
The parallel solve is handled entirely by PETSc.  The only exception to that are some aspects of the eigensolver routines but even there the large parallel solves are done by PETSc.

Adding dynamic load balancing etc. would be relatively complex, though not undoable.

[pjarz]

Thx a lot for your answers.

One more question. What you say means that when I use PETSc View command in parFEAP for let's say 2 and 4 processes for one task, I'll get two files with stiffness matrices let's say stiff_2.m and stiff_4.m. Both of these matrices have exactly same values, but different locations?

[FEAP_Admin]

Not sure.  Try it or have a look at the PETSc documentation; it may say.