setting up parallel contact

[blackbird]

Dear All,

I would like to use contact in parallel FEAP (version 8.4 with petsc 3.3-p7). Therefore I created a simple benchmark example (see appended) - two blocks colliding-  which runs fine in a serial feap and shows expected behaviour. I extended this example to parallel, by simply splitting the serial file in Mesh and solution and apply a parallel solution with 4 cores. The question arising is, where to put the contact pair definition. I tried both, putting it into the Mesh definition (para_ContactInMesh.zip) as well as putting it into the solution file (para_ContactInSolu.zip). Both cases abort with

Code: [Select]

[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
see in file "hpcrun.log". Yet, when defining the Contact within the mesh file, the partitioning aborts directly after creation of partition one.

Do you know this issue? Is there a fix to it? Do you know alternative approaches to realize a contact in parallel feap simulation?

Best,
Christian

[FEAP_Admin]

parFEAP should automatically generate all the necessary control information for contact when you run the graph partitioner.  However, I checked your serial input file and it crashes.  I checked with a simpler 2D problem and it worked.  So it looks like there is an issue in 3D.  I am seeing the error both with ver84 and ver85.

We will look into it and get back to you.

[FEAP_Admin]

It seems that the problem has to do with problem size.  If you make your blocks 4x4x4 instead of 10x10x10, it works file.  The steps are
take your Icont from seriell.zip; run the graph partitioner and out the domains; then mpirun parallel feap (I needed to use a direct solver for it to be happy).

We'll get back to you when we figure out the sizing problem.

[Prof. S. Govindjee]

Here is a patched version of parfeap/pmacr7.F (for version 8.4) that fixes the problem.  The changes are in the make_node_graph subroutine.  It now checks, on the fly, when it has run out of memory and reallocates as needed. 

For version 8.5 users, this will be part of the next bug fix release later this month.

[blackbird]

Dear Prof. Govindjee,
Dear Feap_Admin,

we have had an update to FEAP 8.5 using petsc 3.8.3, where I would like to retry the above mentioned example.

Again, for this version, the serial simulation is fine, while I am running into troubles with the parallel execution for blocks with 10x10x10 elements and with 4x4x4 its fine. I tried several solver setups:

Code: [Select]

-ksp_type cg -pc_type jacobi is not converging - as you already stated, an unsymmetric solve is required, while this setup is valid only for symmetric problems

Code: [Select]

-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package superlu_dist as the superlu is possible for unsymmetric problems, it was surprising to see it fail here. There is a SIGABORT traced back to the usolve.F in line 290

Code: [Select]

-ksp_type bcgs the only option that produces a meaningful and converged result for me

So my first questions are:
- Is there any news on the size problem?
- Which solver do you use/ recommend for large systems?

While I would really like to simulate two colliding objects, I also tried to simplify the setup to a single block being in contact to a rigid surface (see below input file). However, also this case is not working very well. I used the bcgs solver setup from above and run the example on 8 cores. However, there are two problems I have identified:

1. each O-file contains a bunch of warnings in the rigid body specification data:

Code: [Select]

   R i g i d   B o d y    D a t a
      ndf       Partition   Rot. Update
       6           1          -3

                Rigid Body     Type
                  Number    (0=R,1=M)     1-Coord     2-Coord     3-Coord
   *WARNING* Length allocation for:REQRB Length =           0
   *WARNING* Length allocation for:RCG   Length =           0
   *WARNING* Length allocation for:RINER Length =           0
   *WARNING* Length allocation for:RIRB  Length =           0
   *WARNING* Length allocation for:RLAMB Length =           0
   *WARNING* Length allocation for:RMASS Length =           0
   *WARNING* Length allocation for:RUROT Length =           0
   *WARNING* Length allocation for:RBOU  Length =           0
   *WARNING* Length allocation for:RPROP Length =           0

Are there any errors in the input file? Do you know these warnings or can they be ignored?

2. the 5th partition (Ocontp_0005) seems to have a problem after the domain information specification:

Code: [Select]

     DOMAIN Information

          Number of partition nodes    :     456
          Number of total problem nodes:    3636
          Number of total problem eqns.:   10800

           Number of partition equations:        1368
 *ERROR* DEFAULTP: Solution method must be input.
         OPTIONS : pena, lagm, croc, cons

However, it is not clear for me, how there is no solution method given for only a single partition out of 8 - they all stem from the same Input. Are you able to reproduce the output? Is there a solution for this problem?

[Prof. S. Govindjee]

To my re-collection, the parallel contact only includes the NODE TO SEGMENT option.  This is the main problem.

Secondarily, I notice that if one runs OUTM on your input, then we seem to loose your function definition for the rigid surface.

(So one missing feature and one bug.  It is not clear what it will take to lift these issues.  The work around is to define a (stiff)solid
to contact with.)

[Prof. S. Govindjee]

I figured out how to get it to work.

(1) Go into each parallel input file Icontp_000* and search for RIGID

(2) If there are contact FACETS present for SURFACE 2, then for SURFACE 1 replace the word FACET with

Code: [Select]

FUNCTION CART  3,-1e-3,1e-3,1
(3) If there are NO contact FACETS present for SURFACE 2, then delete the cards which have been generated for the contact, viz. delete the following 8 lines

Code: [Select]

SURFACE    1
  RIGID
    FACET

PAIR    1
  ntor       2       1
  SOLM pena  1.0000000E+10  1.0000000E+10
  TOLE,,  1.0000000E-08  1.0000000E-08  1.0000000E-05

[blackbird]

Thank you Prof. Govindjee,

the way you proposed is working fine - now the parallel is giving the same result as the serial contact simulation.

I would like to do an analysis on a big mesh with several hundreds of partitions. Therefore, I will need to fix the output routine for the partitioning. I made a first try, modifiing parFEAP's cpoutm.f and cpoutms.f (see appended). However, I am not sure where to find the correct values for the line with "FUNCTION CART" and I did put them manually for my case (line 62 in cpoutms.f). Maybe you can give me a hand on where to find the proper variables?

[Prof. R.L. Taylor]

Thank you for the files.  I will work on extracting the rigid surfaces to get all the types and values.