Pressure Elements and Euler angles in Parallel FEAP

[avakirb]

Dear FEAP forum,

I am using feap v8.6.1j and I am encountering severe problems using parfeap. I want to simulate the inflation of a hollow object by using pressure elements (follower loads) at the internal surface. The object is fixed by using nodal stiffness springs and fixed displacement boundary conditions, also using Euler angles. I work with a user element.

Here is my problem: When I run the simulation using serial feap everything is working fine. But when I use parfeap with NPROC = 4 partitions the simulation prematurely terminates, due to poor convergence behaviour. I already made sure that the mesh settings are correct in the parallel input files. I compared the displacements of the serial and parallel simulations up to the point where the simulation terminates, and the displacements are similar.
I also ran a simulation using parfeap with NPROC = 1 partition. In this case, the new node numbering (in the single parallel input file) was the same as the old node numbering (in the serial input file) and the setting was identical to the serial case. However, the simulation terminated again. My conclusion is that the solver is probably the reason, that my simulation does not converge after some time, using parfeap. I use the superlu solver because the other options do not work at all.

Do you have any suggestions about how I can make my simulation run with parfeap? I really want to use parfeap because of the high amount of elements in my mesh.

As an additional note, I observed that the euler angles are not correctly assigned to the new nodes when using parfeap with NPROC = 4 partitions. So here I always manually adjust the list of euler angles in each parallel input file. I also created a flat file with OUTMesh command before partition but the problem remains.

Thank you in advance

[Prof. R.L. Taylor]

Can you test using a regular feap element instead of the user element? The follower force will create an unsymmetric matrix so using a direct solver with UTAN is probably necessary.

The other issue is the Euler angles so we would like to have a test problem to use for that too.

[Prof. S. Govindjee]

Are you partitioning a flat file?  or does it have a TIE command in it?  partitioning always needs to be done on a flat file.

[avakirb]

Thanks for the reply. While creating a test file to upload I noticed that the euler angles are correctly assigned in feap v8.6.1j. The problem occured with the older version 8.6.1e I used, so this is fine now. However, I still provide a test file for you to see the problem. I have my original input file with includes and a tie command (Ipt_orig). I always create a flat file with the outm command before partitioning (Ipt_orig.rev). However, regardless which feap version I use, the euler angles are missing in the flat file, so I insert them manually (Ipt). Then I partition the problem, which gives me four input files with incorrect nodal assignments for the euler angles using feap version 8.6.1e.

I tested my simulation using a solid feap element and the modified neo hookean material and both serial feap and parallel feap (with 1 partition) gave similar results. But what can be the reason that my user element works in serial feap but not in parfeap? I used superlu solver with utan.

[Prof. R.L. Taylor]

I will fix the euler angle problem in the flat file.  Your user element has 4 dofs. This may cause some problem.  There should not be any problem otherwise with a user element unless there is a memory problem in your element.  Does it have history variables?

[Prof. R.L. Taylor]

In the routine poutm.f in ./program

1. add small array uu(3) ( real (kind=8) )

2. add after the set of ANGLe following:
!         Euler angle conditions lists
          if(np(242).ne.0) then
            lread = .true.
            do nn = 1,numnp1
              pflg = .false.
              do i = 1,3
                uu(i) = hr(np(242)+3*(ip(nn)-1)+i-1)
                if(uu(i).ne.0.0d0) then
                  pflg = .true.
                end if
              end do ! i
              if(pflg) then
                if(lread) then
                  write(ios,2001) 'EULEr angle conditions'
                  lread = .false.
                endif
                write(ios,2002) nn,iz,uu(1:3)
              end if
            end do ! nn
          end if

Hopefully this will output the EULEr angles to the flat file. 

I do not know what the other problem related to the parallel solution could be without more information on what is happening.

[avakirb]

Thanks for fixing the euler angles in the flat file. I tested it and it is working.

Regarding the problem with the parallel solution: Yes I have 4 dofs (1-3 displacement and 4 electric potential), but when testing with parfeap I use the partition command and solve only for the displacement dofs. I also tested now having only 3 dofs by changing my user element and input file respectively, but the parallel simulation still terminates prematurely. It seems that the 4th dof is not the problem.

I also have history variables (non-zero values for nh1, nh2, and nh3). The time-dependent history variables are only neccessary for the calculation of the 4th dof, so I tested with nh1 = nh2 = 0, but again, the parallel simulation terminates.

I have fibre orientation saved in an external file which is read in the initialization phase of feap and saved in the history variables of nh3, which I need for my anisotropic material model. At first I assumed that these might cause a problem when partitioning for parallel execution, but it also does not work for NPROC = 1 partition so this can not be the problem.

I use a pressure of -0.13 kPa at the internal surface to inflate the object, starting from the beginning of the simulation. The pressure is reached after 440 time steps and is then held constant. The first 100 time steps are discretised with increments of 0.002 s and the next 350 time steps with increments of 0.005 s. The serial execution works without problems. For the parallel execution, after 164 time steps I can see the NO CONVERGENCE Warning message in the output file for every following time step, because the iterations reach the maximum number of 30 iterations, and after nearly 200 time steps the simulation terminates. But the displacements are similar for serial and parallel solution up to this point.

[Prof. S. Govindjee]

What happens if you use a direct solver in the parallel computation, say, superlu?  It could be that the matrix properties are preventing the Krylov solver from giving you a sufficiently accurate solution to Ax = b.

[Prof. R.L. Taylor]

I read through your last post and note you are trying to use the feap PARTition command to solve for only the 4th dof.  My recollection is that you cannot use FEAP PARTitions in the parallel version.  We only write out the equation numbers for solving the problem with all dof's.

Maybe some other user has a suggestion.

[avakirb]

I just want to give an update. Before partitioning I always wrote

batch
graph node 4
outd aij 1
end

in the serial input file, which created the parallel input files. Now I changed these lines to only

batch
graph node 4
outd
end

and left out the 'aij 1'. This change lead to the successful run of the parallel solution of my user element with superlu. I can not say why it does not work with 'aij 1', including the dirichlet BCs in the matrix assembly.

[Prof. R.L. Taylor]

Are you still using PARTitions?

If so, FEAP must assemble only the individual partion equations and leave zero diagonals for the missing equations. SuperLU must not care about the zero equations??

[avakirb]

No, I am not using PARTition. After you mentioned that the command cannot be used in parfeap I removed the option. Now I have one user element for three dofs and one for four dofs. Both did not work in parfeap when using 'outd aij 1', but work when I only use 'outd'.

[Prof. S. Govindjee]

That is odd.  If you can make a test case using FEAPs built in elements that shows this same behavior, we can try to debug it.