module files in parfeap

[blackbird]

Dear all,

I am updating from 8.4 to 8.5 and I am struggling with porting the module files of fortran. I organized most of my new functions in a module in the user directory. After compilation of the serial feap they are available as *.o and *.mod in the user folder. In the previous FEAP 8.4, it has been possible to use these function by the manipulation of the makefile in the parfeap folder by adding

Code: [Select]

-J$(FEAPHOME)/user
to the variable PETSC_FC_INCLUDES. However, for 8.5 there is the error

Fatal Error: File 'tensor.mod' opened at (1) is not a GNU Fortran module file

This is due to a version mismatch between the module file (created in serial FEAP with /usr/bin/gfortran-6) and the parallel compiler ($(PETSC_DIR)/$(PETSC_ARCH)/bin/mpif90 that referred to gfortran-4.8 in my case). In order to update the mpif90 wrapper, it is necessary to rebuild the whole petsc library. I tried

Code: [Select]

./configure --with-cc=/usr/bin/gcc-6 --with-fc=/usr/bin/gfortran-6 ...
With such a PETSC-version, it is possible to use the modules created in the serial FEAP without any error during compilation. However, my benchmark simulations do not converge any more. I tried both my elements as well as the feap linear elastic material - the solution diverges into residual NAN for the very first step.

Did anyone use parFEAP 8.5 with a PETSC compiled with gfortran-6. Is there a solution to this issue?

UPDATE: tried with petsc-3.8.3 as well as the recent 3.10.5

[Prof. S. Govindjee]

You need to make sure to use the same compilers for petsc as you use for feap and it should work.  I am running petsc 3.10.1 with v8.5 without problem (modulo the posted corrections to the parfeap makefile to deal with the 'use' issue introduced in recent petsc releases).  Search the forum for further information.

[blackbird]

My point is that the problem occurs only if I use gcc-6/gfortran-6 for both serial feap, petsc and parfeap ...

could you please check whether it is possible for you to use a module in a parfeap subroutine, when that module has been created by the serial feap? If so, I would be glad for detailed information on the compilers you use and the configure command for petsc, so I can reset my system with such an installation

[Prof. S. Govindjee]

Modules work fine with gcc/gfortran and petsc but you have to observe a number of points (these will be automatic when ver8.6 is released).  The basic point is that modules in serial FEAP are built before anything else, likewise for parFEAP.

1.  create a module folder if you do not have it yet $(FEAPHOME8_5)/modules
2.  in $(FEAPHOME8_5)/makefile.in
  a.  add -I$(FEAPHOME8_5)/modules to FINCLUDE
  b.  create a new variable  PMODULEDIR = -J$(FEAPHOME8_5)/modules

3.  in all the makefiles in the code add $(PMODULEDIR) to the fortran targets

4. in $(FEAPHOME8_5)/parfeap/makefile
  a. add $(PMODULEDIR) to PETSC_FC_INCLUDES
  b. create a variable MODULES = file1.o file2.o etc., where file1, file2, etc. are module files in the parfeap directory
  c. add $(MODULES) as the first element to OBJECTS
  d. add $(MODULES) as the first dependency to the target install

5. place all serial code fortran files with modules in them in the directory $(FEAPHOME8_5)/modules, if they will be needed via use statements before the source file may be encountered during the build sequence.

6. in $(FEAPHOME8_5)/makefile
  a. add modules as the first element of CLEANDIRS
  b. add (cd modules; make archive) as the first build item under the target archive

[blackbird]

Dear Prof. Govindjee,

thank you very much for the detailed description on how the modules should be included in feap.

Could you please tell me which compilers and petsc-configuration you used for the module incorporation. To help me build exactly your working setup could you please post the results of the commands on your terminal:

Code: [Select]

$PETSC_DIR/$PETSC_ARCH/bin/mpicc -v

Code: [Select]

$PETSC_DIR/$PETSC_ARCH/bin/mpif90 -v
same for the compilers you use for serial feap (FF/CC shorts in the makefile.in)

also I would like to know the information behind "Configure Options:" of petsc (probably around line 13 of the file configure.log of your petsc location)

Thank you very much

[Prof. S. Govindjee]

Hear you go:

Code: [Select]

$PETSC_DIR/$PETSC_ARCH/bin/mpicc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin


Code: [Select]

$PETSC_DIR/$PETSC_ARCH/bin/mpif90 -v
Using built-in specs.
COLLECT_GCC=/usr/local/bin/gfortran
COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/8.3.0/libexec/gcc/x86_64-apple-darwin18.2.0/8.3.0/lto-wrapper
Target: x86_64-apple-darwin18.2.0
Configured with: ../configure --build=x86_64-apple-darwin18.2.0 --prefix=/usr/local/Cellar/gcc/8.3.0 --libdir=/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 --disable-nls --enable-checking=release --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-8 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl --with-system-zlib --with-pkgversion='Homebrew GCC 8.3.0' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --disable-multilib --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
Thread model: posix
gcc version 8.3.0 (Homebrew GCC 8.3.0)


Code: [Select]

gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin


Code: [Select]

gfortran -v
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/8.3.0/libexec/gcc/x86_64-apple-darwin18.2.0/8.3.0/lto-wrapper
Target: x86_64-apple-darwin18.2.0
Configured with: ../configure --build=x86_64-apple-darwin18.2.0 --prefix=/usr/local/Cellar/gcc/8.3.0 --libdir=/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 --disable-nls --enable-checking=release --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-8 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl --with-system-zlib --with-pkgversion='Homebrew GCC 8.3.0' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --disable-multilib --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
Thread model: posix
gcc version 8.3.0 (Homebrew GCC 8.3.0)


From my petsc configure.log

Code: [Select]

Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --download-parmetis --download-superlu_dist --download-openmpi --download-ml --download-hypre --download-metis --download-mumps --download-scalapack --download-blacs --with-debugging=0


[blackbird]

Dear Prof. Govindjee,

thank you very much for these details. I will try to follow that setup in the next days to resolve my problems during the update from 8.4 to 8.5.

First issue is about the configuration of petsc. At my place, there is a problem with your configuration setup. I get the output:

Code: [Select]

         
...
*******************************************************************************
UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
Cannot use SuperLU_DIST without enabling C++11, see --with-cxx-dialect=C++11
*******************************************************************************

I guess you did not observe this kind of problem, did you? Is there any additional OS setup for you?

[Prof. S. Govindjee]

I don't have any problem on my end.  Looks like your C++ is not the correct version for superlu.  If you do not use superlu then you can just remove it from the configuration line.

[blackbird]

Dear Prof. Govindjee,

you are right. One option is to exclude the superlu from the configuration and it is fine.

Another option is to include another option in the configure line, that is --with-cxx-dialect=C++11. That way, PETSC is configured also with SUPERLU.

This gives rise to the next problem - the parfeap 8.5 is not working properly, even when every compiler used is updated to the recent version 8. I prepared a small benchmark file (plate with hole and displacement boundary), that is working in 8.4 along with the output during the parfeap 8.5 solution. Could you please help?

After partitioning with

Code: [Select]

$FEAPHOME8_5/parfeap/feap, the solution has been obtained with

Code: [Select]

$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 8 $FEAPHOME8_5/parfeap/feap -ksp_type cg -pc_type jacobi

[Prof. S. Govindjee]

I was not even able to run your file in serial mode.  You are using 2 proportional loads but they are not defined in the file.

[blackbird]

It is an example in parfeap. The proportional loads are defined in "Ipar" in the lines 1296 to 1307

[Prof. S. Govindjee]

I was able to generate an error with your input file.  The error I see are of the form

Code: [Select]

[1]PETSC ERROR: Argument out of range
[1]PETSC ERROR: key 424 is greater than largest key allowed 423
Is that what you see too?

[blackbird]

This is the output I get on the terminal:

Code: [Select]

    F I N I T E   E L E M E N T   A N A L Y S I S   P R O G R A M

           FEAP (C) Regents of the University of California
                         All Rights Reserved.
                       VERSION: Release 8.5.2h     
                          DATE: 23 January 2018     

         Files are set as:   Status    Filename

           Input   (read ) : Exists  Ipar_0001                       
           Output  (write) : Exists  Opar_0001                       
           Restart (read ) : New     Rpar_0001                       
           Restart (write) : New     Rpar_0001                       
           Plots   (write) : New     Ppar_0001                       

         Caution, existing write files will be overwritten.

         Are filenames correct?( y or n; r = redefine all, s = stop) :y

         R U N N I N G    F E A P    P R O B L E M    N O W

         Parallel Solution: Total Number of Tasks =    8

          --> Please report errors by e-mail to:
              feap@berkeley.edu

  *ERROR* Residual norm is NaN or Inf
 --> ERRORS OCCURRED: For details see file: Opar_0001

Note: The following floating-point exceptions are signalling: IEEE_OVERFLOW_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
Note: The following floating-point exceptions are signalling: IEEE_OVERFLOW_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
Note: The following floating-point exceptions are signalling: IEEE_OVERFLOW_FLAG
Note: The following floating-point exceptions are signalling: IEEE_OVERFLOW_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
Note: The following floating-point exceptions are signalling: IEEE_OVERFLOW_FLAG
Note: The following floating-point exceptions are signalling: IEEE_OVERFLOW_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
Note: The following floating-point exceptions are signalling: IEEE_OVERFLOW_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
Note: The following floating-point exceptions are signalling: IEEE_OVERFLOW_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL

The Ofile is included in the initial problem description

[arktik]

I am facing a similar problem. I want to compile v8.5 with GCC-8.3.1, OpenMPI-3.1.3 and PETSc-3.11.1.

To use PETSc-3.11.1, I had to make a minor change in parfeap/upremas.F and parfeap/usolve.F. The compilation finished without error. Serial feap binary works. Parfeap binary works only when PETSc is OFF in solve.xxx. With PETsc ON, the residual norm is NaN.

Previously, I had perfectly running v8.5 with GCC-7.3.1, OpenMPI-3.1.0 and PETSc-3.9.2. Has anyone found a solution yet?

Thanks! 

[blackbird]

Allthough I can not provide a proper explanation, the workaround is to use a GCC-6.3.0 compiler both for your PETSC installation as well as for the compilation of your (par)FEAP