parfeap 8.5 on Ubuntu

[blackbird]

I tried to compile parFEAP 8.5 with petsc 3.8.3 on a Ubuntu OS. Given the configure for petsc as

Code: [Select]

./configure --with-cc=gcc --with-fc=gfortran --download-mpich --download-fblaslapack --download-parmetis --download-metis

the subsequent "make install" in the parfeap folder results in a couple of errors like

Code: [Select]

/home/cs/FEAP85/petsc-3.8.3/gnu-opt/bin/mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g    -I/home/cs/FEAP85/petsc-3.8.3/include -I/home/cs/FEAP85/petsc-3.8.3/gnu-opt/include -I/home/cs/FEAP85/ver85/include -I/home/cs/FEAP85/ver85/include/integer8         -o parstop.o parstop.F
pfeapc.h:14.35:
    Included at parstop.F:35:

      type(tKSP)             kspsol                                     
                                   1
Error: Derived type variable 'kspsol' in COMMON at (1) has neither the SEQUENCE nor the BIND(C) attribute
make: [parstop.o] Fehler 1 (ignoriert)

the errors occur for parbmat.o, parstop.o, ppmodin.o, parform.o, psprojb.o, mpi_start.o, psubsp.o, scalev.o, uasble.o, uasblem.o, upremas.o, uassdk.o, uassdm.o, usolve.o - each for the kspsol

Do you know this issue? Did I miss something in the petsc configuration?

[Prof. S. Govindjee]

To use the 3.8.x PETSc releases there are a number of updates to version 8.5.  Please see the download site which has a patch that you can download.

[blackbird]

do you talk about the

parfeapPETSCv383update.tgz

in your download folder? Of course, this was the first thing to update the code with it. However, the problem is given with that update.

[Prof. S. Govindjee]

That should have worked then.

Can you make sure your PETSc build is correct; make sure that the FORTRAN test cases in petsc-3.8.3/src/ksp/ksp/examples/tests compile for you.  Try also the FORTRAN examples in petsc-3.8.3/src/ksp/ksp/examples/tutorials

[Prof. S. Govindjee]

One other question.  Which version of gfortran are you using?

[blackbird]

I updated my gfortran to version 7 (see post in http://feap.berkeley.edu/forum/index.php?topic=1861.msg7663#msg7663) and for compilation I use the direct path to this version.Therefore, my petsc configurewas actually:

Code: [Select]

./configure --with-cc=/usr/bin/gcc-7 --with-fc=/usr/bin/gfortran-7 --download-mpich --download-fblaslapack --download-parmetis --download-metis
I really think something is wrong with the PETSC build. It is not possible to compile the tutorial fotran file you mentioned. In this folder (~/FEAP85/petsc-3.8.3/src/ksp/ksp/examples/tutorials), the call

Code: [Select]

make ex1f
gives the error

Code: [Select]

~/FEAP85/petsc-3.8.3/src/ksp/ksp/examples/tutorials$ make ex1f
/home/cs/FEAP85/petsc-3.7.7/gnu-opt/bin/mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g   -I/home/cs/FEAP85/petsc-3.7.7/include -I/home/cs/FEAP85/petsc-3.7.7/gnu-opt/include    -o ex1f.o ex1f.F90
ex1f.F90:12:18:

       use petscksp
                  1
Error: USE statement at (1) cannot follow PARAMETER statement at (2)
ex1f.F90:13:19:

       implicit none
                   1
Error: IMPLICIT NONE statement at (1) cannot follow PARAMETER statement at (2)
ex1f.F90:55:9:

          SETERRA(PETSC_COMM_WORLD,1,' ')
         1
Error: Unclassifiable statement at (1)
ex1f.F90:63:48:

       call PetscOptionsGetInt(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,         &
                                                1
Error: Symbol ‘petsc_null_options’ at (1) has no IMPLICIT type
make: [ex1f.o] Fehler 1 (ignoriert)
/home/cs/FEAP85/petsc-3.7.7/gnu-opt/bin/mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g  -o ex1f ex1f.o  -Wl,-rpath,/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -L/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -Wl,-rpath,/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -lpetsc -lparmetis -lmetis -lflapack -lfblas -lX11 -lssl -lcrypto -lpthread -lm -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath -lmpicxx -lstdc++ -lm -Wl,-rpath,/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -L/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -lmpi -lgcc_s -ldl
gfortran-7: error: ex1f.o: Datei oder Verzeichnis nicht gefunden
make: [ex1f] Fehler 1 (ignoriert)
/bin/rm -f ex1f.o

As far as I can tell, gfortran-7 gives the error.

[Prof. S. Govindjee]

I see your problem!

Your compile line is calling off to petsc-3.7.7!

You need to properly set your PETSC_DIR and PETSC_ARCH variables for the testing.

Also check that
make PETSC_DIR=/home/cs/FEAP85/petsc-3.8.3  PETSC_ARCH=gnu-opt test
works correctly.

[blackbird]

Oh, I am sorry for that. It was an artefact of my try to compile with petsc3.7.7 - with this older version 3.7.7 everything works fine.

The problem is 3.8.3 and I tried with the correct PETSC_DIR and PETSC_ARCH variables. Also I tried 3.8.4 now, it shows the same problem with kspsol.

[blackbird]

okay, the test cases of petsc are running now properly for me for version 3.8.4. They also use KSP for variable declaration, e.g. ex1f, however, the declaration is local. Could the common in pfeapc.h be the problem here?

[Prof. S. Govindjee]

The common block should not be a problem.  Can you try changing the petsc test to use a common block for the KSP object?

Do you have another compiler you can try?  Maybe there is an issue with version 7.

[blackbird]

I checked with gfortran-6 - the issue remains the same. Therefore, I do not think, it is an issue with the compiler. Which compiler do you use?

I modified ex1f.F90 in the PETSC testcase folder ($(PETSC_DIR)/src/ksp/ksp/examples/tutorials) such, that the variable declaration

Code: [Select]

KSP ksp

in line 31 of the original file is now given in a header file ksp_test.h - which works fine. For the case that this header file additionally contains a common declaration for the ksp similar to the use in feap, i.e.

Code: [Select]

     
KSP             ksp
      common /pfeapc/ ksp

then the compilation has the error

Code: [Select]

~/FEAP85/petsc-3.8.4/src/ksp/ksp/examples/tutorials$ make ex1f_mod
/home/cs/FEAP85/petsc-3.8.4/gnu-opt/bin/mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g    -I/home/cs/FEAP85/petsc-3.8.4/include -I/home/cs/FEAP85/petsc-3.8.4/gnu-opt/include    -o ex1f_mod.o ex1f_mod.F90
/home/cs/FEAP85/petsc-3.8.4/include/petsc/finclude/ksp_test.h:1:32:

       KSP             ksp
                                1
Error: Derived type variable ‘ksp’ in COMMON at (1) has neither the SEQUENCE nor the BIND(C) attribute
make: [ex1f_mod.o] Fehler 1 (ignoriert)
/home/cs/FEAP85/petsc-3.8.4/gnu-opt/bin/mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g   -o ex1f_mod ex1f_mod.o  -Wl,-rpath,/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -L/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -Wl,-rpath,/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/6 -L/usr/lib/gcc/x86_64-linux-gnu/6 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -lpetsc -lflapack -lfblas -lparmetis -lmetis -lX11 -lpthread -lm -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath -lmpicxx -lstdc++ -lm -Wl,-rpath,/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -L/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/6 -L/usr/lib/gcc/x86_64-linux-gnu/6 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -lmpi -lgcc_s -ldl
gfortran-6: error: ex1f_mod.o: Datei oder Verzeichnis nicht gefunden
make: [ex1f_mod] Fehler 1 (ignoriert)
/bin/rm -f ex1f_mod.o

As far as I can tell, the problem here is the usage of the KSP type within the common block. Do you agree? I append the corresponding files and the modified makefile, so the compile is done via

Code: [Select]

make ex1f_mod
- is it possible for you to reproduce this error? Note that the ksp_test.h needs to be in the folder $(PETSC_DIR)/include/petsc/finclude

[FEAP_Admin]

I have managed to reproduce the error.  It seems to be an issue with gfortran as it works just fine with the intel compiler.
We will look into it further.

[FEAP_Admin]

I have one work around.

Convert the common block to a module and then use that module.  So for your simple example

Code: [Select]

      module pfeapc
#include <petsc/finclude/petscksp.h>
                     use petscksp
        KSP  ksp
      end module pfeapc
followed by a

Code: [Select]

      use pfeapc
in the source files.   To get this correct in FEAP will take a bit of work and I am not completely sure what the best structure will be for a clean implementation.  Eventually FEAP will have to convert to modules as common is slated for obsolescence.

[blackbird]

Thank you a lot for this workaround!

[blackbird]

Dear FEAP Admin,

with the workaround, the ksp derived type issue is fixed. However, the next problem is in subroutine mpi_start.F

Code: [Select]

mpi_start.F:22:9:

       use                      petscksp
         1
Error: ‘mpi_start’ of module ‘petscksp’, imported at (1), is also the name of the current program unit
make: [mpi_start.o]

As far I can tell, the name of this subroutine is a duplicate of something already declared in the petsc module. Furthermore, I did not find any call of the mpi_start subroutine at all.

However, without this routine, the run of parfeap is not possible and fails with the meesages

Code: [Select]

PetscInitialize() must be called before PetscFinalize()

and

Code: [Select]

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7ff032f34f0d in ???
#1  0x7ff032f3414d in ???
#2  0x7ff032432caf in ???
#3  0x7ff0338d456a in PetscOptionsFindPair_Private
at /home/cs/FEAP85/petsc-3.8.3/src/sys/objects/options.c:1294
#4  0x7ff0338d8f9e in PetscOptionsGetString
at /home/cs/FEAP85/petsc-3.8.3/src/sys/objects/options.c:2267
#5  0x7ff0338fb411 in petscoptionsgetstring_
at /home/cs/FEAP85/petsc-3.8.3/src/sys/objects/ftn-custom/zoptionsf.c:233
#6  0x408807 in pdomain_
at /home/cs/FEAP85/ver85/parfeap/pdomain.F:316
#7  0x43b973 in pcontr_
at /home/cs/FEAP85/ver85/program/pcontr.f:972
#8  0x4045fd in feap
at /home/cs/FEAP85/ver85/main/feap85.f:185
#9  0x404634 in main
at /home/cs/FEAP85/ver85/main/feap85.f:191