FEAP User Forum
FEAP => Parallel FEAP => Topic started by: blackbird on March 22, 2018, 05:04:18 AM
-
I tried to compile parFEAP 8.5 with petsc 3.8.3 on a Ubuntu OS. Given the configure for petsc as
./configure --with-cc=gcc --with-fc=gfortran --download-mpich --download-fblaslapack --download-parmetis --download-metis
the subsequent "make install" in the parfeap folder results in a couple of errors like
/home/cs/FEAP85/petsc-3.8.3/gnu-opt/bin/mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/home/cs/FEAP85/petsc-3.8.3/include -I/home/cs/FEAP85/petsc-3.8.3/gnu-opt/include -I/home/cs/FEAP85/ver85/include -I/home/cs/FEAP85/ver85/include/integer8 -o parstop.o parstop.F
pfeapc.h:14.35:
Included at parstop.F:35:
type(tKSP) kspsol
1
Error: Derived type variable 'kspsol' in COMMON at (1) has neither the SEQUENCE nor the BIND(C) attribute
make: [parstop.o] Fehler 1 (ignoriert)
the errors occur for parbmat.o, parstop.o, ppmodin.o, parform.o, psprojb.o, mpi_start.o, psubsp.o, scalev.o, uasble.o, uasblem.o, upremas.o, uassdk.o, uassdm.o, usolve.o - each for the kspsol
Do you know this issue? Did I miss something in the petsc configuration?
-
To use the 3.8.x PETSc releases there are a number of updates to version 8.5. Please see the download site which has a patch that you can download.
-
do you talk about the
parfeapPETSCv383update.tgz
in your download folder? Of course, this was the first thing to update the code with it. However, the problem is given with that update.
-
That should have worked then.
Can you make sure your PETSc build is correct; make sure that the FORTRAN test cases in petsc-3.8.3/src/ksp/ksp/examples/tests compile for you. Try also the FORTRAN examples in petsc-3.8.3/src/ksp/ksp/examples/tutorials
-
One other question. Which version of gfortran are you using?
-
I updated my gfortran to version 7 (see post in http://feap.berkeley.edu/forum/index.php?topic=1861.msg7663#msg7663) and for compilation I use the direct path to this version.Therefore, my petsc configurewas actually:
./configure --with-cc=/usr/bin/gcc-7 --with-fc=/usr/bin/gfortran-7 --download-mpich --download-fblaslapack --download-parmetis --download-metis
I really think something is wrong with the PETSC build. It is not possible to compile the tutorial fotran file you mentioned. In this folder (~/FEAP85/petsc-3.8.3/src/ksp/ksp/examples/tutorials), the call
make ex1f
gives the error
~/FEAP85/petsc-3.8.3/src/ksp/ksp/examples/tutorials$ make ex1f
/home/cs/FEAP85/petsc-3.7.7/gnu-opt/bin/mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/home/cs/FEAP85/petsc-3.7.7/include -I/home/cs/FEAP85/petsc-3.7.7/gnu-opt/include -o ex1f.o ex1f.F90
ex1f.F90:12:18:
use petscksp
1
Error: USE statement at (1) cannot follow PARAMETER statement at (2)
ex1f.F90:13:19:
implicit none
1
Error: IMPLICIT NONE statement at (1) cannot follow PARAMETER statement at (2)
ex1f.F90:55:9:
SETERRA(PETSC_COMM_WORLD,1,' ')
1
Error: Unclassifiable statement at (1)
ex1f.F90:63:48:
call PetscOptionsGetInt(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER, &
1
Error: Symbol ‘petsc_null_options’ at (1) has no IMPLICIT type
make: [ex1f.o] Fehler 1 (ignoriert)
/home/cs/FEAP85/petsc-3.7.7/gnu-opt/bin/mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -o ex1f ex1f.o -Wl,-rpath,/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -L/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -Wl,-rpath,/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -lpetsc -lparmetis -lmetis -lflapack -lfblas -lX11 -lssl -lcrypto -lpthread -lm -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath -lmpicxx -lstdc++ -lm -Wl,-rpath,/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -L/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/cs/FEAP85/petsc-3.7.7/gnu-opt/lib -lmpi -lgcc_s -ldl
gfortran-7: error: ex1f.o: Datei oder Verzeichnis nicht gefunden
make: [ex1f] Fehler 1 (ignoriert)
/bin/rm -f ex1f.o
As far as I can tell, gfortran-7 gives the error.
-
I see your problem!
Your compile line is calling off to petsc-3.7.7!
You need to properly set your PETSC_DIR and PETSC_ARCH variables for the testing.
Also check that
make PETSC_DIR=/home/cs/FEAP85/petsc-3.8.3 PETSC_ARCH=gnu-opt test
works correctly.
-
Oh, I am sorry for that. It was an artefact of my try to compile with petsc3.7.7 - with this older version 3.7.7 everything works fine.
The problem is 3.8.3 and I tried with the correct PETSC_DIR and PETSC_ARCH variables. Also I tried 3.8.4 now, it shows the same problem with kspsol.
-
okay, the test cases of petsc are running now properly for me for version 3.8.4. They also use KSP for variable declaration, e.g. ex1f, however, the declaration is local. Could the common in pfeapc.h be the problem here?
-
The common block should not be a problem. Can you try changing the petsc test to use a common block for the KSP object?
Do you have another compiler you can try? Maybe there is an issue with version 7.
-
I checked with gfortran-6 - the issue remains the same. Therefore, I do not think, it is an issue with the compiler. Which compiler do you use?
I modified ex1f.F90 in the PETSC testcase folder ($(PETSC_DIR)/src/ksp/ksp/examples/tutorials) such, that the variable declaration
KSP ksp
in line 31 of the original file is now given in a header file ksp_test.h - which works fine. For the case that this header file additionally contains a common declaration for the ksp similar to the use in feap, i.e.
KSP ksp
common /pfeapc/ ksp
then the compilation has the error
~/FEAP85/petsc-3.8.4/src/ksp/ksp/examples/tutorials$ make ex1f_mod
/home/cs/FEAP85/petsc-3.8.4/gnu-opt/bin/mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/home/cs/FEAP85/petsc-3.8.4/include -I/home/cs/FEAP85/petsc-3.8.4/gnu-opt/include -o ex1f_mod.o ex1f_mod.F90
/home/cs/FEAP85/petsc-3.8.4/include/petsc/finclude/ksp_test.h:1:32:
KSP ksp
1
Error: Derived type variable ‘ksp’ in COMMON at (1) has neither the SEQUENCE nor the BIND(C) attribute
make: [ex1f_mod.o] Fehler 1 (ignoriert)
/home/cs/FEAP85/petsc-3.8.4/gnu-opt/bin/mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -o ex1f_mod ex1f_mod.o -Wl,-rpath,/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -L/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -Wl,-rpath,/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/6 -L/usr/lib/gcc/x86_64-linux-gnu/6 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -lpetsc -lflapack -lfblas -lparmetis -lmetis -lX11 -lpthread -lm -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath -lmpicxx -lstdc++ -lm -Wl,-rpath,/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -L/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/6 -L/usr/lib/gcc/x86_64-linux-gnu/6 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/cs/FEAP85/petsc-3.8.4/gnu-opt/lib -lmpi -lgcc_s -ldl
gfortran-6: error: ex1f_mod.o: Datei oder Verzeichnis nicht gefunden
make: [ex1f_mod] Fehler 1 (ignoriert)
/bin/rm -f ex1f_mod.o
As far as I can tell, the problem here is the usage of the KSP type within the common block. Do you agree? I append the corresponding files and the modified makefile, so the compile is done via
make ex1f_mod
- is it possible for you to reproduce this error? Note that the ksp_test.h needs to be in the folder $(PETSC_DIR)/include/petsc/finclude
-
I have managed to reproduce the error. It seems to be an issue with gfortran as it works just fine with the intel compiler.
We will look into it further.
-
I have one work around.
Convert the common block to a module and then use that module. So for your simple example
module pfeapc
#include <petsc/finclude/petscksp.h>
use petscksp
KSP ksp
end module pfeapc
followed by a
use pfeapc
in the source files. To get this correct in FEAP will take a bit of work and I am not completely sure what the best structure will be for a clean implementation. Eventually FEAP will have to convert to modules as common is slated for obsolescence.
-
Thank you a lot for this workaround!
-
Dear FEAP Admin,
with the workaround, the ksp derived type issue is fixed. However, the next problem is in subroutine mpi_start.F
mpi_start.F:22:9:
use petscksp
1
Error: ‘mpi_start’ of module ‘petscksp’, imported at (1), is also the name of the current program unit
make: [mpi_start.o]
As far I can tell, the name of this subroutine is a duplicate of something already declared in the petsc module. Furthermore, I did not find any call of the mpi_start subroutine at all.
However, without this routine, the run of parfeap is not possible and fails with the meesages
PetscInitialize() must be called before PetscFinalize()
and
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x7ff032f34f0d in ???
#1 0x7ff032f3414d in ???
#2 0x7ff032432caf in ???
#3 0x7ff0338d456a in PetscOptionsFindPair_Private
at /home/cs/FEAP85/petsc-3.8.3/src/sys/objects/options.c:1294
#4 0x7ff0338d8f9e in PetscOptionsGetString
at /home/cs/FEAP85/petsc-3.8.3/src/sys/objects/options.c:2267
#5 0x7ff0338fb411 in petscoptionsgetstring_
at /home/cs/FEAP85/petsc-3.8.3/src/sys/objects/ftn-custom/zoptionsf.c:233
#6 0x408807 in pdomain_
at /home/cs/FEAP85/ver85/parfeap/pdomain.F:316
#7 0x43b973 in pcontr_
at /home/cs/FEAP85/ver85/program/pcontr.f:972
#8 0x4045fd in feap
at /home/cs/FEAP85/ver85/main/feap85.f:185
#9 0x404634 in main
at /home/cs/FEAP85/ver85/main/feap85.f:191
-
strange you see this with 3.8.3 -- I did not. But I know the problem occurs with version 3.9.0.
the fix is to rename mpi_start to mpi_start_feap. do this throughout the whole code.
-
Maybe this is an issue of MPICH version? I configured the petsc today and the log says, mpich-3.3a2 has been found in git and installed.
I changed the subroutine name (in $FEAPHOME8_5/parfeap/mpi_start.F and $FEAPHOME8_5/program/mpi_start.f) and the respective call in ($FEAPHOME8_5/unix/pstart.f) - now my benchmark problem is working fine.
Many Thanks!
-
Could be. I normally use openmpi for the MPI interface, previously v1.8.3 but now v3.0.1.
FEAP version 8.6 will have these updates made to it.
-
Hi,
I just wanted to add here that I faced very similar problems with parallel feap installation using petsc-3.9.2. Those problems were also resolved by following the above discussion. Thanks.
Here is a summary of to do tasks for someone who is facing similar problems:
1. create module pfeapc.F to replace common block pfeapc.h.
module pfeapc
# include <petsc/finclude/petscksp.h>
use petscksp
Vec :: rhs, sol, xvec, diag
Vec :: yvec, zvec
Vec :: Mdiag, Msqrt
Mat :: Kmat, Mmat, Pmat
KSP :: kspsol, ksp
end module pfeapc
2. find all subroutines where pfeapc.h header file is used.
grep -r 'pfeapc' ver85/
3. Add use pfeapc and comment include "pfeapc.h" in those subroutines.
4. Find all subroutines where mpi_start routine is used.
grep -r 'mpi_start' ver85/
5. Rename mpi_start to mpi_start_feap in all those subroutines as well as makefiles.
6. make install in parfeap directory.
-
You need to change the name of our routine. Later releases will use mpi_start_feap.F as the name.
-
Could be. I normally use openmpi for the MPI interface, previously v1.8.3 but now v3.0.1.
FEAP version 8.6 will have these updates made to it.
Dear Prof. Govindjee,
could you please give a little bit more detailed information on how to use openmpi with feap. Specifically, when petsc is configured, the option --download-mpich is organizing the mpi installation. Which steps are required for openmpi instead?
-
change to --download-openmpi instead of --download-mpich
-
Please, can you provide a FEAP 8.5 release which contains all necessary changes to run ParFEAP 8.5? It is not just two or three changes and the explanation is distributed over several forum threads. Would be really nice to have a clear and unique code basis for discussion.
-
we renamed to mpi_start_feap to avoid the conflict. Be sure to do in both serial and parallel code where used.
-
I think that these changes affect about 50% of the parfeap files, due to adding use pfeapc and removing include "pfeapc.h" There are also changes necessary at FEAP regarding PETSc 3.8 (http://feap.berkeley.edu/forum/index.php?topic=1880.msg7528#msg7528).
I would like to ask politely to provide a new FEAP 8.5 release with these changes. I believe that we will have issues with reporting bugs and posting FEAP improvements, if we use different code bases.
-
Yes it is a bit of a challenge to keep up with the recent releases of petsc, though following the release notes it is certainly possible to make the updates.
We are actively working on a new release (8.6) which will play nice with the latest petsc release.