Author Topic: Identify ghost elements in each partition  (Read 11252 times)

lzhu

  • New Member
  • *
  • Posts: 9
Identify ghost elements in each partition
« on: February 18, 2017, 01:13:44 PM »
Hello,

I need to do an integral over the entire domain. My current workflow for domain integral is:

Integral within elements --> Each CPU collect the element-wise integration --> MPI_Allreduce to sum the results from each CPU

However, I notice that there are some ghost elements in each partition, which results in the error of the integration. Is there a list or reference that I can use to identify the ghost elements in each partition?

Regards,
Lixing

FEAP_Admin

  • Administrator
  • FEAP Guru
  • *****
  • Posts: 993
Re: Identify ghost elements in each partition
« Reply #1 on: February 18, 2017, 02:54:36 PM »
I'm not sure that we write this out.  But have a look in the partitioned input files for the keyword
EREGion and check if the list generated is the ghost list.  If so, then you can check for element with region code 21 and they will correspond to ghost elements.

If this does not work then you will need to add the output to parfeap/pmarc7.F .

Let us know what you find.

lzhu

  • New Member
  • *
  • Posts: 9
Re: Identify ghost elements in each partition
« Reply #2 on: February 18, 2017, 03:22:13 PM »
Yes, the partitioned input files have this section: "EREGions  ! Region 21 ignored for STREss"

Thank you very much for this hint.

FEAP_Admin

  • Administrator
  • FEAP Guru
  • *****
  • Posts: 993
Re: Identify ghost elements in each partition
« Reply #3 on: February 18, 2017, 05:49:26 PM »
The main question now is, is this the list of ghost elements...

lzhu

  • New Member
  • *
  • Posts: 9
Re: Identify ghost elements in each partition
« Reply #4 on: February 18, 2017, 07:43:47 PM »
I briefly look at the "pmarc7.F". The array "domn(2,*)" includes the elements and ghost elements in the domain and if domn(1,*) = -d, then it's been output to the region 21. I did not go over the conditional statement very carefully, but I think these elements are likely to be ghost elements. I'll perhaps visualize the element list to check if elements in region 21 locate at the domain boundary of each CPU. If you could confirm this conclusion, that would be very helpful.

FEAP_Admin

  • Administrator
  • FEAP Guru
  • *****
  • Posts: 993
Re: Identify ghost elements in each partition
« Reply #5 on: February 19, 2017, 01:21:17 PM »
I checked a simple example and it looks like the EREGions list is in fact the list of ghost elements.
Thus when you process your elements you can check IX(nen1-1,e).  If it is equal to 21, then element e is a ghost element.

Please double check on your own examples to confirm.

lzhu

  • New Member
  • *
  • Posts: 9
Re: Identify ghost elements in each partition
« Reply #6 on: February 19, 2017, 01:42:01 PM »
Thank you very much for your time!

lzhu

  • New Member
  • *
  • Posts: 9
Re: Identify ghost elements in each partition
« Reply #7 on: February 21, 2017, 08:30:40 AM »
Can we identify the "Partn Stress Elements" also?

I tried to do integration over the domain in parallel FEAP by neglecting the contributions from Ghost elements. However, the results are not accurate. I realized that some elements, though not tagged as Ghost elements, show in more than one partition. I think the right way to do integration over the entire domain is summation from Ghost Stress Elements and Partn Stress Elements.

Now I just use check ix(1:nel,e) with numpn to see if it is a Partn Stress Elements
« Last Edit: February 21, 2017, 09:02:34 AM by lzhu »

FEAP_Admin

  • Administrator
  • FEAP Guru
  • *****
  • Posts: 993
Re: Identify ghost elements in each partition
« Reply #8 on: February 21, 2017, 08:19:29 PM »
I will try to check also.  If you do it the way you describe, you have to make sure that ALL nodes 1:nel are less than or equal to numpn.

lzhu

  • New Member
  • *
  • Posts: 9
Re: Identify ghost elements in each partition
« Reply #9 on: February 21, 2017, 10:42:13 PM »
Thank you again for your suggestion.

jbruss

  • Jr. Member
  • **
  • Posts: 36
Re: Identify ghost elements in each partition
« Reply #10 on: September 08, 2018, 07:35:16 PM »
Hello -

Was this issue ever resolved? I have been trying to perform the same task (integrating over the domain ignoring ghost elements) and my result is not correct. I'm not sure what the difference is between ghost elements and Partn Stress Elements but I also tried the suggested strategy for avoiding those as well, which also did not work for me. I'm wondering now if it is possible to perform an integration over the domain accurately in parFEAP?

Thank you for your help,
Jonathan

Prof. S. Govindjee

  • Administrator
  • FEAP Guru
  • *****
  • Posts: 1164
Re: Identify ghost elements in each partition
« Reply #11 on: September 09, 2018, 09:08:13 AM »
I think you have to be careful here and define more precisely what you mean by ghost elements.  parFEAP does not operate with ghost elements.  It uses ghost nodes.

The basic idea in parFEAP is that we partition the node graph.  Thus each processor owns certain nodes.  In order to assemble the internal forces on a node, one needs additional nodes which are known as the ghost nodes.  Nodes 1 to numpn are a processors nodes, nodes numpn+1 to numnp are the ghost nodes.

In this conception, if you think of a simple mesh sliced down the middle for two partitions, then there is a column/row of elements that split.  Naively one would want to think of these split elements as ghost elements but in the traditional sense they are not, as they appear in both partitions and neither partition is flagged as owning them.  To construct a concept of ghost elements, one needs to introduce a rule for deciding which processor 'owns' the element but we never do that since parFEAP is based on a node graph partition and does not need that information as all FE integral computations we perform are node related: $$\int N_A(x) f(x) dx$$.

Notwithstanding, if you keep track of the elements contributing to a domain integral then it should be possible to avoid double counting parts of the domain and also importantly not missing any parts, knowing the global element numbers is perhaps essential for this task; most likely this is most easily handled in parfeap/pmacr7.F during the partition phase.

« Last Edit: September 09, 2018, 09:10:43 AM by Prof. S. Govindjee »

jbruss

  • Jr. Member
  • **
  • Posts: 36
Re: Identify ghost elements in each partition
« Reply #12 on: September 10, 2018, 05:33:38 PM »
Okay thank you Professor Govindjee. I'll try to figure out how to extract the global element numbers from pmacr7.F