oomph::DoubleVectorHaloScheme Class Reference

#include <double_vector_with_halo.h>

Public Member Functions
	DoubleVectorHaloScheme (LinearAlgebraDistribution *const &dist_pt, const Vector< unsigned > &required_global_eqn)
unsigned	n_halo_values () const
	Return the number of halo values. More...
LinearAlgebraDistribution *&	distribution_pt ()
void	setup_halo_dofs (const std::map< unsigned, double * > &halo_data_pt, Vector< double * > &halo_dof_pt)
unsigned	local_index (const unsigned &global_eqn)
	Return the local index associated with the global equation. More...

Private Attributes
std::map< unsigned, unsigned >	Local_index
Vector< unsigned >	Haloed_eqns
Vector< int >	Haloed_n
Vector< int >	Haloed_displacement
Vector< unsigned >	Halo_eqns
Vector< int >	Halo_n
Vector< int >	Halo_displacement
LinearAlgebraDistribution *	Distribution_pt
	Store the distribution that was used to setup the halo scheme. More...

Friends
class	DoubleVectorWithHaloEntries

Detailed Description

A class that stores the halo/haloed entries required when using a DoubleVectorWithHaloEntries. This is a separate class so thay many different Vectors can share the same object. The constructor requires the distribution of the DoubleVector (if you pass in a different distribution things will go badly wrong) and a vector that specifies which GLOBAL eqn numbers are required on each processor.

Constructor & Destructor Documentation

DoubleVectorHaloScheme()

oomph::DoubleVectorHaloScheme::DoubleVectorHaloScheme	(	LinearAlgebraDistribution *const &	dist_pt,
		const Vector< unsigned > &	required_global_eqn
	)

Constructor that sets up the required information communicating between all processors. Requires two "all to all" communications. Arguments are the distribution of the DoubleVector and a Vector of global unknowns required on this processor.

    : Distribution_pt(dist_pt)
  {
#ifdef OOMPH_HAS_MPI
    // Only bother to do anything if the vector is distributed
    if (dist_pt->distributed())
    {
      // First create temporary maps for halo requests.
      // Using the map structure ensures that the data will be sorted
      // into processor order: the order of the unsigned integer key
 
      // These are the halo requests indexed by master processor and the
      // local equation number that is to be haloed on that processor
      std::map<unsigned, Vector<unsigned>> to_be_haloed;
      // These are the halo requests indexed by master processor and the
      // index in the additional storage
      // that corresponds to the halo data on this processor
      std::map<unsigned, Vector<unsigned>> halo_entries;
 
      // Find rank of current processor
      const unsigned my_rank =
        static_cast<int>(dist_pt->communicator_pt()->my_rank());
      // Loop over the desired equations (on this processor)
      // and work out whether we need to halo them according to the
      // given distribution
      const unsigned n_global_eqn = required_global_eqn.size();
      // Index for the locally stored halo values
      unsigned index = 0;
      for (unsigned n = 0; n < n_global_eqn; n++)
      {
        // Cache the required GLOBAL equation number
        const unsigned i_global = required_global_eqn[n];
        // Where is it stored?
        unsigned rank_of_global = dist_pt->rank_of_global_row(i_global);
        // If the equation is not stored locally then
        // populate the two maps
        if (my_rank != rank_of_global)
        {
          // Work out the local entry on the appropriate processor
          unsigned i_local = i_global - dist_pt->first_row(rank_of_global);
          // Mark the local storage index as halo with rank_of_global as master
          halo_entries[rank_of_global].push_back(index);
          // Mark the local equation of the rank_of_global as to be
          // haloed
          to_be_haloed[rank_of_global].push_back(i_local);
          // Store the local index corresponding to the global equation
          Local_index[i_global] = index;
          // Increment the index
          ++index;
        }
      }
 
      // We now need to tell the other processors which of their data are
      // haloed on this processor
 
      // First find out how many processors there are!
      const int n_proc = dist_pt->communicator_pt()->nproc();
 
      // Setup storage for number of data to be sent to each processor
      Vector<int> send_n(n_proc, 0);
      Vector<int> send_displacement(n_proc, 0);
      int send_data_count = 0;
 
      // Iterate over the entries in the map
      // This will be in rank order because of the ordering of the map
      for (std::map<unsigned, Vector<unsigned>>::iterator it =
             to_be_haloed.begin();
           it != to_be_haloed.end();
           ++it)
      {
        const unsigned rank = it->first;
        const unsigned size_ = it->second.size();
        // The displacement is the current number of data
        send_displacement[rank] = send_data_count;
        // The number to send is just the size of the array
        send_n[rank] = static_cast<int>(size_);
        send_data_count += size_;
      }
 
      // Now send the number of haloed entries from every processor
      // to every processor
 
      // Receive the data directly into the storage for haloed daa
      Haloed_n.resize(n_proc, 0);
      MPI_Alltoall(&send_n[0],
                   1,
                   MPI_INT,
                   &Haloed_n[0],
                   1,
                   MPI_INT,
                   dist_pt->communicator_pt()->mpi_comm());
 
      // Prepare the data to be sent
      // Always resize to at least one
      if (send_data_count == 0)
      {
        ++send_data_count;
      }
      Vector<unsigned> send_data(send_data_count);
      // Iterate over the entries in the map
      unsigned count = 0;
      for (std::map<unsigned, Vector<unsigned>>::iterator it =
             to_be_haloed.begin();
           it != to_be_haloed.end();
           ++it)
      {
        // Iterate over the vector
        for (Vector<unsigned>::iterator it2 = it->second.begin();
             it2 != it->second.end();
             ++it2)
        {
          send_data[count] = (*it2);
          ++count;
        }
      }
 
      // Prepare the data to be received,
      // Again this can go directly into Haloed storage
      int receive_data_count = 0;
      Haloed_displacement.resize(n_proc);
      for (int d = 0; d < n_proc; d++)
      {
        // The displacement is the amount of data received so far
        Haloed_displacement[d] = receive_data_count;
        receive_data_count += Haloed_n[d];
      }
 
      // Now resize the receive buffer
      // Always make sure that it has size of at least one
      if (receive_data_count == 0)
      {
        ++receive_data_count;
      }
      Haloed_eqns.resize(receive_data_count);
      // Send the data between all the processors
      MPI_Alltoallv(&send_data[0],
                    &send_n[0],
                    &send_displacement[0],
                    MPI_UNSIGNED,
                    &Haloed_eqns[0],
                    &Haloed_n[0],
                    &Haloed_displacement[0],
                    MPI_UNSIGNED,
                    dist_pt->communicator_pt()->mpi_comm());
 
      // Finally, we translate the map of halo entries into the permanent
      // storage
      Halo_n.resize(n_proc, 0);
      Halo_displacement.resize(n_proc, 0);
 
      // Loop over all the entries in the map
      unsigned receive_haloed_count = 0;
      for (int d = 0; d < n_proc; d++)
      {
        // Pointer to the map entry
        std::map<unsigned, Vector<unsigned>>::iterator it =
          halo_entries.find(d);
        // If we don't have it in the map, skip
        if (it == halo_entries.end())
        {
          Halo_displacement[d] = receive_haloed_count;
          Halo_n[d] = 0;
        }
        else
        {
          Halo_displacement[d] = receive_haloed_count;
          const int size_ = it->second.size();
          Halo_n[d] = size_;
          // Resize the equations to be sent
          Halo_eqns.resize(receive_haloed_count + size_);
          for (int i = 0; i < size_; i++)
          {
            Halo_eqns[receive_haloed_count + i] = it->second[i];
          }
          receive_haloed_count += size_;
        }
      }
    }
#endif
  }

References oomph::LinearAlgebraDistribution::communicator_pt(), oomph::LinearAlgebraDistribution::distributed(), oomph::LinearAlgebraDistribution::first_row(), Halo_displacement, Halo_eqns, Halo_n, Haloed_displacement, Haloed_eqns, Haloed_n, i, Local_index, oomph::OomphCommunicator::my_rank(), n, oomph::OomphCommunicator::nproc(), and oomph::LinearAlgebraDistribution::rank_of_global_row().

Member Function Documentation

distribution_pt()

LinearAlgebraDistribution*& oomph::DoubleVectorHaloScheme::distribution_pt ( )

inline

Return the pointer to the distirbution used to setup the halo information

    {
      return Distribution_pt;
    }

References Distribution_pt.

Referenced by oomph::Problem::global_dof_pt().

local_index()

unsigned oomph::DoubleVectorHaloScheme::local_index ( const unsigned &  global_eqn )

inline

Return the local index associated with the global equation.

    {
      // Does the entry exist in the map
      std::map<unsigned, unsigned>::iterator it = Local_index.find(global_eqn);
      // If it does return it
      if (it != Local_index.end())
      {
        return it->second;
      }
      // Otherwise throw an error
      else
      {
        std::ostringstream error_stream;
        error_stream << "Global equation " << global_eqn << " "
                     << "has not been set as halo\n";
        throw OomphLibError(
          error_stream.str(), OOMPH_CURRENT_FUNCTION, OOMPH_EXCEPTION_LOCATION);
        return 0;
      }
    }

References Local_index, OOMPH_CURRENT_FUNCTION, and OOMPH_EXCEPTION_LOCATION.

Referenced by oomph::Problem::global_dof_pt(), and oomph::DoubleVectorWithHaloEntries::global_value().

n_halo_values()

unsigned oomph::DoubleVectorHaloScheme::n_halo_values ( ) const

inline

Return the number of halo values.

    {
      return Local_index.size();
    }

References Local_index.

setup_halo_dofs()

void oomph::DoubleVectorHaloScheme::setup_halo_dofs	(	const std::map< unsigned, double * > &	halo_data_pt,
		Vector< double * > &	halo_dof_pt
	)

Function that sets up a vector of pointers to halo data, index using the scheme in Local_index

Function that sets up a vector of pointers to halo data, index using the scheme in Local_index. The first arguement is a map of pointers to all halo data index by the global equation number

  {
    // How many entries are there in the map
    unsigned n_halo = Local_index.size();
    // Resize the vector
    halo_dof_pt.resize(n_halo);
 
    // Loop over all the entries in the map
    for (std::map<unsigned, unsigned>::iterator it = Local_index.begin();
         it != Local_index.end();
         ++it)
    {
      // Find the pointer in the halo_data_pt map
      std::map<unsigned, double*>::const_iterator it2 =
        halo_data_pt.find(it->first);
      // Did we find it
      if (it2 != halo_data_pt.end())
      {
        // Now set the entry
        halo_dof_pt[it->second] = it2->second;
      }
      else
      {
        std::ostringstream error_stream;
        error_stream << "Global equation " << it->first
                     << " reqired as halo is not stored in halo_data_pt\n";
 
        throw OomphLibError(
          error_stream.str(), OOMPH_CURRENT_FUNCTION, OOMPH_EXCEPTION_LOCATION);
      }
    }
  }

References Local_index, OOMPH_CURRENT_FUNCTION, and OOMPH_EXCEPTION_LOCATION.

Friends And Related Function Documentation

DoubleVectorWithHaloEntries

friend class DoubleVectorWithHaloEntries

friend

The DoubleVectorWithHaloEntries should be able to access the private data.

Member Data Documentation

Distribution_pt

LinearAlgebraDistribution* oomph::DoubleVectorHaloScheme::Distribution_pt

private

Store the distribution that was used to setup the halo scheme.

Referenced by distribution_pt().

Halo_displacement

Vector<int> oomph::DoubleVectorHaloScheme::Halo_displacement

private

Storage for the offsets of the processor data in the receive buffer

Referenced by DoubleVectorHaloScheme(), oomph::DoubleVectorWithHaloEntries::sum_all_halo_and_haloed_values(), and oomph::DoubleVectorWithHaloEntries::synchronise().

Halo_eqns

Vector<unsigned> oomph::DoubleVectorHaloScheme::Halo_eqns

private

Storage for all the entries that are to be received from other processors (received_from_proc0,received_from_proc1,...received_from_procn)

Referenced by DoubleVectorHaloScheme(), oomph::DoubleVectorWithHaloEntries::sum_all_halo_and_haloed_values(), and oomph::DoubleVectorWithHaloEntries::synchronise().

Halo_n

Vector<int> oomph::DoubleVectorHaloScheme::Halo_n

private

Storage for the number of entries to be received from each other processor

Referenced by DoubleVectorHaloScheme(), oomph::DoubleVectorWithHaloEntries::sum_all_halo_and_haloed_values(), and oomph::DoubleVectorWithHaloEntries::synchronise().

Haloed_displacement

Vector<int> oomph::DoubleVectorHaloScheme::Haloed_displacement

private

Storage for the offsets of the haloed entries for each processor in the packed Haloed_eqns array

Referenced by DoubleVectorHaloScheme(), oomph::DoubleVectorWithHaloEntries::sum_all_halo_and_haloed_values(), and oomph::DoubleVectorWithHaloEntries::synchronise().

Haloed_eqns

Vector<unsigned> oomph::DoubleVectorHaloScheme::Haloed_eqns

private

The haloed entries that will be sent in a format compatible with MPI_Alltoallv i.e. (send_to_proc0,send_to_proc1 ... send_to_procn)

Referenced by DoubleVectorHaloScheme(), oomph::DoubleVectorWithHaloEntries::sum_all_halo_and_haloed_values(), and oomph::DoubleVectorWithHaloEntries::synchronise().

Haloed_n

Vector<int> oomph::DoubleVectorHaloScheme::Haloed_n

private

Storage for the number of haloed entries to be sent to each processor

Referenced by DoubleVectorHaloScheme(), oomph::DoubleVectorWithHaloEntries::sum_all_halo_and_haloed_values(), and oomph::DoubleVectorWithHaloEntries::synchronise().

Local_index

std::map<unsigned, unsigned> oomph::DoubleVectorHaloScheme::Local_index

private

Storage for the translation scheme from global unknown to local index in the additional storage vector.

Referenced by oomph::DoubleVectorWithHaloEntries::build_halo_scheme(), DoubleVectorHaloScheme(), local_index(), n_halo_values(), and setup_halo_dofs().

---

The documentation for this class was generated from the following files:

• double_vector_with_halo.h
• double_vector_with_halo.cc