#include <trilinos_solver.h>

Inheritance diagram for oomph::TrilinosAztecOOSolver:

Public Types
enum	AztecOO_solver_types { CG , GMRES , BiCGStab }
	Enumerated list to define which AztecOO solver is used. More...

Public Member Functions
	TrilinosAztecOOSolver ()
	Constructor. More...

	~TrilinosAztecOOSolver ()
	Destructor - delete the solver and the matrices. More...

	TrilinosAztecOOSolver (const TrilinosAztecOOSolver &)=delete
	Broken copy constructor. More...

void	operator= (const TrilinosAztecOOSolver &)=delete
	Broken assignment operator. More...

void	enable_aztecoo_workaround_for_epetra_matrix_setup ()

void	disable_aztecoo_workaround_for_epetra_matrix_setup ()

bool	is_aztecoo_workaround_for_epetra_matrix_setup_enabled ()

void	clean_up_memory ()

void	solve (Problem *const &problem_pt, DoubleVector &solution)

void	solve (DoubleMatrixBase *const &matrix_pt, const DoubleVector &rhs, DoubleVector &solution)

void	resolve (const DoubleVector &rhs, DoubleVector &solution)

void	disable_resolve ()

void	enable_delete_matrix ()
	Call if the matrix can be deleted. More...

void	disable_delete_matrix ()
	Call if the matrix can not be deleted (default) More...

unsigned &	max_iter ()
	Access function to Max_iter. More...

unsigned	iterations () const
	Acess function to Iterations. More...

double &	tolerance ()
	Access function to Tolerance. More...

unsigned &	solver_type ()
	Access function to Solver_type. More...

double	jacobian_setup_time ()
	Function to return Jacobian_setup_time;. More...

double	linear_solver_solution_time ()
	Function to return Linear_solver_solution_time. More...

void	enable_assemble_serial_jacobian ()

void	disable_assemble_serial_jacobian ()
	Unset the assembly of the serial jacobian. More...

Public Member Functions inherited from oomph::IterativeLinearSolver
	IterativeLinearSolver ()

	IterativeLinearSolver (const IterativeLinearSolver &)=delete
	Broken copy constructor. More...

void	operator= (const IterativeLinearSolver &)=delete
	Broken assignment operator. More...

virtual	~IterativeLinearSolver ()
	Destructor (empty) More...

Preconditioner *&	preconditioner_pt ()
	Access function to preconditioner. More...

Preconditioner *const &	preconditioner_pt () const
	Access function to preconditioner (const version) More...

double &	tolerance ()
	Access to convergence tolerance. More...

unsigned &	max_iter ()
	Access to max. number of iterations. More...

void	enable_doc_convergence_history ()
	Enable documentation of the convergence history. More...

void	disable_doc_convergence_history ()
	Disable documentation of the convergence history. More...

void	open_convergence_history_file_stream (const std::string &file_name, const std::string &zone_title="")

void	close_convergence_history_file_stream ()
	Close convergence history output stream. More...

double	jacobian_setup_time () const

double	linear_solver_solution_time () const
	return the time taken to solve the linear system More...

virtual double	preconditioner_setup_time () const
	returns the the time taken to setup the preconditioner More...

void	enable_setup_preconditioner_before_solve ()
	Setup the preconditioner before the solve. More...

void	disable_setup_preconditioner_before_solve ()
	Don't set up the preconditioner before the solve. More...

void	enable_error_after_max_iter ()
	Throw an error if we don't converge within max_iter. More...

void	disable_error_after_max_iter ()
	Don't throw an error if we don't converge within max_iter (default). More...

void	enable_iterative_solver_as_preconditioner ()

void	disable_iterative_solver_as_preconditioner ()

Public Member Functions inherited from oomph::LinearSolver
	LinearSolver ()
	Empty constructor, initialise the member data. More...

	LinearSolver (const LinearSolver &dummy)=delete
	Broken copy constructor. More...

void	operator= (const LinearSolver &)=delete
	Broken assignment operator. More...

virtual	~LinearSolver ()
	Empty virtual destructor. More...

void	enable_doc_time ()
	Enable documentation of solve times. More...

void	disable_doc_time ()
	Disable documentation of solve times. More...

bool	is_doc_time_enabled () const
	Is documentation of solve times enabled? More...

bool	is_resolve_enabled () const
	Boolean flag indicating if resolves are enabled. More...

virtual void	enable_resolve ()

virtual void	solve (DoubleMatrixBase *const &matrix_pt, const Vector< double > &rhs, Vector< double > &result)

virtual void	solve_transpose (Problem *const &problem_pt, DoubleVector &result)

virtual void	solve_transpose (DoubleMatrixBase *const &matrix_pt, const DoubleVector &rhs, DoubleVector &result)

virtual void	solve_transpose (DoubleMatrixBase *const &matrix_pt, const Vector< double > &rhs, Vector< double > &result)

virtual void	resolve_transpose (const DoubleVector &rhs, DoubleVector &result)

virtual void	enable_computation_of_gradient ()

void	disable_computation_of_gradient ()

void	reset_gradient ()

void	get_gradient (DoubleVector &gradient)
	function to access the gradient, provided it has been computed More...

Public Member Functions inherited from oomph::DistributableLinearAlgebraObject
	DistributableLinearAlgebraObject ()
	Default constructor - create a distribution. More...

	DistributableLinearAlgebraObject (const DistributableLinearAlgebraObject &matrix)=delete
	Broken copy constructor. More...

void	operator= (const DistributableLinearAlgebraObject &)=delete
	Broken assignment operator. More...

virtual	~DistributableLinearAlgebraObject ()
	Destructor. More...

LinearAlgebraDistribution *	distribution_pt () const
	access to the LinearAlgebraDistribution More...

unsigned	nrow () const
	access function to the number of global rows. More...

unsigned	nrow_local () const
	access function for the num of local rows on this processor. More...

unsigned	nrow_local (const unsigned &p) const
	access function for the num of local rows on this processor. More...

unsigned	first_row () const
	access function for the first row on this processor More...

unsigned	first_row (const unsigned &p) const
	access function for the first row on this processor More...

bool	distributed () const
	distribution is serial or distributed More...

bool	distribution_built () const

void	build_distribution (const LinearAlgebraDistribution *const dist_pt)

void	build_distribution (const LinearAlgebraDistribution &dist)

Protected Member Functions
void	solve_using_AztecOO (Epetra_Vector &rhs_pt, Epetra_Vector &soln_pt)

void	solver_setup (DoubleMatrixBase *const &matrix_pt)

Protected Member Functions inherited from oomph::DistributableLinearAlgebraObject
void	clear_distribution ()

Protected Attributes
bool	Use_aztecoo_workaround_for_epetra_matrix_setup

unsigned	Iterations
	Stores number of iterations used. More...

AztecOO *	AztecOO_solver_pt
	Pointer to the AztecOO solver. More...

double	Jacobian_setup_time
	Stores set up time for Jacobian. More...

double	Linear_solver_solution_time
	Stores time for the solution (excludes time to set up preconditioner) More...

bool	Delete_matrix

bool	Assemble_serial_jacobian

unsigned	Solver_type

bool	Using_problem_based_solve

Epetra_CrsMatrix *	Epetra_matrix_pt
	A pointer for the linear system matrix in Epetra_CrsMatrix format. More...

Epetra_Operator *	Epetra_preconditioner_pt

DoubleMatrixBase *	Oomph_matrix_pt
	Oomph lib matrix pointer. More...

Problem *	Problem_pt

bool	If_oomphlib_preconditioner_use_epetra_values

Protected Attributes inherited from oomph::IterativeLinearSolver
bool	Doc_convergence_history

std::ofstream	Output_file_stream
	Output file stream for convergence history. More...

double	Tolerance
	Convergence tolerance. More...

unsigned	Max_iter
	Maximum number of iterations. More...

Preconditioner *	Preconditioner_pt
	Pointer to the preconditioner. More...

double	Jacobian_setup_time
	Jacobian setup time. More...

double	Solution_time
	linear solver solution time More...

double	Preconditioner_setup_time
	Preconditioner setup time. More...

bool	Setup_preconditioner_before_solve

bool	Throw_error_after_max_iter

bool	Use_iterative_solver_as_preconditioner
	Use the iterative solver as preconditioner. More...

bool	First_time_solve_when_used_as_preconditioner

Protected Attributes inherited from oomph::LinearSolver
bool	Enable_resolve

bool	Doc_time
	Boolean flag that indicates whether the time taken. More...

bool	Compute_gradient

bool	Gradient_has_been_computed
	flag that indicates whether the gradient was computed or not More...

DoubleVector	Gradient_for_glob_conv_newton_solve

Additional Inherited Members
Static Protected Attributes inherited from oomph::IterativeLinearSolver
static IdentityPreconditioner	Default_preconditioner

Detailed Description

An interface to the Trilinos AztecOO classes allowing it to be used as an Oomph-lib LinearSolver. The AztecOO solver is a Krylov Subspace solver; the solver type (either CG, GMRES or BiCGStab) can be set using solver_type(). This solver can be preconditioned with Trilinos Preconditioners (derived from TrilinosPreconditionerBase) or Oomph-lib preconditioners (derived from Preconditioner). Preconditioners are set using preconditioner_pt().

Member Enumeration Documentation

◆ AztecOO_solver_types

enum oomph::TrilinosAztecOOSolver::AztecOO_solver_types

Enumerated list to define which AztecOO solver is used.

if this solver is using an oomph-lib preconditioner then the vectors passed to preconditioner_solve(...) should be using the values in the epetra vector as external data. If the vectors are using external data then rebuild(...) methods cannot be used be used in the preconditioner.

Enumerator
CG
GMRES
BiCGStab

     {
       CG,
       GMRES,
       BiCGStab
     };

Constructor & Destructor Documentation

◆ TrilinosAztecOOSolver() [1/2]

oomph::TrilinosAztecOOSolver::TrilinosAztecOOSolver ( )

inline

Constructor.

     {
       // By default use workaround for creating of epetra matrix that
       // respects aztecoo's ordering requirements
       Use_aztecoo_workaround_for_epetra_matrix_setup = true;
  
       // set pointer to Null
       AztecOO_solver_pt = 0;
  
       // initially assume not problem based solve
       Using_problem_based_solve = false;
  
       // if a problem based solve is performed then it should generate a
       // serial matrix
       Assemble_serial_jacobian = false;
  
       // by default we do not use external values in the oomph-lib
       // preconditioner
       If_oomphlib_preconditioner_use_epetra_values = false;
  
       // null the pts
       Problem_pt = 0;
       Epetra_matrix_pt = 0;
       Epetra_preconditioner_pt = 0;
  
       // set solver defaults
       Solver_type = GMRES;
       Tolerance = 10e-10;
  
       // don't delete the matrix
       Delete_matrix = false;
     }

References Assemble_serial_jacobian, AztecOO_solver_pt, Delete_matrix, e(), Epetra_matrix_pt, Epetra_preconditioner_pt, GMRES, If_oomphlib_preconditioner_use_epetra_values, Problem_pt, Solver_type, oomph::IterativeLinearSolver::Tolerance, Use_aztecoo_workaround_for_epetra_matrix_setup, and Using_problem_based_solve.

◆ ~TrilinosAztecOOSolver()

oomph::TrilinosAztecOOSolver::~TrilinosAztecOOSolver ( )

inline

Destructor - delete the solver and the matrices.

     {
       // delete
       clean_up_memory();
  
       // if Problem_based solve then the oomph matrix was generated by this
       // class and should be deleted
       if (Using_problem_based_solve)
       {
         delete Oomph_matrix_pt;
         Oomph_matrix_pt = 0;
       }
     }

References clean_up_memory(), Oomph_matrix_pt, and Using_problem_based_solve.

◆ TrilinosAztecOOSolver() [2/2]

oomph::TrilinosAztecOOSolver::TrilinosAztecOOSolver ( const TrilinosAztecOOSolver & )

delete

Broken copy constructor.

Member Function Documentation

◆ clean_up_memory()

void oomph::TrilinosAztecOOSolver::clean_up_memory ( )

inlinevirtual

Clean up method - deletes the solver, the matrices and the preconditioner

Reimplemented from oomph::LinearSolver.

     {
       // delete the solver
       delete AztecOO_solver_pt;
       AztecOO_solver_pt = 0;
  
       // delete the Epetra_Operator preconditioner (only if it is a wrapper to
       // an oomphlib preconditioner in which case only the wrapper is deleted
       // and not the actual preconditioner). if the preconditioner is Trilinos
       // preconditioner then the Epetra_Operator is deleted when that
       // preconditioner is deleted
       if (Epetra_preconditioner_pt != 0)
       {
         if (dynamic_cast<OomphLibPreconditionerEpetraOperator*>(
               Epetra_preconditioner_pt) != 0)
         {
           delete Epetra_preconditioner_pt;
           Epetra_preconditioner_pt = 0;
         }
       }
  
       // don't delete the preconditioner but call its clean up memory method
       if (this->preconditioner_pt() != 0)
       {
         this->preconditioner_pt()->clean_up_memory();
       }
  
  
       // delete the matrices
       // This must now happen after the preconditioner delete because the
       // ML preconditioner (and maybe others) use the communicator from the
       // matrix in the destructor
       delete Epetra_matrix_pt;
       Epetra_matrix_pt = 0;
     }

References AztecOO_solver_pt, oomph::Preconditioner::clean_up_memory(), Epetra_matrix_pt, Epetra_preconditioner_pt, and oomph::IterativeLinearSolver::preconditioner_pt().

Referenced by disable_resolve(), solve(), solver_setup(), and ~TrilinosAztecOOSolver().

◆ disable_assemble_serial_jacobian()

void oomph::TrilinosAztecOOSolver::disable_assemble_serial_jacobian ( )

inline

Unset the assembly of the serial jacobian.

     {
       Assemble_serial_jacobian = false;
     }

References Assemble_serial_jacobian.

◆ disable_aztecoo_workaround_for_epetra_matrix_setup()

void oomph::TrilinosAztecOOSolver::disable_aztecoo_workaround_for_epetra_matrix_setup ( )

inline

Disable workaround for creating of epetra matrix that respects aztecoo's ordering requirements

     {
       Use_aztecoo_workaround_for_epetra_matrix_setup = false;
     }

References Use_aztecoo_workaround_for_epetra_matrix_setup.

◆ disable_delete_matrix()

void oomph::TrilinosAztecOOSolver::disable_delete_matrix ( )

inline

Call if the matrix can not be deleted (default)

     {
       Delete_matrix = false;
     }

References Delete_matrix.

◆ disable_resolve()

void oomph::TrilinosAztecOOSolver::disable_resolve ( )

inlinevirtual

Disable resolve function (overloads the LinearSolver disable_resolve function).

Reimplemented from oomph::LinearSolver.

     {
       Enable_resolve = false;
       clean_up_memory();
     }

References clean_up_memory(), and oomph::LinearSolver::Enable_resolve.

◆ enable_assemble_serial_jacobian()

void oomph::TrilinosAztecOOSolver::enable_assemble_serial_jacobian ( )

inline

Set the assembly of the serial jacobian when performing a problem-based solve

     {
       Assemble_serial_jacobian = true;
     }

References Assemble_serial_jacobian.

◆ enable_aztecoo_workaround_for_epetra_matrix_setup()

void oomph::TrilinosAztecOOSolver::enable_aztecoo_workaround_for_epetra_matrix_setup ( )

inline

Enable workaround for creating of epetra matrix that respects aztecoo's ordering requirements

     {
       Use_aztecoo_workaround_for_epetra_matrix_setup = true;
     }

References Use_aztecoo_workaround_for_epetra_matrix_setup.

◆ enable_delete_matrix()

void oomph::TrilinosAztecOOSolver::enable_delete_matrix ( )

inline

Call if the matrix can be deleted.

     {
       Delete_matrix = true;
     }

References Delete_matrix.

◆ is_aztecoo_workaround_for_epetra_matrix_setup_enabled()

bool oomph::TrilinosAztecOOSolver::is_aztecoo_workaround_for_epetra_matrix_setup_enabled ( )

inline

Is workaround for creating of epetra matrix that respects aztecoo's ordering requirements enabled?

     {
       return Use_aztecoo_workaround_for_epetra_matrix_setup;
     }

References Use_aztecoo_workaround_for_epetra_matrix_setup.

◆ iterations()

unsigned oomph::TrilinosAztecOOSolver::iterations ( ) const

inlinevirtual

Acess function to Iterations.

Implements oomph::IterativeLinearSolver.

     {
       return Iterations;
     }

References Iterations.

◆ jacobian_setup_time()

double oomph::TrilinosAztecOOSolver::jacobian_setup_time ( )

inline

Function to return Jacobian_setup_time;.

     {
       return Jacobian_setup_time;
     }

References Jacobian_setup_time.

◆ linear_solver_solution_time()

double oomph::TrilinosAztecOOSolver::linear_solver_solution_time ( )

inline

Function to return Linear_solver_solution_time.

     {
       return Linear_solver_solution_time;
     }

References Linear_solver_solution_time.

◆ max_iter()

unsigned& oomph::TrilinosAztecOOSolver::max_iter ( )

inline

Access function to Max_iter.

     {
       return Max_iter;
     }

References oomph::IterativeLinearSolver::Max_iter.

Referenced by PseudoElasticCollapsibleChannelProblem< FLUID_ELEMENT, SOLID_ELEMENT >::set_pseudo_elastic_fsi_solver().

◆ operator=()

void oomph::TrilinosAztecOOSolver::operator= ( const TrilinosAztecOOSolver & )

delete

Broken assignment operator.

◆ resolve()

void oomph::TrilinosAztecOOSolver::resolve	(	const DoubleVector &	rhs,
		DoubleVector &	solution
	)

virtual

Function to resolve a linear system using the existing solver data, allowing a solve with a new right hand side vector. This function must be used after a call to solve(...) with enable_resolve set to true.

Reimplemented from oomph::LinearSolver.

   {
     // start the timer
     double start_t = TimingHelpers::timer();
  
 #ifdef PARANOID
     if (Epetra_matrix_pt->NumGlobalRows() != static_cast<int>(rhs.nrow()))
     {
       std::ostringstream error_message;
       error_message
         << "The rhs vector and the matrix must have the same number "
         << "of rows.\n"
         << "The rhs vector has " << rhs.nrow() << " rows.\n"
         << "The matrix has " << Epetra_matrix_pt->NumGlobalRows() << " rows.\n";
       throw OomphLibError(
         error_message.str(), OOMPH_CURRENT_FUNCTION, OOMPH_EXCEPTION_LOCATION);
     }
  
     // if the result vector is setup then check it is not distributed and has
     // the same communicator as the rhs vector
     if (solution.built())
     {
       if (!(*solution.distribution_pt() == *rhs.distribution_pt()))
       {
         std::ostringstream error_message_stream;
         error_message_stream
           << "The result vector distribution has been setup; it must have the "
           << "same distribution as the rhs vector.";
         throw OomphLibError(error_message_stream.str(),
                             OOMPH_CURRENT_FUNCTION,
                             OOMPH_EXCEPTION_LOCATION);
       }
     }
 #endif
  
     // build the result if not built
     if (!solution.built())
     {
       solution.build(rhs.distribution_pt(), 0.0);
     }
  
  
     // create Epetra version of r
     Epetra_Vector* epetra_r_pt =
       TrilinosEpetraHelpers::create_distributed_epetra_vector(rhs);
  
     // create an empty Epetra vector for z
     Epetra_Vector* epetra_z_pt =
       TrilinosEpetraHelpers::create_distributed_epetra_vector(solution);
  
     // solve the system
     solve_using_AztecOO(epetra_r_pt, epetra_z_pt);
  
     // Copy result to z
     if (!solution.distribution_built())
     {
       solution.build(rhs.distribution_pt(), 0.0);
     }
     TrilinosEpetraHelpers::copy_to_oomphlib_vector(epetra_z_pt, solution);
  
     // clean up memory
     delete epetra_r_pt;
     delete epetra_z_pt;
  
     double end_t = TimingHelpers::timer();
     Linear_solver_solution_time = end_t - start_t;
  
     // output timings and info
     if (this->Doc_time)
     {
       oomph_info << "Time for resolve                        : "
                  << Linear_solver_solution_time << "s" << std::endl;
     }
   }

References oomph::TrilinosEpetraHelpers::copy_to_oomphlib_vector(), oomph::TrilinosEpetraHelpers::create_distributed_epetra_vector(), oomph::DistributableLinearAlgebraObject::distribution_pt(), oomph::LinearSolver::Doc_time, Epetra_matrix_pt, Linear_solver_solution_time, oomph::DistributableLinearAlgebraObject::nrow(), OOMPH_CURRENT_FUNCTION, OOMPH_EXCEPTION_LOCATION, oomph::oomph_info, BiharmonicTestFunctions1::solution(), solve_using_AztecOO(), and oomph::TimingHelpers::timer().

◆ solve() [1/2]

void oomph::TrilinosAztecOOSolver::solve	(	DoubleMatrixBase *const &	matrix_pt,
		const DoubleVector &	rhs,
		DoubleVector &	result
	)

virtual

Function to solve the linear system defined by matrix_pt and rhs. NOTE 1. The matrix has to be of type CRDoubleMatrix or DistributedCRDoubleMatrix. NOTE 2. This function will delete any existing internal data and generate a new AztecOO solver.

Reimplemented from oomph::LinearSolver.

   {
     // start the timer
     double start_t = TimingHelpers::timer();
  
 #ifdef PARANOID
     // check that the matrix is square
     if (matrix_pt->nrow() != matrix_pt->ncol())
     {
       std::ostringstream error_message_stream;
       error_message_stream << "The matrix at matrix_pt must be square.";
       throw OomphLibError(error_message_stream.str(),
                           OOMPH_CURRENT_FUNCTION,
                           OOMPH_EXCEPTION_LOCATION);
     }
     // check that the matrix and the rhs vector have the same nrow()
     if (matrix_pt->nrow() != rhs.nrow())
     {
       std::ostringstream error_message_stream;
       error_message_stream
         << "The matrix and the rhs vector must have the same number of rows.";
       throw OomphLibError(error_message_stream.str(),
                           OOMPH_CURRENT_FUNCTION,
                           OOMPH_EXCEPTION_LOCATION);
     }
  
     // if the matrix is distributable then it too should have the same
     // communicator as the rhs vector and should not be distributed
     CRDoubleMatrix* cr_matrix_pt = dynamic_cast<CRDoubleMatrix*>(matrix_pt);
     if (cr_matrix_pt != 0)
     {
       OomphCommunicator temp_comm(*rhs.distribution_pt()->communicator_pt());
       if (!(temp_comm == *cr_matrix_pt->distribution_pt()->communicator_pt()))
       {
         std::ostringstream error_message_stream;
         error_message_stream
           << "The matrix matrix_pt must have the same communicator as the "
              "vectors"
           << " rhs and result must have the same communicator";
         throw OomphLibError(error_message_stream.str(),
                             OOMPH_CURRENT_FUNCTION,
                             OOMPH_EXCEPTION_LOCATION);
       }
     }
     else
     {
       throw OomphLibError("Matrix must be of type CRDoubleMatrix",
                           OOMPH_CURRENT_FUNCTION,
                           OOMPH_EXCEPTION_LOCATION);
     }
  
     // if the result vector is setup then check it is not distributed and has
     // the same communicator as the rhs vector
     if (result.built())
     {
       if (!(*result.distribution_pt() == *rhs.distribution_pt()))
       {
         std::ostringstream error_message_stream;
         error_message_stream
           << "The result vector distribution has been setup; it must have the "
           << "same distribution as the rhs vector.";
         throw OomphLibError(error_message_stream.str(),
                             OOMPH_CURRENT_FUNCTION,
                             OOMPH_EXCEPTION_LOCATION);
       }
     }
 #endif
  
     // build the result if not built
     if (!result.built())
     {
       result.build(rhs.distribution_pt(), 0.0);
     }
  
     if (Use_iterative_solver_as_preconditioner)
     {
       // Only call the setup method if this is the first time we call the
       // solve method
       if (First_time_solve_when_used_as_preconditioner)
       {
         // setup the solver
         solver_setup(matrix_pt);
         // Do not call the setup again
         First_time_solve_when_used_as_preconditioner = false;
         // Enable resolve since we are not going to build the solver, the
         // matrix and the wrapper to the preconditioner again
         Enable_resolve = true;
       }
     }
     else
     {
       // setup the solver
       solver_setup(matrix_pt);
     }
  
     // create Epetra version of r
     Epetra_Vector* epetra_r_pt =
       TrilinosEpetraHelpers::create_distributed_epetra_vector(rhs);
  
     // create an empty Epetra vector for z
     Epetra_Vector* epetra_z_pt =
       TrilinosEpetraHelpers::create_distributed_epetra_vector(result);
  
     double start_t_trilinos = TimingHelpers::timer();
  
     // solve the system
     solve_using_AztecOO(epetra_r_pt, epetra_z_pt);
  
     double end_t_trilinos = TimingHelpers::timer();
     if (this->Doc_time)
     {
       oomph_info << "Time for trilinos solve itself                 : "
                  << end_t_trilinos - start_t_trilinos << "s" << std::endl;
     }
     // Copy result to z
     TrilinosEpetraHelpers::copy_to_oomphlib_vector(epetra_z_pt, result);
  
     // clean up memory
     delete epetra_r_pt;
     delete epetra_z_pt;
  
     // delete solver data if required
     if (!Enable_resolve)
     {
       clean_up_memory();
     }
  
     // stop timers and compute solve time
     double end_t = TimingHelpers::timer();
     Linear_solver_solution_time = end_t - start_t;
  
     // output timings and info
     if (this->Doc_time)
     {
       oomph_info << "Time for complete trilinos solve                  : "
                  << Linear_solver_solution_time << "s" << std::endl;
     }
   }

References oomph::DoubleVector::build(), oomph::DoubleVector::built(), clean_up_memory(), oomph::LinearAlgebraDistribution::communicator_pt(), oomph::TrilinosEpetraHelpers::copy_to_oomphlib_vector(), oomph::TrilinosEpetraHelpers::create_distributed_epetra_vector(), oomph::DistributableLinearAlgebraObject::distribution_pt(), oomph::LinearSolver::Doc_time, oomph::LinearSolver::Enable_resolve, oomph::IterativeLinearSolver::First_time_solve_when_used_as_preconditioner, Linear_solver_solution_time, oomph::DoubleMatrixBase::ncol(), oomph::DistributableLinearAlgebraObject::nrow(), oomph::DoubleMatrixBase::nrow(), OOMPH_CURRENT_FUNCTION, OOMPH_EXCEPTION_LOCATION, oomph::oomph_info, solve_using_AztecOO(), solver_setup(), oomph::TimingHelpers::timer(), and oomph::IterativeLinearSolver::Use_iterative_solver_as_preconditioner.

◆ solve() [2/2]

void oomph::TrilinosAztecOOSolver::solve	(	Problem *const &	problem_pt,
		DoubleVector &	solution
	)

virtual

Function which uses problem_pt's get_jacobian(...) function to generate a linear system which is then solved. This function deletes any existing internal data and then generates a new AztecOO solver.

//////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// Function which uses problem_pt's get_jacobian(...) function to generate a linear system which is then solved. This function deletes any existing internal data and then generates a new AztecOO solver.

Implements oomph::LinearSolver.

   {
     //   MemoryUsage::doc_memory_usage("start of TrilinosAztecOOSolver::solve");
     //   MemoryUsage::insert_comment_to_continous_top(
     //    "start of TrilinosAztecOOSolver::solve");
  
     // clean up from previous solve
     clean_up_memory();
  
     // if we were previously using a problem based solve then we should delete
     // the oomphlib matrix as it was created during the last solve
     if (Using_problem_based_solve)
     {
       delete Oomph_matrix_pt;
       Oomph_matrix_pt = NULL;
     }
  
     // this is a problem based solve
     Using_problem_based_solve = true;
  
     // store the problem_pt
     Problem_pt = problem_pt;
  
     // Get oomph-lib Jacobian matrix and residual vector
  
     // record the start time
     double start_t = TimingHelpers::timer();
  
     //   MemoryUsage::doc_memory_usage("start of get_jacobian()");
     //   MemoryUsage::insert_comment_to_continous_top("start of
     //   get_jacobian()");
  
     // create the residual
     DoubleVector residual;
  
     // create the jacobian
     CRDoubleMatrix* cr_matrix_pt = new CRDoubleMatrix;
     Oomph_matrix_pt = cr_matrix_pt;
     problem_pt->get_jacobian(residual, *cr_matrix_pt);
     this->build_distribution(residual.distribution_pt());
  
     // record the end time and compute the matrix setup time
     double end_t = TimingHelpers::timer();
     Jacobian_setup_time = end_t - start_t;
     if (this->Doc_time)
     {
       oomph_info << "Time to generate Jacobian [sec]    : "
                  << Jacobian_setup_time << std::endl;
     }
  
  
     //   MemoryUsage::doc_memory_usage("after get_jacobian() in trilinos
     //   solver"); MemoryUsage::insert_comment_to_continous_top(
     //    "after get_jacobian() in trilinos solver");
  
     // store the distribution of the solution vector
     if (!solution.built())
     {
       solution.build(this->distribution_pt(), 0.0);
     }
     LinearAlgebraDistribution solution_dist(solution.distribution_pt());
  
     // redistribute the distribution
     solution.redistribute(this->distribution_pt());
  
     //   MemoryUsage::doc_memory_usage("before trilinos solve");
     //   MemoryUsage::insert_comment_to_continous_top("before trilinos solve ");
  
     // continue solving using matrix based solve function
     solve(Oomph_matrix_pt, residual, solution);
  
  
     //   MemoryUsage::doc_memory_usage("after trilinos solve");
     //   MemoryUsage::insert_comment_to_continous_top("after trilinos solve ");
  
     // return to the original distribution
     solution.redistribute(&solution_dist);
  
     //   MemoryUsage::doc_memory_usage("end of TrilinosAztecOOSolver::solve");
     //   MemoryUsage::insert_comment_to_continous_top(
     //    "end of TrilinosAztecOOSolver::solve");
   }

References oomph::DistributableLinearAlgebraObject::build_distribution(), clean_up_memory(), oomph::DistributableLinearAlgebraObject::distribution_pt(), oomph::LinearSolver::Doc_time, oomph::Problem::get_jacobian(), Jacobian_setup_time, oomph::oomph_info, Oomph_matrix_pt, Problem_pt, BiharmonicTestFunctions1::solution(), oomph::TimingHelpers::timer(), and Using_problem_based_solve.

◆ solve_using_AztecOO()

void oomph::TrilinosAztecOOSolver::solve_using_AztecOO	(	Epetra_Vector *&	rhs_pt,
		Epetra_Vector *&	soln_pt
	)

protected

Helper function performs the actual solve once the AztecOO solver is set up

Helper function performs the actual solve once the AztecOO solver is set up (i.e. solver_setup() is called)

   {
 #ifdef PARANOID
     // check the matrix and rhs are of consistent sizes
     if (AztecOO_solver_pt == 0)
     {
       std::ostringstream error_message;
       error_message << "Solver must be called with solve(...) "
                     << "before resolve(...) to set it up.\n";
       throw OomphLibError(
         error_message.str(), OOMPH_CURRENT_FUNCTION, OOMPH_EXCEPTION_LOCATION);
     }
 #endif
  
     // set the vectors
     AztecOO_solver_pt->SetLHS(soln_pt);
     AztecOO_solver_pt->SetRHS(rhs_pt);
  
     // perform solve
     AztecOO_solver_pt->Iterate(Max_iter, Tolerance);
  
  
     // output iterations and final norm
     Iterations = AztecOO_solver_pt->NumIters();
     if (Doc_time)
     {
       double norm = AztecOO_solver_pt->TrueResidual();
       oomph_info << "Linear solver iterations    : " << Iterations << std::endl;
       oomph_info << "Final Relative Residual Norm: " << norm << std::endl;
     }
   }

References AztecOO_solver_pt, oomph::LinearSolver::Doc_time, Iterations, oomph::IterativeLinearSolver::Max_iter, OOMPH_CURRENT_FUNCTION, OOMPH_EXCEPTION_LOCATION, oomph::oomph_info, and oomph::IterativeLinearSolver::Tolerance.

Referenced by resolve(), and solve().

◆ solver_setup()

void oomph::TrilinosAztecOOSolver::solver_setup ( DoubleMatrixBase *const & matrix_pt )

protected

Helper function for setting up the solver. Converts the oomph-lib matrices to Epetra matrices, sets up the preconditioner, creates the Trilinos Aztec00 solver and passes in the matrix, preconditioner and parameters.

   {
     // clean up the memory
     //  - delete all except Oomph_matrix_pt, which may have been set in the
     //    problem based solve
     clean_up_memory();
  
     // cast to CRDoubleMatrix
     // note cast check performed in matrix based solve(...) method
     CRDoubleMatrix* cast_matrix_pt = dynamic_cast<CRDoubleMatrix*>(matrix_pt);
  
     // store the distribution
     // distribution of preconditioner is same as matrix
     this->build_distribution(cast_matrix_pt->distribution_pt());
  
     // create the new solver
     AztecOO_solver_pt = new AztecOO();
  
     // if the preconditioner is an oomph-lib preconditioner then we set it up
     TrilinosPreconditionerBase* trilinos_prec_pt =
       dynamic_cast<TrilinosPreconditionerBase*>(Preconditioner_pt);
     if (trilinos_prec_pt == 0)
     {
       if (Setup_preconditioner_before_solve)
       {
         // setup the preconditioner
         // start of prec setup
         double prec_setup_start_t = TimingHelpers::timer();
         Preconditioner_pt->setup(matrix_pt);
         // start of prec setup
         double prec_setup_finish_t = TimingHelpers::timer();
         if (Doc_time)
         {
           double t_prec_setup = prec_setup_finish_t - prec_setup_start_t;
           oomph_info << "Time for preconditioner setup [sec]: " << t_prec_setup
                      << std::endl;
         }
 #ifdef PARANOID
         if (*Preconditioner_pt->distribution_pt() != *this->distribution_pt())
         {
           std::ostringstream error_message;
           error_message << "The oomph-lib preconditioner and the solver must "
                         << "have the same distribution";
           throw OomphLibError(error_message.str(),
                               OOMPH_CURRENT_FUNCTION,
                               OOMPH_EXCEPTION_LOCATION);
         }
 #endif
       }
  
       // wrap the oomphlib preconditioner in the Epetra_Operator derived
       // OoomphLibPreconditionerEpetraOperator to allow it to be passed to the
       // trilinos preconditioner
       Epetra_preconditioner_pt =
         new OomphLibPreconditionerEpetraOperator(Preconditioner_pt);
     }
  
     // create the matrix
     double start_t_matrix = TimingHelpers::timer();
     if (Use_aztecoo_workaround_for_epetra_matrix_setup)
     {
       Epetra_matrix_pt =
         TrilinosEpetraHelpers ::create_distributed_epetra_matrix_for_aztecoo(
           cast_matrix_pt);
     }
     else
     {
       Epetra_matrix_pt =
         TrilinosEpetraHelpers ::create_distributed_epetra_matrix(
           cast_matrix_pt, cast_matrix_pt->distribution_pt());
     }
  
     // record the end time and compute the matrix setup time
     double end_t_matrix = TimingHelpers::timer();
     if (trilinos_prec_pt == 0)
     {
       if (Using_problem_based_solve)
       {
         dynamic_cast<CRDoubleMatrix*>(Oomph_matrix_pt)->clear();
         delete Oomph_matrix_pt;
         Oomph_matrix_pt = NULL;
       }
  
       // delete Oomph-lib matrix if requested
       else if (Delete_matrix)
       {
         dynamic_cast<CRDoubleMatrix*>(matrix_pt)->clear();
       }
     }
  
     // output times
     if (Doc_time)
     {
       oomph_info << "Time to generate Trilinos matrix      : "
                  << double(end_t_matrix - start_t_matrix) << "s" << std::endl;
     }
  
     // set the matrix
     AztecOO_solver_pt->SetUserMatrix(Epetra_matrix_pt);
  
     // set the preconditioner
     if (trilinos_prec_pt == 0)
     {
       AztecOO_solver_pt->SetPrecOperator(Epetra_preconditioner_pt);
     }
  
 #ifdef PARANOID
     // paranoid check the preconditioner exists
     if (Preconditioner_pt == 0)
     {
       std::ostringstream error_message;
       error_message << "Preconditioner_pt == 0. (Remember default "
                     << "preconditioner is IdentityPreconditioner)";
       throw OomphLibError(
         error_message.str(), OOMPH_CURRENT_FUNCTION, OOMPH_EXCEPTION_LOCATION);
     }
 #endif
  
     // if the preconditioner is a trilinos preconditioner then setup the
     // preconditioner
     if (trilinos_prec_pt != 0)
     {
       // only setup the preconditioner if required
       if (Setup_preconditioner_before_solve)
       {
         // start of prec setup
         double prec_setup_start_t = TimingHelpers::timer();
  
         // setup the preconditioner
         trilinos_prec_pt->set_matrix_pt(matrix_pt);
         trilinos_prec_pt->setup(Epetra_matrix_pt);
  
  
         // set the preconditioner
         AztecOO_solver_pt->SetPrecOperator(
           trilinos_prec_pt->epetra_operator_pt());
  
         // start of prec setup
         double prec_setup_finish_t = TimingHelpers::timer();
         if (Doc_time)
         {
           double t_prec_setup = prec_setup_finish_t - prec_setup_start_t;
           oomph_info << "Time for preconditioner setup [sec]: " << t_prec_setup
                      << std::endl;
         }
       }
  
       // delete the oomph-matrix if required
       if (Using_problem_based_solve)
       {
         dynamic_cast<CRDoubleMatrix*>(Oomph_matrix_pt)->clear();
         delete Oomph_matrix_pt;
         Oomph_matrix_pt = NULL;
       }
  
       // delete Oomph-lib matrix if requested
       else if (Delete_matrix)
       {
         dynamic_cast<CRDoubleMatrix*>(matrix_pt)->clear();
       }
     }
  
     // set solver options
     if (Doc_time)
     {
       AztecOO_solver_pt->SetAztecOption(AZ_output, AZ_warnings);
     }
     else
     {
       AztecOO_solver_pt->SetAztecOption(AZ_output, AZ_none);
     }
     AztecOO_solver_pt->SetAztecOption(AZ_kspace, Max_iter);
  
     // set solver type
     switch (Solver_type)
     {
       case CG:
         AztecOO_solver_pt->SetAztecOption(AZ_solver, AZ_cg);
         break;
  
       case GMRES:
         AztecOO_solver_pt->SetAztecOption(AZ_solver, AZ_gmres);
         break;
  
       case BiCGStab:
         AztecOO_solver_pt->SetAztecOption(AZ_solver, AZ_bicgstab);
         break;
  
       default:
         std::ostringstream error_message;
         error_message << "Solver_type set to incorrect value. "
                       << "Acceptable values are " << CG << ", " << GMRES
                       << " and " << BiCGStab << ". Current value is "
                       << Solver_type << ".";
         throw OomphLibError(error_message.str(),
                             OOMPH_CURRENT_FUNCTION,
                             OOMPH_EXCEPTION_LOCATION);
     }
   }

References AztecOO_solver_pt, BiCGStab, oomph::DistributableLinearAlgebraObject::build_distribution(), CG, clean_up_memory(), oomph::TrilinosEpetraHelpers::create_distributed_epetra_matrix(), oomph::TrilinosEpetraHelpers::create_distributed_epetra_matrix_for_aztecoo(), Delete_matrix, oomph::DistributableLinearAlgebraObject::distribution_pt(), oomph::LinearSolver::Doc_time, Epetra_matrix_pt, oomph::TrilinosPreconditionerBase::epetra_operator_pt(), Epetra_preconditioner_pt, GMRES, oomph::IterativeLinearSolver::Max_iter, OOMPH_CURRENT_FUNCTION, OOMPH_EXCEPTION_LOCATION, oomph::oomph_info, Oomph_matrix_pt, oomph::IterativeLinearSolver::Preconditioner_pt, oomph::Preconditioner::set_matrix_pt(), oomph::TrilinosPreconditionerBase::setup(), oomph::Preconditioner::setup(), oomph::IterativeLinearSolver::Setup_preconditioner_before_solve, Solver_type, oomph::TimingHelpers::timer(), Use_aztecoo_workaround_for_epetra_matrix_setup, and Using_problem_based_solve.

Referenced by solve().

◆ solver_type()

unsigned& oomph::TrilinosAztecOOSolver::solver_type ( )

inline

Access function to Solver_type.

     {
       return Solver_type;
     }

References Solver_type.

Referenced by main(), parallel_test(), PseudoElasticCollapsibleChannelProblem< FLUID_ELEMENT, SOLID_ELEMENT >::set_pseudo_elastic_fsi_solver(), set_pseudo_elastic_fsi_solver(), and TiltedCavityProblem< ELEMENT >::TiltedCavityProblem().

◆ tolerance()

double& oomph::TrilinosAztecOOSolver::tolerance ( )

inline

Access function to Tolerance.

     {
       return Tolerance;
     }

References oomph::IterativeLinearSolver::Tolerance.

Referenced by main(), PseudoElasticCollapsibleChannelProblem< FLUID_ELEMENT, SOLID_ELEMENT >::set_pseudo_elastic_fsi_solver(), and set_pseudo_elastic_fsi_solver().

Member Data Documentation

◆ Assemble_serial_jacobian

bool oomph::TrilinosAztecOOSolver::Assemble_serial_jacobian

protected

If true, when performing a problem based solve a serial matrix will be requested from Problem::get_jacobian(...). Defaults to true

Referenced by disable_assemble_serial_jacobian(), enable_assemble_serial_jacobian(), and TrilinosAztecOOSolver().

◆ AztecOO_solver_pt

AztecOO* oomph::TrilinosAztecOOSolver::AztecOO_solver_pt

protected

Pointer to the AztecOO solver.

Referenced by clean_up_memory(), solve_using_AztecOO(), solver_setup(), and TrilinosAztecOOSolver().

◆ Delete_matrix

bool oomph::TrilinosAztecOOSolver::Delete_matrix

protected

Trilinos copies matrix data from oomph-lib's own CRDoubleMatrix or DistributedCRDoubleMatrix to Trilinos's Epetra format - the Trilinos solver no longer requires the oomph-lib matrices and therefore they could be deleted to save memory. This must be requested explicitly by setting this flag to TRUE. NOTE: The matrix is deleted after the preconditioner is setup.

Referenced by disable_delete_matrix(), enable_delete_matrix(), solver_setup(), and TrilinosAztecOOSolver().

◆ Epetra_matrix_pt

Epetra_CrsMatrix* oomph::TrilinosAztecOOSolver::Epetra_matrix_pt

protected

A pointer for the linear system matrix in Epetra_CrsMatrix format.

Referenced by clean_up_memory(), resolve(), solver_setup(), and TrilinosAztecOOSolver().

◆ Epetra_preconditioner_pt

Epetra_Operator* oomph::TrilinosAztecOOSolver::Epetra_preconditioner_pt

protected

A pointer to the Epetra_Operator for the preconditioner. This is only used if the preconditioner NOT a Trilinos preconditioner.

Referenced by clean_up_memory(), solver_setup(), and TrilinosAztecOOSolver().

◆ If_oomphlib_preconditioner_use_epetra_values

bool oomph::TrilinosAztecOOSolver::If_oomphlib_preconditioner_use_epetra_values

protected

if this solver is using an oomph-lib preconditioner then the vectors passed to preconditioner_solve(...) should be using the values in the epetra vector as external data. If the vectors are using external data then rebuild(...) methods cannot be used.

Referenced by TrilinosAztecOOSolver().

◆ Iterations

unsigned oomph::TrilinosAztecOOSolver::Iterations

protected

Stores number of iterations used.

Referenced by iterations(), and solve_using_AztecOO().

◆ Jacobian_setup_time

double oomph::TrilinosAztecOOSolver::Jacobian_setup_time

protected

Stores set up time for Jacobian.

Referenced by jacobian_setup_time(), and solve().

◆ Linear_solver_solution_time

double oomph::TrilinosAztecOOSolver::Linear_solver_solution_time

protected

Stores time for the solution (excludes time to set up preconditioner)

Referenced by linear_solver_solution_time(), resolve(), and solve().

◆ Oomph_matrix_pt

DoubleMatrixBase* oomph::TrilinosAztecOOSolver::Oomph_matrix_pt

protected

Oomph lib matrix pointer.

Referenced by solve(), solver_setup(), and ~TrilinosAztecOOSolver().

◆ Problem_pt

Problem* oomph::TrilinosAztecOOSolver::Problem_pt

protected

A pointer to the underlying problem (NULL if MATRIX based solve) The problem_pt is stored here in a problem based solve for the preconditioner

Referenced by solve(), and TrilinosAztecOOSolver().

◆ Solver_type

unsigned oomph::TrilinosAztecOOSolver::Solver_type

protected

Defines which solver is set up - available types are defined in AztecOO_solver_types

Referenced by solver_setup(), solver_type(), and TrilinosAztecOOSolver().

◆ Use_aztecoo_workaround_for_epetra_matrix_setup

bool oomph::TrilinosAztecOOSolver::Use_aztecoo_workaround_for_epetra_matrix_setup

protected

Use workaround for creating of epetra matrix that respects aztecoo's ordering requirements

Referenced by disable_aztecoo_workaround_for_epetra_matrix_setup(), enable_aztecoo_workaround_for_epetra_matrix_setup(), is_aztecoo_workaround_for_epetra_matrix_setup_enabled(), solver_setup(), and TrilinosAztecOOSolver().

◆ Using_problem_based_solve

bool oomph::TrilinosAztecOOSolver::Using_problem_based_solve

protected

Helper flag keeping track of whether we called the linear algebra or problem-based solve function.

Referenced by solve(), solver_setup(), TrilinosAztecOOSolver(), and ~TrilinosAztecOOSolver().

The documentation for this class was generated from the following files:

Public Types

Public Member Functions

Protected Member Functions

Protected Attributes

Additional Inherited Members

Detailed Description

Member Enumeration Documentation

◆ AztecOO_solver_types

Constructor & Destructor Documentation

◆ TrilinosAztecOOSolver() [1/2]

◆ ~TrilinosAztecOOSolver()

◆ TrilinosAztecOOSolver() [2/2]

Member Function Documentation

◆ clean_up_memory()

◆ disable_assemble_serial_jacobian()

◆ disable_aztecoo_workaround_for_epetra_matrix_setup()

◆ disable_delete_matrix()

◆ disable_resolve()

◆ enable_assemble_serial_jacobian()

◆ enable_aztecoo_workaround_for_epetra_matrix_setup()

◆ enable_delete_matrix()

◆ is_aztecoo_workaround_for_epetra_matrix_setup_enabled()

◆ iterations()

◆ jacobian_setup_time()

◆ linear_solver_solution_time()

◆ max_iter()

◆ operator=()

◆ resolve()

◆ solve() [1/2]

◆ solve() [2/2]

◆ solve_using_AztecOO()

◆ solver_setup()

◆ solver_type()

◆ tolerance()

Member Data Documentation

◆ Assemble_serial_jacobian

◆ AztecOO_solver_pt

◆ Delete_matrix

◆ Epetra_matrix_pt

◆ Epetra_preconditioner_pt

◆ If_oomphlib_preconditioner_use_epetra_values

◆ Iterations

◆ Jacobian_setup_time

◆ Linear_solver_solution_time

◆ Oomph_matrix_pt

◆ Problem_pt

◆ Solver_type

◆ Use_aztecoo_workaround_for_epetra_matrix_setup

◆ Using_problem_based_solve