HP MPI Version 2.0 for Linux Intel IA-32 and Itanium2
Release Note
Manufacturing Part Number: T1919-90001 E0803 Release
U.S.A.
© Copyright 1979-2003 © Hewlett-Packard Development Company, L.P.
Legal Notices
The information in this document is subject to change without notice.
Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose.
Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental, or consequential damages in connection with the furnishing, performance, or use of this material.
Warranty. A copy of the specific warranty terms applicable to your Hewlett-Packard product and replacement parts can be obtained from your local Sales and Service Office.
Restricted Rights Legend. Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in sub-paragraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause in DFARS 252.227-7013.
HEWLETT-PACKARD COMPANY 3000 Hanover Street
Palo Alto, California 94304 U.S.A.
Rights for non-DOD U.S. Government Departments and Agencies are as set forth in FAR 52.227-19(c)(1,2).
Use of this manual and flexible disk(s), tape cartridge(s), CD(s), or DVD(s) supplied for this pack is restricted to this product only. Additional copies of the programs may be made for security and back-up purposes only. Resale of the programs in their present form or with alterations is expressly prohibited.
Copyright Notices. ©Copyright 1979-2003 Hewlett-Packard Development Company, L.P.
Reproduction, adaptation, or translation of this document without prior written permission is prohibited, except as allowed under the copyright laws.
Trademark Notices.
UNIX® is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Limited.
HP-UX Release 11i and later (in both 32 and 64-bit configurations) on all HP 9000 computers, is an Open Group UNIX 95-branded product.
PostScript® and PDF® are registered trademarks of Adobe Systems Incorporated.
subsidiaries in the United States and other countries.
All other trademarks and copyrights referred to are the property of their respective owners.
Announcement
HP MPI 2.0 for Linux
Intel IA-32 and Itanium2 Release Note
Announcement
This release note describes the release of HP MPI (Version 2.0) for Linux Intel® IA-32 and Itanium2®.
HP MPI 2.0 is the August 2003 release of HP MPI, Hewlett-Packard’s first fully functional implementation of the Message Passing Interface standard for Linux. HP MPI 2.0 for Linux is supported on servers and workstations running Red Hat Linux operating systems on Intel IA-32 and Itanium2 systems.
HP MPI 2.0 for Linux includes the following features:
• Full MPI-2 Standard functionality including:
— Dynamic processes
— Extended collectives
— One-sided communications
— Thread safety
— MPI_BOTTOM language interoperability
— Updated ROMIO
• Interconnects supported include:
— Quadrics Elan3 protocol on Itanium® 2-based systems with support for prun startup
— TCP/IP on clusters
— Shared memory for intranode communication
• Visual MPI for debugging and analysis
• Support for truncated messages
• Limited H/A mode
Announcement
• MPI versioning
• mpirun global environment variable settings via command line
• New mpirun command line options
What’s in This Version
What’s in This Version
HP MPI is a high-performance implementation of the Message Passing Interface standard.
HP MPI complies fully with the MPI-1.2 standard. HP MPI also complies fully with the MPI-2 standard, with restrictions. HP MPI provides an application programming interface and software libraries to support parallel, message-passing applications that are efficient, portable, and flexible.
HP MPI is available as both archive and shared libraries. Application performance is better when archive libraries are used. In order to use shared libraries, HP MPI must be installed on all machines in the same directory.
Platforms Supported
HP MPI 2.0 for Linux is supported on the following systems:
NOTE The generic term compiler used throughout this document always specifically refers to Intel compilers. HP MPI 2.0 for Linux was tested with the Intel Fortran compiler efc, version 7.0.087, the Intel C compiler ecc, version 7.0.086, and the GNU C compiler version 2.5.6. HP MPI 2.0 for Linux was built and tested with Kernel series 2.4 and glibc 2.2.
Table 1 Platforms Supported
Product
Number System
T1919
HP MPI Version 2.0 for Linux runs on Red Hat Linux V7.2 operating systems on Intel IA-32 and Itanium2 systems.
HP MPI is supported on:
• Workstations
• Midrange servers
• High-end servers
What’s in This Version
Benefits and Features
MPI has become the industry standard for distributed technical applications and is widely supported on most technical computing platforms. HP MPI offers several benefits.
• Applications developed using HP MPI port easily to other platforms, even those with a different architecture.
• HP MPI takes advantage of HP's shared-memory server architectures in order to get the most performance out of the system without compromising portability and flexibility.
• Technical application providers using HP MPI protect their software investment using a highly-efficient, portable tool for parallel application development.
HP MPI 2.0 for Linux includes the following features:
• MPI-2 Standard functionality including:
— Dynamic processes
— Extended collectives
— One-sided communications
— Thread safety
— MPI_BOTTOM language interoperability
— Updated ROMIO
• Interconnects supported include:
— Quadrics Elan3 protocol on Itanium® 2-based systems with support for prun startup
— TCP/IP on clusters
— Shared memory for intranode communication
• Visual MPI for debugging and analysis
• Support for truncated messages
• Limited H/A mode
• MPI versioning
• mpirun global environment variable settings via command line
• New mpirun command line options
What’s in This Version
Description of Features
The following section provides brief descriptions of the features included in this release (which are new since our V1.8 release for HP-UX). For more information on HP MPI, refer to the HP MPI User’s Guide available at http://docs.hp.com.
MPI-2 Standard Functionality HP MPI 2.0 for Linux complies fully with the MPI-2 Standard. MPI-2 is the latest revision of the MPI communication standard. The MPI-2 features in this release include:
• Dynamic Processes—spawn, connect and join functionality
• Extended Collectives—new intercommunicator creation and collective operations over intercommunicators
• One-Sided Communications
• Section 4 (Miscellany) features and functions
• Section 8 (External Interfaces) features and functions
• Section 10 (Language Bindings) features and functions
• Thread Safety—All MPI-2 entry points are thread safe
• MPI_BOTTOM Language Interoperability—Previous versions of HP MPI were not
compliant with Section 4.12.6.1 of the MPI-2 Standard which requires that sends/receives based at MPI_BOTTOM on a data type created with absolute addresses must access the same data regardless of the language in which the data type was created. If compliance with the standard is desired, set MPI_FLAGS=i to turn on language interoperability concerning the MPI_BOTTOM constant. Compliance with the standard can break source compatibility with some MPICH code.
• ROMIO—updated to the latest version (1.2.4)
Quadrics Elan3 protocol on Itanium2-based systems with support for prun startup The Quadrics Elan3 communication processor (referred to as the Elan) is supported on Itanium2-based HP systems.
All Elan usage requires the prun command, or mpirun with the -prun option. See “Running Applications” on page 11 for more information about prun usage.
We also support TCP/IP on clusters and Shared Memory for intranode communication.
Visual MPI for Debugging and Analysis Visual MPI is an MPI analysis tool focused on error detection and visualization, with automatic correlation to application source code. While Visual MPI includes a range of features, there are several highlights: ease of use (near-zero initial learning curve), automated analysis capabilities, and reporting of a range of
What’s in This Version
programming errors. For more information about Visual MPI, refer to the documents available at http://www.hp.com/go/mpi, /usr/doc/VisualMPI-**/README (where ** is the version number) after installation, and in the Visual MPI online help.
NOTE Visual MPI usage requires that your application is linked with the MPI shared libraries, and be started with the mpirun command.
Support for Truncated Messages We now allow an application to continue running normally after receiving truncated messages if the MPI_Errhandler is set to
MPI_ERRORS_RETURN.
1. The receive call returns an error class of MPI_ERR_TRUNCATE.
2. The count field in the resulting status argument represents the amount of data received by the receiver.
3. The receive buffer will only contain as much data as was specified in the receive call.
Limited H/A Mode If the user specifies the -ha option, HP MPI no longer initiates a teardown when ranks exit abnormally. Further communications involved with ranks that went away return error class MPI_ERR_EXITED, but do not force the application to teardown, as long as the MPI_Errhandler is set to MPI_ERRORS_RETURN. Some restrictions apply to this option:
• Cannot be used with prun
• Communication is done via TCP/IP (Does not use shared memory for intranode communication.)
• Cannot be used with the diagnostic library
• No instrumentation
MPI Versioning mpirun now includes an option to print the version number. The -version option used with mpirun displays the major and minor version numbers. The mpi.h header includes matching constants as HP_MPI and HP_MPI_MINOR.
mpirun Global Environment Variable Settings Via Command Line Previously in a multihost run, if a user wanted to specify an environment setting (e.g. MPI_FLAGS=y) the following options would need to be input on each line of the appfile:
-e MPI_FLAGS=y
With this release, environment variables can be set globally on the mpirun command line:
What’s in This Version
In the above example, if some MPI_FLAGS setting was specified in the appfile, then the global setting on the command line would override the setting in the appfile. To add to an environment variable rather than replacing it, use the following command:
%mpirun -e MPI_FLAGS=%MPI_FLAGS,y -f appfile
In the above example, if the appfile specified MPI_FLAGS=z, then the resulting MPI_FLAGS seen by the application would be z, y.
New mpirun Command Line Options New mpirun options are provided with HP MPI 2.0 for Linux (which were not available in HP MPI 1.8 for HP-UX).
• Previously, -commd provided both spawn and one-sided functions. New options included in this release are -spawn and -1sided. Use -spawn when you want dynamic processes. Use -1sided when you want one-sided communication. However, these new options my cause scalability issues when used with the Elan. -commd is still a valid option and its functionality has not changed.
NOTE -commd will cause an error when used with prun.
• -T prints user and system times for each MPI rank.
• -prot prints the communication protocol between each host (i.e. TCP/IP, Elan, or shared memory).
Running Applications
1. Set the following environment variables:
• Set the MPI_ROOT environment variable to point to the location where MPI is installed.
• Add $MPI_ROOT/bin to PATH.
• Add $MPI_ROOT/share/man to MANPATH.
MPI must be installed in the same directory on every execution host.
2. The following is an example using prun with mpirun:
% mpirun [mpirun options] -prun [prun options]
This method runs with no restrictions on MPI-2 functionality. Some features like mpirun -stdio processing are still unavailable.
The -np option is not allowed with -prun. The following options are allowed with -prun:
% mpirun [-help] [-version] [-jv] [-i <spec>] [-universe_size=#] [-sp
<paths>] [-T] [-prot] [-spawn] [-1sided] [-e var[=val]] -prun <prun options> <program> [<args>]
What’s in This Version
For more information on prun usage:
% man prun
3. All Elan usage requires the prun command, or mpirun with the -prun option. The following is an example using prun without mpirun:
% prun [options] application
The method has restrictions. It does not support MPI-2 dynamic processes or one-sided communication. We recommend certain environment variables be set before using this method. They are:
• LD_LIBRARY_PATH=$MPI_ROOT/lib/linux_[ia32|ia64]
if shared libraries were linked, which is the default. prun will not execute if this is not set.
• LIBELAN_SHM_ENABLE=0
This tells the Elan system not to allocate its own shared memory. Since we allocate our own shared memory, the Elan shared memory would be ignored.
NOTE Some versions of Quadrics have a bug that causes multithreaded applications to hang. Do not set LIBELAN_SHM_ENABLE if you are running multithreaded applications.
Known Problems and Workarounds
Known Problems and Workarounds
• Profiling routines built for C calls will no longer cause the corresponding Fortran calls to be wrapped automatically. In order to profile Fortran routines, separate wrappers need to be written for the Fortran calls.
• Calling MPI from Fortran 90 or C++ programs
HP MPI complies with the 1.2 version of the MPI standard, which defines bindings for Fortran 77 and C, but not Fortran 90 or C++. Some features of Fortran 90 may interact with MPI non-blocking semantics to produce unexpected results. Consult the HP MPI User’s Guide for details. C++ applications should be able to use the existing C binding for MPI with no problems.
• Locating your instrumentation output file (prefix.instr) when you enable instrumentation for multi-host runs
Whether you invoke mpirun on a host where at least one MPI process is running, or on a host remote from all your MPI processes, HP MPI writes the instrumentation output file to the working directory on the host that is running rank 0.
• Certain compiler options are incompatible with the TotalView debug information.
Use option -notv to exclude incompatible debug information. This option allows the use of the special compiler options that don’t tolerate debug information. But, this option also disables TotalView usage on the resulting executable.
The -notv option is only applicable when using archive libraries. Using -notv when linking with shared libraries is not advised.
• Extended collectives with intercommunicators are not profiled by our lightweight instrumentation mode.
• H/A mode and the diagnostic library are not allowed at the same time.
• The diagnostic library strict mode is not compatible with some MPI-2 features.
• Some versions of Quadrics have a memory leak. The error received will look like:
0 MAP_SDRAM(140008800): can't map SDRAM 2824000(2404000) - 4020000(3c00000) (25149440 bytes) : -1
ELAN_EXCEPTION @ 0: 5 (Memory exhausted) newRxDesc: Elan memory exhausted: port 2b200
This error can occur in following two cases:
Known Problems and Workarounds
1. If the application calls MPI_Cancel repeatedly.
2. If the application receives on MPI_ANY_SOURCE.
In either of the above cases, contact Quadrics for the fix (which is unavailable at the time of this release).
Installation Requirements
Installation Requirements
HP MPI 2.0 for Linux is supported on Red Hat Linux. You must install the correct HP MPI product for your system.
HP MPI requires a minimum of 50 MB of disk space in /opt.
Installation Instructions
Installation Instructions
1. Become root by running the su command and entering the superuser password.
2. Place the downloaded file into the /tmp directory (or another suitable directory).
3. Run the rpm command to install the package.
% rpm -ivh /tmp/hpmpi-2.00.00-00.[i386|ia64].rpm
Licensing Policy
Licensing Policy
HP MPI 2.0 for Linux uses FLEXlm licensing technology. A license is required to use HP MPI for Linux. Licenses can be purchased from HP’s software depot at
http://www.software.hp.com, or contact your HP representative.
Demo licenses for HP MPI are also available from HP’s software depot.
Licensing
HP MPI 2.0 for Linux uses FLEXlm licensing technology. A license file can be named either as license.dat or any file name with an extension of .lic. The default location to place MPI license files is in the default installation directory /opt/mpi/licenses.
You will need to provide the hostname and hostid number of the system where the FLEXlm daemon for HP MPI for Linux will run. The hostid can be obtained either by typing the following command if HP MPI is already installed on the system:
% /opt/mpi/bin/lmutil lmhostid Or:
% /sbin/ifconfig | egrep "^eth0" | awk'{print $5}' | sed s/://g The hostname can be obtained by entering the command hostname.
The default search path used to find an MPI license file is:
% $MPI_ROOT/licenses:/opt/mpi/licenses:.
For example, if MPI_ROOT=/home/mpi, license files will be searched in the following order:
/home/mpi/licenses/license.dat /home/mpi/licenses/*.lic /opt/mpi/licenses/license.dat /opt/mpi/licenses/*.lic ./license.dat
./*.lic
If the license needs to be placed in another location which would not be found by the above search, the user may set the environment variable LM_LICENSE_FILE to explicitly specify the location of the license file.
For more information, see http://licensing.hp.com.
Licensing Policy
Installing License Files
A valid license file contains the system hostid and the associated license key. License files can be named either as license.dat or any name with extension of *.lic (like mpi.lic, for example). Copy the license file under the directory /opt/mpi/licenses.
The command to run the license server is:
% $MPI_ROOT/bin/lmgrd -c mpi.lic
License Testing
Build and run the hello_world program in $MPI_ROOT/help/hello_world.c to check for a license. If your system is not properly licensed, you will receive the following error message:
(“MPI BUG: Valid MPI license not found in search path”)
Product Documentation
Product Documentation
The HP MPI Documentation Kit is an optional product (product number B6281AA) consisting of the following hardcopy books:
• MPI: The Complete Reference (2 volume set) Product number B6011-96012
• HP MPI User’s Guide (Eighth Edition) Product number B6060-96013 The HP MPI User’s Guide and HP MPI Release Notices are available online
• In /opt/mpi/doc after you install the product
• At http://docs.hp.com
• At http://www.hp.com/go/mpi
Refer to http://www.hp.com/go/mpi for information about the HP MPI product.
Additional product documentation:
• mpi(3m) man pages installed in /opt/mpi/man
• Visual MPI and vmpitrace man pages installed in /usr/share/man
• /usr/doc/VisualMPI-**/README (Where ** is the version number.)
• Visual MPI online help
On systems that have the Version 7 Intel compiler, documentation is installed by default with the compiler at /opt/intel/compiler70/docs.
Product Packaging
Product Packaging
HP MPI is packaged as an optional software product installed in /opt/mpi.
Software Availability in Native Languages
Software Availability in Native Languages
There is no information on non-English languages for HP MPI for Linux.