Manuale d’uso / di manutenzione del prodotto IB6054601-00 D del fabbricante Q-Logic
Vai alla pagina of 122
IB6054601-00 D Page i Q Simplify InfiniPath User Guide V ersion 2.0.
InfiniPath User Guide Version 2.0 Q Page ii IB6054601-00 D Information fu rnished in this manual is believed to be accurate and reliab le. However , QLogic Corporation assumes no responsibility for its use, nor for any infringements of patent s or othe r rights of third pa rti es which may result from its use .
InfiniPath User Guide Version 2.0 Q IB6054601-00 D Page iii Added info about using MPI over uDAPL. Need to load modules rdma_cm and rdma_ucm. 3.7 Added section: Error me ssages gener ated by mpirun. T his explains more about the types of errors fo und in the sub-secti ons.
InfiniPath User Guide Version 2.0 Q Page iv IB6054601-00 D © 2006, 2007 QLogic Cor poration. All rights reserved worldwide . © PathScale 2004, 20 05, 2006.
IB6054601-00 D Page v Table of Contents Section 1 Introduction 1.1 Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 1.2 How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . .
InfiniPath User Guide Version 2.0 Page vi IB6054601-00 D Q 2.10 Performance and Management Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-17 2.10.1 Remove Unneeded Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
InfiniPath User Guide IB6054601-00 D Page vii Q InfiniPath User Guide Version 2.0 3.11 Debugging MPI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 3.11.1 MPI Errors . . . . . . . . . . . . . . . . . . . . . . . .
InfiniPath User Guide Version 2.0 Page viii IB6054601-00 D Q C.4.5 OpenFabrics Load Errors If ib_ipath Driver Load Fails . . . . . . . . . . C-10 C.4.6 InfiniPath ib_ipath Initialization Failure . . . . . . . . . . . . . . . . . . . . . . C-1 1 C.4.7 MPI Job Failures Due to Initialization Problems .
InfiniPath User Guide IB6054601-00 D Pa ge ix Q InfiniPath User Guide Version 2.0 C.9.1 1 ipath_pkt_test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-35 C.9.12 ipathstats . . . . . . . . . . . . . . . . . . . . . .
InfiniPath User Guide Version 2.0 Page x IB6054601-00 D Q Notes.
IB6054601-00 D 1-1 Section 1 Introduction This chapter describes the ob jectives, in tended audience, and organization of the InfiniPath User Guide. The InfiniPath User Guide is intended to give the end users of an Inifin iPath cluster what they need to know to use it.
1 – Introduction Interoperability 1-2 IB6054601-00 D Q ■ Appendix E Glossary of technical terms ■ Index In addition, the InfiniPath Install Guide contain s information on InfiniPath hardware and software inst allation. 1.3 Overview The material in this docu mentation pertains to an In finiPath cluster .
1 – Introduction What’s New in this Release IB6054601-00 D 1-3 Q NOTE: OpenFabrics was known as OpenIB until March 2006. All relevant references to OpenIB in this documentation have been updated to reflect this change. See th e OpenFabrics website at http://www .
1 – Introduction Supported Distrib utions and Kernels 1-4 IB6054601-00 D Q Support for multiple versio ns of MPI has been added. Y ou can use a different version of MPI and achieve the high-ba ndwidth and low-latency performance that is standard with InfiniPath MPI.
1 – Introduction Software Components IB6054601-00 D 1-5 Q 1.8 S o ft w a r e C o m p o n e n ts The software p rovided with the InfiniPath Interconnect product co nsists of: ■ InfiniPath driver (i.
1 – Introduction Documentation and T echnical Support 1-6 IB6054601-00 D Q NOTE: 32 bit OpenFabrics programs using the verb interfaces are not supported in this InfiniPath release, but w ill be suppo rted in a future release. 1.9 Conventions Used in this Document This Guide uses these ty pographical conventions: 1.
1 – Introduction Documentation and Technica l Support IB6054601-00 D 1-7 Q ■ Readme file The T roubleshooting Appendix for inst allation, InfiniPath and OpenFabrics administration, and MPI issues is located in the InfiniPath User Guide . Visit the QLogic support Web site for document ation and the latest software updates.
1 – Introduction Documentation and T echnical Support 1-8 IB6054601-00 D Q Notes.
IB6054601-00 D 2-1 Section 2 InfiniPath Cluster Administration This chapter describes what the cluster administra tor needs to know about the InfiniPath sof tware and system administration.
2 – InfiniPath Clus ter Administratio n Memory Footpr int 2-2 IB6054601-00 D Q MPI include files are in: /usr/include MPI programming examples and source for several MPI benchmarks are in: /usr/shar.
2 – InfiniPath Cluster Administration Memory Footprint IB6054601-00 D 2-3 Q on system configura tion. OpenFabrics support is under develo pment and has not been fully characterized.
2 – InfiniPath Clus ter Administratio n Configuration and Startup 2-4 IB6054601-00 D Q This breaks down to a memory footprint of 331MB per no de, as follows: 2.4 Configuration and S tartup 2.4.1 BIOS Settings A properly configured BIOS is required .
2 – InfiniPath Cluster Administration Configuration and Startup IB6054601-00 D 2-5 Q Y ou can check and adjust these BIOS setti ngs using t he BIOS Setup Utility . For specific instructions on how to do this, follow the hardware document ation that came with your system.
2 – InfiniPath Clus ter Administratio n Configuration and Startup 2-6 IB6054601-00 D Q and unmounted when the infinip ath script is invoked with the "stop" option (e.g. at system shutdown). The layout of the filesystem is as follows: atomic_stats 00/ 01/ .
2 – InfiniPath Cluster Administration Configuration and Startup IB6054601-00 D 2-7 Q Y ou must create a network device configuration file for the layered Ethernet device on the InfiniPath adapter . This configuration file will resemble the configuration files for the other Ethernet devices on the no des.
2 – InfiniPath Clus ter Administratio n Configuration and Startup 2-8 IB6054601-00 D Q If you are using DHCP (dynamic host configuration protoco l), add the following lines to ifcfg-eth2 : # QLogic .
2 – InfiniPath Cluster Administration Configuration and Startup IB6054601-00 D 2-9 Q S tep 3 is applicable only to SLES 10; it is required because SLES 10 uses a newer version of the udev subsystem. NOTE: The MAC address (media access control address) is a unique identifier attached to most forms of networking equipment.
2 – InfiniPath Clus ter Administratio n Configuration and Startup 2-10 IB6054601-00 D Q Check each of the lines st arting with SUBSYSTEM= , to find the highest numbered interface. (For st andard motherboards, the highest numbered interface will typically be 1.
2 – InfiniPath Cluster Administration Configuration and Startup IB6054601-00 D 2-11 Q 6. T o verify that the configuration files are correct, you will normally now be able to run the commands: # ifup eth2 # ifconfig eth2 Note that it may be necessary to reboot the syste m before the configuration changes will work.
2 – InfiniPath Clus ter Administratio n Configuration and Startup 2-12 IB6054601-00 D Q T o verify the configuration, type: # ifconfig ib0 The output from this command should be similar to this: ib0 Link encap:InfiniBand HWaddr 00:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:10.
2 – InfiniPath Cluster Administration Starting and Stopping the InfiniPath Software IB6054601-00 D 2-13 Q and you can stop it ag ain like this: # /etc/init.d/opensmd stop If you wish to pass any arguments to the OpenSM program, mo dify the file: /etc/init.
2 – InfiniPath Clus ter Administratio n Starting and Stopping the InfiniPath Software 2-14 IB6054601-00 D Q T o disable the driver on the next system boot, use the command (as ro ot): # chkconfig infinipath off NOTE: This does not stop and unload the driver , if it is already loaded.
2 – InfiniPath Cluster Administration Configuring ssh and sshd Using shosts.equiv IB6054601-00 D 2-15 Q If there is output, you should look at the ou tput from this command to determine if it is con.
2 – InfiniPath Clus ter Administratio n Configuring ssh and sshd Using shosts.equiv 2-16 IB6054601-00 D Q This next example assumes the following: ■ Both the cluster nodes and the front en d system are running the openssh package as d istributed in current Linux systems.
2 – InfiniPath Cluster Administration Performance and Manag ement Tips IB6054601-00 D 2-17 Q 0zwxSL7GP1nEyFk9wAxCrXv3xPKxQaezQKs+KL95FouJvJ4qrSxxHdd1NYNR0D avEBVQgCaspgWvWQ8cL 0aUQmTbggLrtD9zETVU5PCgRlQL6I3Y5sCCHuO7/UvTH9nneCg== Change the file to mode 60 0 when finished editing.
2 – InfiniPath Clus ter Administratio n Performance a nd Management Tips 2-18 IB6054601-00 D Q nodes. Since these are presumed t o be specialized computi ng appliances, they do not need many of the service daemons normally running on a general Linux computer .
2 – InfiniPath Cluster Administration Performance and Manag ement Tips IB6054601-00 D 2-19 Q For SUSE 9.3 and 10.0 run this comman d as root: # /sbin/chkconfig --level 12345 powersaved off After running e ither of these commands, the system will need to be rebooted for these changes to t ake effect.
2 – InfiniPath Clus ter Administratio n Performance a nd Management Tips 2-20 IB6054601-00 D Q 2.10.6 Hyper-Threading If using Intel processors that support Hyper-Th reading, it is recommended that HyperThreading is turned o ff in the BIOS. This will provide more consistent performance.
2 – InfiniPath Cluster Administration Performance and Manag ement Tips IB6054601-00 D 2-21 Q 00: LID=0x30 MLID=0x0 GUID=00:11:75:00:00:07:11:97 Serial: 1236070407 Note that i path_control will report whether the inst alled adapter is the QHT7040, QHT7140, or the QLE7 140.
2 – InfiniPath Clus ter Administratio n Customer Acceptance Utility 2-22 IB6054601-00 D Q $Id: kernel.org InfiniPath Release 2.0 $ $Date: 2006-09-15-04:16 $ /lib/modules/2.6.16.21-0.8-smp/updates/ipath.ko: $Id: kernel.org InfiniPath Release2.0 $ $Date: 2006-09-15-04:20 $ NOTE: ident is in the optional rcs RPM, and is not always inst alled.
2 – InfiniPath Cluster Administration Customer Acceptance Utility IB6054601-00 D 2-23 Q 3. Gather a nd analyze system configuration from nodes. 4. Gather a nd analyze RPMs installed on nodes. 5. V erify InfiniPath hardware and software status and co nfiguration.
2 – InfiniPath Clus ter Administratio n Customer Acceptance Utility 2-24 IB6054601-00 D Q Notes.
IB6054601-00 D 3-1 Section 3 Using InfiniPath MPI This chapter provides information on using InfiniPath MPI. Examp les are provided for compiling and running MPI programs. 3.1 InfiniPath MPI QLogic’s imple mentation of the MPI standard is derived fro m the MPICH reference implementation V ersion 1.
3 – Using InfiniPath MPI Getting Started with MPI 3-2 IB6054601-00 D Q These examples assume that: ■ Y our cluster administrator has properly inst alled InfiniPath MPI and the PathScale compilers. ■ Y our cluster ’s policy allows you to use t he mpirun script directly , without having to submit the job to a batch queuing system.
3 – Using InfiniPath MPI Getting Started with MPI IB6054601-00 D 3-3 Q Here ./cpi designates the execut able of the example program in the working directory . The -np parameter to mpirun defines the number of processes to be used in the paralle l computation.
3 – Using InfiniPath MPI Configuring MPI Programs for Infin iPath MPI 3-4 IB6054601-00 D Q and run it with: $ mpirun -np 2 -m mpihosts ./pi3f90 The C++ program hello++.cc is a p arallel processing version of the traditional “Hello, World” program.
3 – Using InfiniPath MPI InfiniPath MPI Details IB6054601-00 D 3-5 Q Y ou may need to instead p ass arguments to configure directly , in a fashion similar to this: $ .
3 – Using InfiniPath MPI InfiniPath MPI Details 3-6 IB6054601-00 D Q The process is shown in the following step s: 1. Create a key pair . Use the default file name, and be sure to enter a p assphrase. $ ssh-keygen -t rsa 2. Enter a passphrase for your key pair when prompted.
3 – Using InfiniPath MPI InfiniPath MPI Details IB6054601-00 D 3-7 Q 3.5.2 Compiling and Linking These scripts invoke the compiler and linker for programs in each of the respective languages, and t ake care of referring to the correct include files and libraries in each case.
3 – Using InfiniPath MPI InfiniPath MPI Details 3-8 IB6054601-00 D Q line options. See the PathScale compiler documen tation and the man p ages for pathcc and pathf90 for complete information o n its options. See the corresponding documentation for any other compiler/linker you may call for it s options.
3 – Using InfiniPath MPI InfiniPath MPI Details IB6054601-00 D 3-9 Q T o use the Intel compiler for Fortran90/Fortran95 programs, use: $ mpif90 -f90=ifort ..... $ mpif95 -f95=ifort ..... Usage for other compilers will be similar to the examples above, substituting the options following -cc , -CC , -f77 , -f90 , or -f95 .
3 – Using InfiniPath MPI InfiniPath MPI Details 3-10 IB6054601-00 D Q The current workaround for this is to comp ile on a supported and compatible distribution, then run the execut able on one of the systems that uses the GNU 4.x compilers and environment.
3 – Using InfiniPath MPI InfiniPath MPI Details IB6054601-00 D 3-11 Q program-name will generally be the p athname to the executable MPI program. I f the MPI program resides in the curr ent directory and the current directory is not in your search path, the n program-name must begin with ‘.
3 – Using InfiniPath MPI InfiniPath MPI Details 3-12 IB6054601-00 D Q programs will be started on that host before using the next entry in the mpihosts file. If the full mpihosts file is processed, and there are still more processes requested, processing st arts again at the st art of the file.
3 – Using InfiniPath MPI InfiniPath MPI Details IB6054601-00 D 3-13 Q LD_LIBRARY_PATH, and other environment variables for the node programs through the use of the -rcfile option o f mpirun: $ mpirun -np n -m mpihosts -rcfile mpirunrc program In the absence of this option, mpirun checks to see if a file called $HOME/.
3 – Using InfiniPath MPI InfiniPath MPI Details 3-14 IB6054601-00 D Q 3.5.9 Multiprocessor Node s Another command line option, -ppn , i nstructs mpirun to assign a fixed numbe r p of node programs t.
3 – Using InfiniPath MPI InfiniPath MPI Details IB6054601-00 D 3-15 Q -verbose Print diagnostic messages from mpir un itself. Can be useful in troubleshooting Default: Off -version, -v Print MPI version. Default: Of f -help, -h Print mpirun help message.
3 – Using InfiniPath MPI InfiniPath MPI Details 3-16 IB6054601-00 D Q -nonmpi Run a non-MPI program. Required if the node program makes no MPI calls. Default: Off -quiescence-timeout, seconds W ait time in seconds for quiescence (absence o f MPI communication) on the nodes.
3 – Using InfiniPath MPI MPD IB6054601-00 D 3-17 Q -statsfile file-prefix S pecifies alternate file to receive the output from the -print-stats option. Default: stderr 3.6 Using Other MPI Implement ations Support for multiple MPI implement ations has been added.
3 – Using InfiniPath MPI File I/O in MPI 3-18 IB6054601-00 D Q 3.8.1 MPD Description The Multi-Purpose Daemon (MPD) was dev eloped by Argonne National Laborato ry (ANL), as part of the MPICH-2 system.
3 – Using InfiniPath MPI InfiniPath MPI and Hybrid MPI/OpenMP Applicatio ns IB6054601-00 D 3-19 Q accessed via some network file system, typically NFS. Paralle l programs usually need to have some dat a in files to be shared by all of the processes of an MPI job.
3 – Using InfiniPath MPI Debugging MP I Programs 3-20 IB6054601-00 D Q may be desirable to run multiple MPI processes and multiple OpenMP threads per node. The number of OpenMP threads is typically controlled by th e OMP_NUM_THREADS environment variable in the .
3 – Using InfiniPath MPI InfiniPath M PI Limitations IB6054601-00 D 3-21 Q Symbolic debugging is easier than machine language debugging. T o enable symbolic debugging you must have compile d with the -g option to mpicc so that the compiler will have included symbol t ables in the compiled object code.
3 – Using InfiniPath MPI InfiniPath M PI Limitations 3-22 IB6054601-00 D Q No ports available on /dev/ipath NOTE: If port sharing is enable d, this limit is raised to 16 and 8 respectively . T o enable port sharing, set PSM_SHAREDPOR TS=1 in your environment There are no C++ bindings to MPI -- use the exte rn C MPI function calls.
IB6054601-00 D A-1 Appendix A Benchmark Programs Several MPI performance measurement programs are inst alled from the mpi-benchmark RPM. This Appendix describe s these useful benchmarks and how to run them. These pr ograms are based on code from the group of Dr .
A – Benchmark Programs Benchmark 2: Mea suring MPI Bandwidth Between Two Node s A-2 IB6054601-00 D Q This benchmark always involves just two node programs. Y ou can run it with the command: $ mpirun -np 2 -ppn 1 -m mpihosts osu_latency The -ppn 1 option is ne eded to be certain that the two communicatin g processes are on differe nt nodes.
A – Benchmark Programs Benchmark 3: Messaging Rate Microbenchmarks IB6054601-00 D A-3 Q MPI_Isend function, while th e receiving node consumes them as quickly as it can using the non-blocking MPI_Irecv , and then returns a zero-length acknowledgement when all of the set has be en received.
A – Benchmark Programs Benchmark 3: Messaging Rate Micr obenchmarks A-4 IB6054601-00 D Q benchmark (as shown in the example above). It ha s been enhanced with the following additional functionality:.
A – Benchmark Programs Benchmark 4: Measuring MPI Latency in Host Rings IB6054601-00 D A-5 Q A.4 Benchmark 4: Measuring MPI Latency in Host Rings The program mpi_latency can be used to measure latency in a ring of hosts.
A – Benchmark Programs Benchmark 4: Measur ing MPI Latency in Host Rings A-6 IB6054601-00 D Q Notes.
IB6054601-00 D B-1 Appendix B Integration with a Batch Queuing System Most cluster systems use some kind of ba tch queuing system as an orderly way to provide users with access to the resou rce s they need to meet their job’s performance requirements.
B – Integration with a Batch Queuing System A Batch Queu ing Script B-2 IB6054601-00 D Q require that his node program be the on ly application running on each node CPU.
B – Integration with a Batch Queuing Syst em A Batch Queuing Script IB6054601-00 D B-3 Q by mpirun. Each line consists of a node name, a colon , and the number of processes to start on that node. NOTE: This is one of two format s that the file may use.
B – Integration with a Batch Queuing System Lock Enough Memory on Nodes When Using SLURM B-4 IB6054601-00 D Q The following command will terminate all processes using the InfiniPath interconnect: # /sbin/fuser -k /dev/ipath For more information, see the man pages for fuser(1) and lsof(8).
IB6054601-00 D C-1 Appendix C T roubleshooting This Appendix describes some of the exis ting provisions fo r diagnosing and fixing problems. The sections a re organized in the following order: ■ C.1 “T roubleshooting InfiniPath adapter inst allation” ■ C.
C – Troubleshooting BIOS Settings C-2 IB6054601-00 D Q states of the LEDs. The gre en LED will normally illuminate first. The normal state is Green On, Amber On.
C – Troubleshooting BIOS Settings IB6054601-00 D C-3 Q C.2.1 MTRR Mapping and Write Combining MTRR (Memory T ype Range Registers) is us ed by the InfiniPath driver to enable write combining to the InfiniPa th on-chip transmit buffers.
C – Troubleshooting BIOS Settings C-4 IB6054601-00 D Q C.2.3 Incorrect MTRR Mapping Causes Unexpected Low Bandwid t h This same MTRR Mapping setting a s described in the previous section can also cause unexpected low bandwid th if it is set incorrectly .
C – Troubleshooting Software Installation Issues IB6054601-00 D C-5 Q C.3 Sof tware Inst allation Issues This section cove rs issues related to sof tware installation. C.3.1 OpenFabrics Depe ndencies Y ou need to install sysfsutils for your distributio n before installing the OpenFabrics RPMs, as there are dependencies.
C – Troubleshooting Software Installation Issues C-6 IB6054601-00 D Q In older distributions, such as RHEL4, the 32-bit glibc will be contained in the libgcc RPM. The RPM will be named similarly to: libgcc-3.4.3-9.EL4.i386.rpm In newer distributions, glibc is an RPM name.
C – Troubleshooting Kernel and Initialization Issues IB6054601-00 D C-7 Q 8. Relo ad all modules by using this command (as root): # /etc/init.d/infinipath start An alternate mechanism can be used, if provide d as part of your alternate installation .
C – Troubleshooting Kernel and Initializ ation Issues C-8 IB6054601-00 D Q C.4.1 Kernel Needs CONFIG_PCI_MSI=y If the InfiniPath driver is being compil ed on a machine without CONFIG_PCI_MSI=y configured, you will get a compilation error similar to this: ib_ipath/ipath_driver.
C – Troubleshooting Kernel and Initialization Issues IB6054601-00 D C-9 Q NOTE: This problem has been fixed in the 2.6.17 kernel.org kernel. C.4.3 Driver Load Fails Due to Unsupported Kernel If you try to load th e InfiniPath driver on a kernel that InfiniPath sof tware does not support, the load fails.
C – Troubleshooting Kernel and Initializ ation Issues C-10 IB6054601-00 D Q A zero count in all CPU columns me ans that no interrupts have bee n delivered to the processor .
C – Troubleshooting Kernel and Initialization Issues IB6054601-00 D C-11 Q C.4.6 InfiniPath ib_ipath Initialization Failure There may be cases where ib_ipath was not properly initialized. Symptoms of this may show up in error messages from an MPI job or another program.
C – Troubleshooting System Administration Troubleshooting C-12 IB6054601-00 D Q C.5 OpenFabrics Issues This section covers items related to Open Fabrics, including OpenSM. C.5.1 S top OpenSM Before Stoppi ng/Rest arting InfiniPath OpenSM must be stopped before stopping or rest arting InfiniPath.
C – Troubleshooting InfiniPath MPI Troubleshooting IB6054601-00 D C-13 Q C.6.1 Broken Intermediate Link Sometimes message traffic p asses through the fabric while other traf fic appears to be blocked.
C – Troubleshooting InfiniPath MPI Tr oubleshooting C-14 IB6054601-00 D Q $ mpirun -v MPIRUN:Infinipath Release2.0 : Built on Wed Nov 19 17:28:58 PDT 2006 by mee The following is the error that occurs when m pirun from the 2.0 release is being used with the 1.
C – Troubleshooting InfiniPath MPI Troubleshooting IB6054601-00 D C-15 Q On a SLES 10 system, you would need: ■ compat-libst dc++ (for FC3) ■ compat-libst dc++5 (for SLES 10) Depending upon the ap plication, you may need to use the -W1 ,- Bstatic o ption to use the static ve rsions of some libraries.
C – Troubleshooting InfiniPath MPI Tr oubleshooting C-16 IB6054601-00 D Q For these examples in Section C.8.5 below , we assume that these new locations are: /path/to/devel (for mpi-devel-*) /path/to/libs (for mpi-libs-*) C.
C – Troubleshooting InfiniPath MPI Troubleshooting IB6054601-00 D C-17 Q The above compiler command insures that the program will run using this path on any machine. For the second option, we change the file /etc/ld.so.conf on the compute nodes rather than using the -Wl,-rpath , option when compiling on the development node .
C – Troubleshooting InfiniPath MPI Tr oubleshooting C-18 IB6054601-00 D Q Examples are given below . In the following command, the HP-MPI version of mpirun is invoked by the full pathname. Howeve r , the program mpi_nxnlatbw was compiled with the QLogic version of mpicc .
C – Troubleshooting InfiniPath MPI Troubleshooting IB6054601-00 D C-19 Q The following two commands will bo th work properly: QLogic mpirun and execut able used together: $ mpirun -m ~/host-bbb -np .
C – Troubleshooting InfiniPath MPI Tr oubleshooting C-20 IB6054601-00 D Q ^ pathf95-389 pathf90: ERROR BORDERS, File = communicate.F, Line = 407, Column = 18 No specific match can be found for the generic subprogram call "MPI_RECV".
C – Troubleshooting InfiniPath MPI Troubleshooting IB6054601-00 D C-21 Q integer count, datatype, root, co mm, ierror ! Call the Fortran 77 style impli cit interface to "mpi_bcast" externa.
C – Troubleshooting InfiniPath MPI Tr oubleshooting C-22 IB6054601-00 D Q If this file is not present or the node has not been rebooted af ter the infinipath RPM has been inst alled, a failure message similar to this will be generated: $ mpirun -m ~/tmp/sm -np 2 -mpi_latency 1000 1000000 node-00:1.
C – Troubleshooting InfiniPath MPI Troubleshooting IB6054601-00 D C-23 Q Found unknown timer type type unknown frame type type recv done: available_tids now n, but max is m (freed p) cancel recv available_tids now n, but max is m (freed %p) [n] Src lid error: sender: x, exp send: y Frame receive from unknown sender.
C – Troubleshooting InfiniPath MPI Tr oubleshooting C-24 IB6054601-00 D Q The following message indicates th at a node program may not be processing incoming packe ts, perhaps due to a very high sys.
C – Troubleshooting InfiniPath MPI Troubleshooting IB6054601-00 D C-25 Q These messages appear in the mpirun output. Most a re followed by an abort, and possibly a backtrace. Ea ch is preceded by the name of the function in which the exception occurred.
C – Troubleshooting InfiniPath MPI Tr oubleshooting C-26 IB6054601-00 D Q There is no route to any host: $ mpirun -np 2 -m ~/tmp/q mpi_latency 100 100 ssh: connect to host <nodename> port 22: .
C – Troubleshooting InfiniPath MPI Troubleshooting IB6054601-00 D C-27 Q $ mpirun -np 2 -m ~/tmp/q -q 60 mpi_latency 1000000 1000000 MPIRUN: MPI progress Quiescence Detected after 9000 seconds. MPIRUN: 2 out of 2 ranks showed no MPI send or receive progress.
C – Troubleshooting InfiniPath MPI Tr oubleshooting C-28 IB6054601-00 D Q C.8.13 MPI St ats Using the -print-stats option to mpirun will result in a listing to stderr of various MPI statistics. Here is example o utput for the -print-stats option when used with an 8-rank run of the HPCC benchmark.
C – Troubleshooting Useful Programs and File s for Debugging IB6054601-00 D C-29 Q C.9 Useful Programs and Files f or Debugging The most useful programs and files for debugging are listed in the sections below .
C – Troubleshooting Useful Programs and Files for Debugging C-30 IB6054601-00 D Q C.9.3 Summary of Useful Programs and Files Useful programs and files are summarized in the table below . Descriptions for some of the programs and files follow . Check ma n pages for mo re information on the programs.
C – Troubleshooting Useful Programs and File s for Debugging IB6054601-00 D C-31 Q C.9.4 boardversion It may be useful to keep track of the current version of the inst alled software. Y ou can check the version of the installed In finiPath software by looking in: /sys/bus/pci/drivers/ib_ipath/00/boardversion Example content s are: Driver 2.
C – Troubleshooting Useful Programs and Files for Debugging C-32 IB6054601-00 D Q C.9.5 ibstatus This program displays basic information on t he st atus of InfiniBand devices that are currently in use when the OpenFabrics modules are loaded .
C – Troubleshooting Useful Programs and File s for Debugging IB6054601-00 D C-33 Q C.9.8 ipath_checkout ipath_checkout is a bash script used to verify that the inst allation is correct and that all the nodes of the network are functioning and mutually connected by the InfiniPath fabric.
C – Troubleshooting Useful Programs and Files for Debugging C-34 IB6054601-00 D Q --workdir=DIR Use DIR to hold intermediate files crea ted while running tests. DIR must not already exist. -k, --keep Keep intermediate files that were created while pe rforming tests and compiling reports.
C – Troubleshooting Useful Programs and File s for Debugging IB6054601-00 D C-35 Q 00: LID=0x30 MLID=0x0 GUID=00:11:75:00:00:07:11:97 Serial: 1236070407 C.
C – Troubleshooting Useful Programs and Files for Debugging C-36 IB6054601-00 D Q C.9.13 lsmod If you need to find which InfiniPath and OpenFabrics modules are running, try the following command: # lsmod | egrep ’ipath_|ib_|rdma_|findex’ C.
C – Troubleshooting Useful Programs and File s for Debugging IB6054601-00 D C-37 Q The following table sh ows the possible contents of the file, with brief explanations of the entries. In this same directory are other files cont aining information related to st atus.
C – Troubleshooting Useful Programs and Files for Debugging C-38 IB6054601-00 D Q C.9.17 strings The command strings can also be used. It s format is as follows: $ strings /usr/lib/libinfinipath.so.4.0 | grep Date: will produce output like this: $Date: 2006-09-15 04:07 Release2.
IB6054601-00 D D-1 Appendix D Recommended Reading Reference material for furthe r reading is provided here. D.1 References for MPI The MPI S tandard specification document s. http://www .mpi-forum.org/docs The MPICH implementation of MPI and its documentation.
D – Recommended Reading Rocks D-2 IB6054601-00 D Q D.6 Clusters Gropp, William, Ewing Lusk, and Thomas S terling, Beowulf Cluster Computing with Linux , Second Edition, 2003, MIT Press, ISBN 0-262-69292-9. D.7 Rocks Extensive document ation on instal ling Rocks and cu stom Rolls.
IB6054601-00 D E-1 Appendix E Glossary A glossary is provided below for technica l terms used in the documentation. bandwid th The rate at which dat a can be transmitted.
E – Glossary E-2 IB6054601-00 D Q GID For Global Identifier . Used for routing between dif ferent InfiniBand subnet s. GUID For Globally Unique Identifier for the InfiniPath chip. Equivalent to Ethernet MAC address. head node Same as front end node .
E – Glossary IB6054601-00 D E-3 Q LID For Local Identifier . Assigned by the Subnet Manager (SM) to each visible node within a sin gle InfiniBand fabric. It is similar conceptually to an IP ad dress for TCP/IP . Lustre Open source project to dev elop scalable cluster file systems.
E – Glossary E-4 IB6054601-00 D Q MTRR For Memory T y pe Range Registers . MTRR For "Memory T ype Range Registers". Used by the InfiniPath driver to enable write combinin g to the InfiniPath on-chip transmit bu f fers.
E – Glossary IB6054601-00 D E-5 Q SDP For S ockets Direct Protocol . An I nfiniBand-specific upper layer protocol. It defines a standard wire protocol to support stream socket s networking over InfiniBand.
E – Glossary E-6 IB6054601-00 D Q Notes.
IB6054601-00 D Index- 1 Index A ACPI, enabling C-9 B Batch queuing for MPI jobs B-1 – B-4 Benchmarking MPI bandwidth A-2 – A-3 MPI latency measurement A-1 – A-2 MPI latency measurement in host r.
InfiniPath User Guide Version 2.0 Beta2 Index-2 IB6054601-00 D Q configuration of on SUSE and SLES 10 2-8 – 2-11 layered Ethernet driver 2-6 ipathbug_helper C-30 , C-35 L LEDs, showing state of syst.
Un punto importante, dopo l’acquisto del dispositivo (o anche prima di acquisto) è quello di leggere il manuale. Dobbiamo farlo per diversi motivi semplici:
Se non hai ancora comprato il Q-Logic IB6054601-00 D è un buon momento per familiarizzare con i dati di base del prodotto. Prime consultare le pagine iniziali del manuale d’uso, che si trova al di sopra. Dovresti trovare lì i dati tecnici più importanti del Q-Logic IB6054601-00 D - in questo modo è possibile verificare se l’apparecchio soddisfa le tue esigenze. Esplorando le pagine segenti del manuali d’uso Q-Logic IB6054601-00 D imparerai tutte le caratteristiche del prodotto e le informazioni sul suo funzionamento. Le informazioni sul Q-Logic IB6054601-00 D ti aiuteranno sicuramente a prendere una decisione relativa all’acquisto.
In una situazione in cui hai già il Q-Logic IB6054601-00 D, ma non hai ancora letto il manuale d’uso, dovresti farlo per le ragioni sopra descritte. Saprai quindi se hai correttamente usato le funzioni disponibili, e se hai commesso errori che possono ridurre la durata di vita del Q-Logic IB6054601-00 D.
Tuttavia, uno dei ruoli più importanti per l’utente svolti dal manuale d’uso è quello di aiutare a risolvere i problemi con il Q-Logic IB6054601-00 D. Quasi sempre, ci troverai Troubleshooting, cioè i guasti più frequenti e malfunzionamenti del dispositivo Q-Logic IB6054601-00 D insieme con le istruzioni su come risolverli. Anche se non si riesci a risolvere il problema, il manuale d’uso ti mostrerà il percorso di ulteriori procedimenti – il contatto con il centro servizio clienti o il servizio più vicino.
