Document Outline

• HP XC System Software User's Guide
  • Intended Audience
  • Document Organization
  • HP XC Information
    • For More Information
    • Supplementary Information
    • Manpages
  • Related Information
  • Typographical Conventions
  • HP Encourages Your Comments
  • 1 Overview of the User Environment
    • 1.1 System Architecture
      • 1.1.1 Operating System
      • 1.1.2 Node Specialization
      • 1.1.3 Storage and I/O
      • 1.1.4 File System
      • 1.1.5 System Interconnect Network
      • 1.1.6 Network Address Translation (NAT)
    • 1.2 User Environment
      • 1.2.1 LVS
      • 1.2.2 Modules
      • 1.2.3 Commands
        • 1.2.3.1 Linux Commands
        • 1.2.3.2 LSF Commands
        • 1.2.3.3 SLURM Commands
        • 1.2.3.4 HP-MPI Commands
        • 1.2.3.5 Modules Commands
    • 1.3 Application Development Environment
      • 1.3.1 Parallel Applications
      • 1.3.2 Serial Applications
    • 1.4 Run-Time Environment
      • 1.4.1 SLURM
      • 1.4.2 Load Sharing Facility ( LSF-HPC )
      • 1.4.3 How LSF-HPC and SLURM Interact
      • 1.4.4 HP-MPI
    • 1.5 Components, Tools, Compilers, Libraries, and Debuggers
  • 2 Using the System
    • 2.1 Logging in to the System
      • 2.1.1 LVS Login Routing
      • 2.1.2 Using ssh to Log In
    • 2.2 Configuring Your Environment with Modulefiles
      • 2.2.1 Notes on Modulefiles
      • 2.2.2 Supplied Modulefiles
      • 2.2.3 Modulefiles Automatically Loaded on the System
      • 2.2.4 Viewing Available Modulefiles
      • 2.2.5 Viewing Loaded Modulefiles
      • 2.2.6 Loading a Modulefile
        • 2.2.6.1 Loading a Modulefile for the Current Session
        • 2.2.6.2 Automatically Loading a Modulefile at Login
      • 2.2.7 Unloading a Modulefile
      • 2.2.8 Modulefile Conflicts
      • Note
      • 2.2.9 Creating a Modulefile
      • 2.2.10 Viewing Modulefile-Specific Help
    • 2.3 Launching and Managing Jobs Quick Start
      • 2.3.1 Introduction
      • 2.3.2 Getting Information About Queues
      • 2.3.3 Getting Information About Resources
      • 2.3.4 Getting Information About the System's Partitions
      • 2.3.5 Launching Jobs
        • 2.3.5.1 Submitting a Serial Job
          • Example 2-1: Submitting a Serial Job
        • 2.3.5.2 Submitting a Non-MPI Parallel Job
          • Example 2-2: Submitting a Non-MPI Parallel Job
          • Example 2-3: Submitting a Non-MPI Parallel Job to Run One Task p
        • 2.3.5.3 Submitting an MPI Job
          • Example 2-4: Running an MPI Job with LSF
          • Example 2-5: Running an MPI Job with LSF Using the External Sche
        • 2.3.5.4 Submitting a Batch Job or Job Script
          • Example 2-6: Submitting a Job Script
      • 2.3.6 Getting Information About Your Jobs
      • 2.3.7 Stopping and Suspending Jobs
      • 2.3.8 Resuming Suspended Jobs
    • 2.4 Performing Other Common User Tasks
      • 2.4.1 Determining the LSF Cluster Name and LSF Execution Host
      • 2.4.2 Installing Third-Party Software
    • 2.5 Getting System Help and Information
  • 3 Developing Applications
    • 3.1 Overview
    • 3.2 Using Compilers
      • 3.2.1 Standard Linux Compilers
      • 3.2.2 Intel Compilers
      • 3.2.3 PGI Compilers
      • 3.2.4 Pathscale Compilers
      • 3.2.5 MPI Compiler
    • 3.3 Checking Nodes and Partitions Before Running Jobs
    • 3.4 Interrupting a Job
    • 3.5 Setting Debugging Options
    • 3.6 Developing Serial Applications
      • 3.6.1 Serial Application Build Environment
        • 3.6.1.1 Using MLIB in Serial Applications
      • 3.6.2 Building Serial Applications
        • 3.6.2.1 Compiling and Linking Serial Applications
    • 3.7 Developing Parallel Applications
      • 3.7.1 Parallel Application Build Environment
        • 3.7.1.1 Modulefiles
        • 3.7.1.2 HP-MPI
        • 3.7.1.3 OpenMP
        • 3.7.1.4 Pthreads
        • 3.7.1.5 Quadrics SHMEM
        • 3.7.1.6 MLIB Math Library
        • 3.7.1.7 MPI Library
        • 3.7.1.8 Intel Fortran and C/C++Compilers
        • 3.7.1.9 PGI Fortran and C/C++ Compilers
        • 3.7.1.10 GNU C and C++ Compilers
        • 3.7.1.11 GNU Parallel Make
        • 3.7.1.12 MKL Library
        • 3.7.1.13 ACML Library
        • 3.7.1.14 Other Libraries
        • 3.7.1.15 Reserved Symbols and Names
      • 3.7.2 Building Parallel Applications
        • 3.7.2.1 Compiling and Linking Non-MPI Applications
        • 3.7.2.2 Compiling and Linking HP-MPI Applications
        • 3.7.2.3 Examples of Compiling and Linking HP-MPI Applications
    • 3.8 Developing Libraries
      • 3.8.1 Designing Libraries for XC4000
        • Example 3-1: Directory Structure
        • Example 3-2: Recommended Directory Structure
    • 3.9 Advanced Topics
      • 3.9.1 Using the GNU Parallel Make Capability
        • 3.9.1.1 Example Procedure 1
        • 3.9.1.2 Example Procedure 2
        • 3.9.1.3 Example Procedure 3
      • 3.9.2 Local Disks on Compute Nodes
      • 3.9.3 I/O Performance Considerations
        • 3.9.3.1 Shared File View
        • 3.9.3.2 Private File View
      • 3.9.4 Communication Between Nodes
  • 4 Debugging Applications
    • 4.1 Debugging Serial Applications
    • 4.2 Debugging Parallel Applications
    • Note
      • 4.2.1 Debugging with TotalView
        • 4.2.1.1 SSH and TotalView
        • 4.2.1.2 Setting Up TotalView
        • Note:
        • 4.2.1.3 Using TotalView with SLURM
        • 4.2.1.4 Using TotalView with LSF-HPC
        • 4.2.1.5 Starting TotalView for the First Time
          • Figure 4-1: TotalView Root Window
          • Figure 4-2: TotalView Preferences Window
        • 4.2.1.6 Debugging an Application
          • Figure 4-3: TotalView Process Window Example
        • 4.2.1.7 Debugging Running Applications
          • Figure 4-4: Unattached Window
          • Figure 4-5: Attached Window
        • 4.2.1.8 Exiting TotalView
  • 5 Tuning Applications
    • 5.1 Using the Intel Trace Collector/Analyzer
      • 5.1.1 Building a Program Intel Trace Collector and HP-MPI
      • 5.1.2 Running a Program Intel Trace Collector and HP-MPI
      • 5.1.3 Visualizing Data Intel Trace Analyzer and HP-MPI
  • 6 Using SLURM
    • 6.1 Introduction
    • 6.2 SLURM Commands
    • 6.3 Accessing the SLURM Manpages
    • 6.4 Launching Jobs with the srun Command
      • Example 6-1: Simple Launch of a Serial Program
      • 6.4.1 The srun Roles and Modes
        • 6.4.1.1 srun Roles
        • 6.4.1.2 srun Modes
      • 6.4.2 srun Signal Handling
      • 6.4.3 srun Run-Mode Options
      • 6.4.4 srun Resource-Allocation Options
      • 6.4.5 srun Control Options
        • 6.4.5.1 Node Management Options
        • 6.4.5.2 Working Features Options
        • 6.4.5.3 Resource Control Options
        • 6.4.5.4 Help Options
      • 6.4.6 srun I/O Options
        • 6.4.6.1 I/O Commands
        • 6.4.6.2 I/O Redirection Alternatives
      • 6.4.7 srun Constraint Options
      • 6.4.8 srun Environment Variables
      • 6.4.9 Using srun with HP-MPI
      • 6.4.10 Using srun with LSF
    • 6.5 Monitoring Jobs with the squeue Command
      • Example 6-2: Displaying Queued Jobs by Their JobIDs
      • Example 6-3: Reporting on Failed Jobs in the Queue
    • 6.6 Killing Jobs with the scancel Command
      • Example 6-4: Killing a Job by Its JobID
      • Example 6-5: Cancelling All Pending Jobs
      • Example 6-6: Sending a Signal to a Job
    • 6.7 Getting System Information with the sinfo Command
      • Example 6-7: Using the sinfo Command (No Options)
      • Example 6-8: Reporting Reasons for Downed, Drained, and Draining
    • 6.8 Job Accounting
    • 6.9 Fault Tolerance
    • 6.10 Security
  • 7 Using LSF
    • 7.1 Introduction to LSF in the HP XC Environment
      • 7.1.1 Overview of LSF
      • 7.1.2 Topology Support
      • 7.1.3 Notes on LSF-HPC
        • Example 7-1: Comparison of Queues and the Configuration of the J
      • 7.1.4 How LSF and SLURM Launch and Manage a Job
        • Figure 7-1: How LSF-HPC and SLURM Launch and Manage a Job
      • 7.1.5 Differences Between LSF on HP XC and Standard LSF
      • 7.1.6 Notes About Using LSF in the HP XC Environment
        • 7.1.6.1 Job Startup and Job Control
        • 7.1.6.2 Preemption Support
    • 7.2 Determining Execution Host
    • 7.3 Determining Available System Resources
      • 7.3.1 Getting Status of LSF
      • 7.3.2 Getting Information About LSF-HPC Execution Host Node
      • 7.3.3 Getting Host Load Information
      • 7.3.4 Checking LSF System Queues
      • 7.3.5 Getting Information About the lsf Partition
    • 7.4 Submitting Jobs
      • 7.4.1 Summary of the LSF bsub Command Format
      • 7.4.2 LSF-SLURM External Scheduler
        • Example 7-2: Using the External Scheduler to Submit a Job to Run
        • Example 7-3: Using the External Scheduler to Submit a Job to Run
        • Example 7-4: Using the External Scheduler to Submit a Job That E
      • 7.4.3 Submitting a Serial Job
        • Example 7-5: Submitting an Interactive Serial Job
      • 7.4.4 Submitting a Job in Parallel
      • 7.4.5 Submitting an HP-MPI Job
        • Example 7-6: Submitting an HP-MPI Job
        • Example 7-7: Submitting an HP-MPI Job with a Specific Topology R
      • 7.4.6 Submitting a Batch Job or Job Script
        • 7.4.6.1 Examples
          • Example 7-8: Submitting a Batch Job Script
          • Example 7-9: Submitting a Batch Script with a Specific Topologic
          • Example 7-10: Submitting a Batch Job Script that uses a Subset o
          • Example 7-11: Submitting a Batch job Script That Uses the srun -
          • Example 7-12: Useful Environment Variables Available in a Batch
      • 7.4.7 Submitting a Job from a Non- HP XC Host
    • 7.5 Getting Information About Jobs
      • 7.5.1 Getting Job Allocation Information
        • 7.5.1.1 Job Allocation Information for a Running Job
        • 7.5.1.2 Job Allocation Information for a Finished Job
      • 7.5.2 Checking Status of a Job
        • Example 7-13: Using the bjobs Command (Short Output)
        • Example 7-14: Using the bjobs Command (Long Output)
      • 7.5.3 Viewing a Job's Historical Information
        • Example 7-15: Using the bhist Command (Short Output)
        • Example 7-16: Using the bhist Command (Long Output)
    • 7.6 Working Interactively Within an LSF-HPC Allocation
      • 7.6.1 Submitting an Interactive Job to Launch the xterm Program
        • Example 7-17: View Your Environment
        • Example 7-18: View Your Allocation in SLURM
        • Example 7-19: View Your Running Job in LSF
        • Example 7-20: View Job Details in LSF
        • Example 7-21: Running Jobs from an xterm Window
      • 7.6.2 Submitting an Interactive Job to Launch a Shell
        • Example 7-22: Submitting an Interactive Shell Program
        • Example 7-23: Submitting an Interactive Shell Program on the LSF
    • 7.7 LSF Equivalents of SLURM srun Options
  • 8 Using HP-MPI
    • 8.1 Overview
    • 8.2 HP-MPI Directory Structure
    • 8.3 Compiling and Running Applications
      • 8.3.1 Setting Environment Variables
      • 8.3.2 Building and Running an Example Application
        • 8.3.2.1 Example Application hello_world
        • 8.3.2.2 Building and Running hello_world
      • 8.3.3 Using srun with HP-MPI
        • 8.3.3.1 Launching MPI Jobs
        • 8.3.3.2 Creating Subshells and Launching Jobsteps
        • 8.3.3.3 System Interconnect Selection
          • Example 8-1: Performing System Interconnect Selection
          • Example 8-2: Using TCP/IP over Gigabit Ethernet
          • Example 8-3: Using TCP/IP over Elan4
      • 8.3.4 Using LSF and HP-MPI
        • Example 8-4: Allocating and Attaching Processors
        • Example 8-5: Allocating 12 Processors on 6 Nodes
      • 8.3.5 MPI Versioning
    • 8.4 System Interconnect Support
      • 8.4.1 HP-MPI Performance on HP XC with Multiple System Interconn
      • 8.4.2 Global Environment Variable Settings on the mpirun Command
    • 8.5 32 Bit Builds on XC4000
    • 8.6 Truncated Messages
    • 8.7 Allowing Windows to Use Exclusive Locks
    • 8.8 The mpirun Command Options
    • 8.9 Environment Variables
      • 8.9.1 MPIRUN_OPTIONS
      • 8.9.2 MPIRUN_SYSTEM_OPTIONS
      • 8.9.3 MPI_IC_ORDER
      • 8.9.4 MPI_PHYSICAL_MEMORY
      • 8.9.5 MPI_PIN_PERCENTAGE
      • 8.9.6 MPI_PAGE_ALIGN_MEM
      • 8.9.7 MPI_MAX_WINDOW
      • 8.9.8 MPI_ELANLOCK
      • 8.9.9 MPI_USE_LIBELAN
      • 8.9.10 MPI_USE_LIBELAN_SUB
    • 8.10 MPICH Object Compatibility
    • 8.11 HP-MPI Documentation and Manpages
    • 8.12 Additional Information, Known Problems, and Work-arounds
  • 9 Using HP MLIB
    • 9.1 Overview
      • 9.1.1 Intel Compiler Notes
      • 9.1.2 MLIB and Module Files
    • 9.2 HP MLIB for the HP XC6000 Platform
      • 9.2.1 Platform Support
      • 9.2.2 Library Support
      • 9.2.3 MPI Parallelism
      • 9.2.4 Modulefiles and MLIB
      • 9.2.5 Using Intel Compilers with HP MLIB
      • 9.2.6 Compiling and Linking
        • 9.2.6.1 Linking VECLIB
        • 9.2.6.2 Linking LAPACK
        • 9.2.6.3 Linking ScaLAPACK
        • 9.2.6.4 Linking SuperLU_DIST
      • 9.2.7 Licensing
      • 9.2.8 MLIB Manpages
    • 9.3 HP MLIB for the HP XC4000 Platform
      • 9.3.1 Platform Support
      • 9.3.2 Library Support
      • 9.3.3 MPI Parallelism
      • 9.3.4 Modulefiles and MLIB
      • 9.3.5 Compiling and Linking
        • 9.3.5.1 Linking VECLIB
        • 9.3.5.2 Linking LAPACK
        • 9.3.5.3 Linking ScaLAPACK
        • 9.3.5.4 Linking SuperLU_DIST
      • 9.3.6 Licensing
      • 9.3.7 MLIB Manpages
  • 10 Advanced Topics
    • 10.1 Enabling Remote Execution with OpenSSH
    • 10.2 Running an X Terminal Session from a Remote Node
  • A Examples
    • A.1 Building and Running a Serial Application
    • A.2 Launching a Serial Interactive Shell Through LSF
    • A.3 Running LSF Jobs with a SLURM Allocation Request
      • A.3.1 Example 1. Two Processors on Any Two Nodes
      • A.3.2 Example 2. Four Processors on Two Specific Nodes
    • A.4 Launching a Parallel Interactive Shell Through LSF
    • A.5 Submitting a Simple Job Script with LSF
    • A.6 Submitting an Interactive Job with LSF
    • A.7 Submitting an HP-MPI Job with LSF
    • A.8 Using a Resource Requirements String in an LSF Command
  • Glossary
    • A
      • Administrative Network
      • admin branch
    • B
      • base image
      • branch switch
    • C
      • cluster
      • cluster alias
      • Console Branch
    • D
      • DHCP
      • Dynamic Host Control Protocol
    • E
      • EFI
      • extensible firmware interface
      • external network node
    • F
      • fairshare
      • FCFS
      • first come first served
    • G
      • global storage
      • golden client
      • golden image
      • golden master
    • H
      • head node
      • host name
    • I
      • iLO
      • image server
      • Integrated Lights Out
      • interconnect
      • interconnect
      • Internet address
      • I/O node
      • IPMI
    • L
      • Linux Virtual Server
      • load file
      • Load Sharing Facility
      • local storage
      • LSF
      • LSF execution host
      • LSF master host
      • LVS
    • M
      • Management Processor
      • master host
      • module
      • modulefile
      • MP
      • MPI
      • MySQL
    • N
      • NAT
      • Network Address Translation
      • Network Information Services
      • NIS
    • P
      • parallel application
      • PXE
    • R
      • resource manager role
      • role
      • Root Administration Switch
      • root node
      • RPM
    • S
      • serial application
      • SMP
      • ssh
      • symmetric multiprocessing
  • Index
• tables
  • Table 2-1: Supplied Modulefiles
  • Table 3-1: Intel Compiler Commands
  • Table 3-2: PGI Compiler Commands
  • Table 6-1: SLURM Commands
  • Table 7-1: Output Provided by the bhist Command
  • Table 7-2: LSF Equivalents of SLURM srun Options
  • Table 8-1: Organization of the /opt/hpmpi Directory
  • Table 8-2: HP-MPI Manpage Categories