Los Alamos National Laboratory
Sitemap  |   Lab Home  |   Phone
Computer Science Research :: Software

HPC-5 Open Source Software Projects

The following is a list of free open source software projects developed by HPC-5.
Updated December 21st, 2011

MPI-FTW    MPI-IO Test (fs_test)    Rompio    LANL-Trace    LANL Checkpoint Tool   


This program is used to traverse a directory tree using MPI. Each time a new directory is located, a process is allocated to walk that particular branch. Upon completion, this program will provide directory count, file count and time to complete statistics. This program also accepts an argument that states that commands are to be run on each directory found.

The MPI-FTW Test is open sourced under LA-CC-06-077.

Release DateSourceDocumentation
1.323 August 2007mpi_ftw-1.3.tar README
1.207 April 2007mpi_ftw-1.2.tar README
1.107 March 2007mpi_ftw-1.1.tar README
1.005 February 2007mpi_ftw-1.0.tar README


This section contains examples of modified mpi_ftw that perform specific functions:

pstat is a modified mpi_ftw that purges files and reports file system statistics. This is an early version so the purging option (-f) is not recommended at this time.

Release DateSourceDocumentation
0.122 May 2008pstat.tgz PSTAT_README

MPI-IO Test (fs_test)

Although there are a host of existing file system and I/O test programs available, most are not designed with parallel I/O in mind and are not useful at the scale of the clusters at Los Alamos National Lab (LANL). LANL's MPI-IO Test was written with parallel I/O and scale in mind. The MPI-IO test is built on top of MPI's I/O calls and is used to gather timing information for reading from and writing to file(s) using a variety of I/O profiles; N processes writing to N files, N processes writing to one file, N processes sending data to M processes writing to M files, or N processes sending data to M processes to one file. These diagrams illustrate the various I/O access patterns. A data aggregation capability is available and the user can pass down MPI-IO, ROMIO and file system specific hints. The MPI-IO Test can be used for performance benchmarking and, in some cases, to diagnose problems with file systems or I/O networks.

The newer versions of MPI-IO test have been renamed to fs_test. The most current version of fs_test is currently available on Sourceforge.

The MPI-IO Test (fs_test) is open sourced under LA-CC-05-013.

Release DateSourceDocumentation
1.000.218 July 2008mpi_io_test_21.tgzREADME
1.000.2013 November 2007mpi_io_test_20.tgzREADME
1.000.0915 December 2006mpi_io_test_09.tgzREADME
1.000.082 March 2006mpi_io_test_08.tgzREADME


This code currently lives on Sourceforge


Software designed to measure MPI collective, independent, asynchonous, and POSIX file I/O performance, synthesizing how applications utilize file systems in the LANL ASC computing environment. File layouts include N processors to N files and N processors to 1 file. Write access patterns include non-strided and strided. Pure network performance measures can also be obtained for relevant MPI collective calls. Refer to the README file for a detailed description of input and output.

The MPI-FTW Test is open sourced under LA-CC-06-018.

Release DateSourceDocumentation
1.0.009 February 2007rompio.v1.0.0.tar.gzREADME


LANL-Trace is a tracing mechanism that integrates with MPI and uses either strace or ltrace to capture the I/O of a parallel application. In addition, LANL-Trace runs a simple MPI program both before and after the traced program in order to provide information about the possible skew and drift of the distributed clocks in order to allow researchers to meaningfully interpret the timestamps in each of the trace files of which one is produced by each process in the parallel application. More detailed information and instructions are available in the README file that comes with the LANL-Trace download.

LANL-Trace is open sourced under LA-CC-07-055.

Release DateSourceDocumentation
1.0.010 November 2007TarballREADME

LANL Checkpoint Research Tool

LANL Checkpoint Research Tool is a set of software tools designed to help researchers understand, explore, and analyze the unique patterns of IO within the checkpoint dumps of highly parallel applications run on LANL supercomputers. It contains:

Ninjat (formerly LANL Trace Viz) is a set of vizualization tools. One tool can be used to show an animated replay of a LANL Trace such as these. An animated tool shows which process writes or reads from which file at what offset over the lifetime of the trace. The other tool creates a static image showing the cumulative amount of I/O to each file as a function of time. Writes are shown as positive values and reads are shown as negative.
(Formerly LANL MPI-IO Viz) A vizualization tool used to analyze experimental results from MPI-IO Test or any other tool that inserts results into a mysql server. All versions of MPI-IO Test since 1.00.020 include the ability to insert their results into a MYSQL database. DBViz queries mysql servers and allows for interactive exploration of the data with flexible vizualization options.
Parallel Log-structured File System (PLFS)
PLFS is a FUSE file-system which transparently converts N-1 patterns into N-N patterns. The user stills sees a logical view of their N-1 file but the underlying file system is presented with the much easier N-N workload.
Incremental Checkpoint
A library that applications can use to monitor which blocks of their data structures have been dirtied since the last checkpoint dump. This allows checkpoint dumps to potentially complete more quickly by sending only the dirty blocks.
LANL Trace Replay
LANL Trace Replay is an MPI program that reads a LANL Trace such as these and does MPI_Barriers wherever there are MPI Collective calls in the traces and replays the file system I/O in the traces. It can be set to sleep during idle times or it can just replay the I/O as fast as possible.

LANL Checkpoint Research Tool is open sourced under LA-CC-08-104 and is made available under a BSD license.

Release DateSourceDocumentation
1.0June 2010 lct.tar.gz in tarball

LANL Trace Viz Powerpoint

LANL Trace Viz Screenshot One

LANL Trace Viz Screenshot Two

LANL MPI-IO Viz Screenshot

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy

Privacy Policy | Copyright © 1993-2006 UC | Web Contact