Differences

This shows you the differences between the selected revision and the current version of the page.

metabot_microbenchmarks 2008/12/11 05:19 metabot_microbenchmarks 2008/12/11 09:13 current
Line 1: Line 1:
-The government says the two were "conspiring to obtain personal financial benefits" for the Illinois governor by leveraging his sole authority to appoint a U.S. senator to replace Obama., http://freenudity.vidilife.com free nudity, 817, http://celebnudityfree.vidilife.com celeb nudity free,  8(((, http://www.mlive.com/forums/profile.ssf?nickname=trishstrat0 trish stratus nudity,  113745, http://www.mlive.com/forums/profile.ssf?nickname=strippingo6 stripping on webcam full nudity, 7839, http://springbreaknudity.vidilife.com spring break nudity,  uxxdk, http://rudeshortjokes3.vidilife.com rude short jokes,  554, http://freenudityshockwaveg.vidilife.com free nudity shockwave games,  5167, http://yogapositionsnude0.vidilife.com yoga positions nude, =-DDD, http://www.mlive.com/forums/profile.ssf?nickname=helenmirre6 helen mirren nudity,  pnbqi, http://sablenudity.vidilife.com sable nudity,  968, http://britneyspearsnudity.vidilife.com britney spears nudity, ruixu, http://databaseofnudityinfi.vidilife.com database of nudity in films,  nxwdm, http://www.mlive.com/forums/profile.ssf?nickname=familynudi3 family nudity,  mli, http://rudeonelinejokes2.vidilife.com rude one line jokes, 922327, +===== Benchmark 1:  Metabots for Metadata Creation ===== 
 + 
 + 
 +==== Motivation ==== 
 + 
 +This experiment seeks to demonstrate how assigning file system metadata creation to out-of-band metabots can improve in-band performance of the application. In situations where applications have to create large file hierarchies, which are accessed at a later point of time, there is considerable overhead for creating file system metadata in-band. In many of those cases, it is reasonable for the application programmer to know the structure of this hierarchy and possibly the number and sizes of these files. When file sizes cannot be predetermined, it is usually possible to parse the contents of a data stream and demarcate the logical end of one file and the beginning of another. In those instances, our proposed framework would have individual processes in the parallel application write to LWFS containers which may be created ahead of time. Out-of-band metabots will then go through these containers based on application programmer supplied information about the hierarchy, and create the requisite file system metadata and store it for future access. This is expected to considerably enhance application write performance. 
 + 
 + 
 +==== Experiment Design ==== 
 + 
 +We designed our micro-benchmark similar to the [[http://www.llnl.gov/icc/lc/siop/downloads/download.html|fdtree]] benchmark which creates a file hierarchy of user specified depth, with each directory containing user specified number of files each having a user defined file size. However, since our implementation has to make calls into the LWFS client libraries, it was implemented as a C program instead of a shell script. In our experiments, we compare the performances of file hierarchy creation which use the following approaches: 
 + 
 + - Making in-band file system metadata using LWFS 
 +  - Writing just the raw data in-band using LWFS and generating the filesystem metadata out-of-band 
 +  - Making the file hierarchy using NFS (Synchronized) 
 + 
 +We designed three series of experiments. In the first, we gradually increase the breadth and depth of the hierarchy (Test **A**). In the second, we restrict the depth to just one and gradually increase the number of files (Test **B**). In B, all files have the fixed size of 4KB. In the third, we have only one file with gradually increasing file sizes (Test **C**). 
 + 
 + 
 +==== Experiment Setup ==== 
 + 
 +We ran our experiment on 4 nodes of a 52 node [Machine and interconnect description].  
 + 
 +==== Measurements and Analysis ==== 
 + 
 + 
 +==== Critical Items and Time Line ==== 
 + 
 + 
 + * Modifying fdtree to write to LWFS containers   
 +  * The Metadata service for storing Metabot generated metadata (file system metadata) 
 +Time: 3-4 days 
 + 
 +  * A UFS-LWFS interface (built using FUSE?) to validate file system Metadata. 
 +Time: (3-4 days)  
 + 
 + * Measurements and Graph Generation 
 +Time: 3-4 days 
 + 
 + 
 + ===== Benchmark 2: LWFS performance ===== 
 + 
 + 
 + 
 + 
 + 
 + 
 +==== Motivation ==== 
 + 
 +This series of experiments are designed to demonstrate LWFS performance for MPI applications in comparison to other parallel file systems. It involves running a series of standard parallel I/O benchmarks: 
 +  * [[http://www.hlrs.de/organization/par/services/models/mpi/b_eff_io/|b_eff_io]] for MPI-IO performance 
 + * [[http://flash.uchicago.edu/~zingale/flash_benchmark_io/|Flash I/O]] for HDF5 performance over MPI-IO 
 + 
 + 
 + 
 + 
 +==== Experiment Setup ==== 
 + 
 +In the control setup, we have an instance of the benchmarks on Lustre. In the experimental setup, LWFS will be used instead. 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 +==== Critical Items and Time Line ==== 
 + 
 + * ROMIO-ADIO/LWFS interface 
 +Time: 7-10 days 
 + 
 +    * Measurements and Graph Generation 
 +Time: 3-4 days  
 + 
 +===== Progress Status ===== 
 + 
 +We are currently in the process of getting LWFS running on one of the machines here. We expect development to start on the ADIO/LWFS interface as soon as possible. 
 + 
 + 
 +---- 
 + 
 +== Errata == 
 + 
 +Benefits Metabots provide: \\ 
 +  - Offline/Out-of-band Processing   
 +  - Closer/Faster Access to datastore 
 + - Specialized Knowledge of interface to datastore (in low level, not high level specification) 
 + 
 +Ideas 
 + 
 +  - Minimal Penalty for extracting metabot functionality from application 
 +  - A more efficient translation to parallel transformations 
 +  - Near Disk Access to "Meta-tized" (metadata annotated) data 
 +  - Translating between two formats netcdf -> hdf 
 +  - Creating filename containers 
 + 
 +To show benefit 1 above:  
 +T(MB) + T(App-MB) = T(App) 
 +This will demonstrate that we can gain performance benefits from metabots. 
 + 
 +Most practical idea: look at during an app on LWFS using a metabot to make filesystem that app expects from a raw container. 
Back to top
metabot_microbenchmarks.txt · Last modified: 2008/12/11 09:13 by pmw
Index Recent changes
Show page
Old revisions Login