Chunking metabot design notes
("reverse engineered" the metabot to figure out what can be assigned to
MC)

Overview of metabot's execution steps:
  - metabot begins execution at any time (before, during, or after the writing app's execution)
  - metabot loops until it finds a file with  "done" as the extension, indicating that app is finished writing
  - counts the number of files that have .dat extension
  - reads files in ascending order of timesteps (nodes aren't taken into account, just 1 node essentially)
  - reads each file a # of bytes at a time (# of bytes defined in metabot)
  - writes the blocks out to single file (name defined in metabot)
  - metabot terminates after all input files read and data written to outfile


What can be pulled out from metabot to metabot controller:
   - location/name of input files
   - name/location of output file(s)
   - checking for app's completion
   - MC should know # of files total, # of timesteps or nodes
   - size of data bytes read from input files at a time (initially in metabot)
   - ideally, format should be conveyed to metabot from MC (currently in types.h)

Metabot can basically have generic functions to read and write files, get
params from MC, be launched by MC, should it tell MC when it has completed?


Nitty-gritty implementation details of metabot:
   - libraries required: libffs, libfm
   - include: ffs.h, types.h


By function:
   1. main.c:
        usage: "metabot run# #of_nodes metabot_rank"
        opens a fixed directory, defined by global DATADIR
        if directory is opened:
          char buffer 'data' is allocated with size SIZE (global, defined)
          calls dir_reader with dir descriptor dp

   2. dir_reader(DIR *dp):
        reads directory in line 195: dirp = readdir(dp);
              ( struct dirent *dirp )

        condition to end the program in lines 197-***:
             while ((dirp != NULL) && !done), done is set in
                check_for_done(dirp->d_name)
             if dirp isn't null after it reads another entry, it sleeps and
                waits
             for a new file to appear that may be the last file, whose
                name indicates

        falls out of loop once last written file is found:
          starts timer:
             rewinds pointer to beginning of directory
             countFiles(dirp, dp); counts # of files
             [Why are the above 2 instructions inside of the timer??? dumb]
             loops while not done:  (finally meat and potatos)
                rewind pointer again
                [currTimeStep: global int, initialized to 0,incremented by 50]
                while dirp = readdir(dp) != NULL:
                   if parse_file_name(dirp->d_name) is true:
          end timer

   3. check_for_done(char *filename):
        parses file name to check the extension; if ext = "done", return 1,
           else 0

   4. parse_file_name(char *filename):
        parses file name, checks for correctness wrt type of file,
           then checks 'tmp'
        (field to see if it equals currTimeStep;
         if yes, increment nodecnt and return 1, else return 0)