Chunking metabot design notes (”reverse engineered” the metabot to figure out what can be assigned to MC)

Overview of metabot’s execution steps:

1. metabot begins execution at any time (before, during, or after the writing app’s execution)
2. metabot loops until it finds a file with “done” as the extension, indicating that app is finished writing
3. counts the number of files that have .dat extension
4. reads files in ascending order of timesteps (nodes aren’t taken into account, just 1 node essentially)
5. reads each file a # of bytes at a time (# of bytes defined in metabot)
6. writes the blocks out to single file (name defined in metabot)
7. metabot terminates after all input files read and data written to outfile

What can be pulled out from metabot to metabot controller:

1. location/name of input files
2. name/location of output file(s)
3. checking for app’s completion
4. MC should know # of files total, # of timesteps or nodes
5. size of data bytes read from input files at a time (initially in metabot)
6. ideally, format should be conveyed to metabot from MC (currently in types.h)

Metabot can basically have generic functions to read and write files, get params from MC, be launched by MC, should it tell MC when it has completed?

Nitty-gritty implementation details of metabot:

1. libraries required: libffs, libfm
2. include: ffs.h, types.h

By function:

 1. main.c:
      usage: "metabot run# #of_nodes metabot_rank"
      opens a fixed directory, defined by global DATADIR
      if directory is opened:
        char buffer 'data' is allocated with size SIZE (global, defined)
        calls dir_reader with dir descriptor dp

 2. dir_reader(DIR *dp):
      reads directory in line 195: dirp = readdir(dp);
            ( struct dirent *dirp )

      condition to end the program in lines 197-***:
           while ((dirp != NULL) && !done), done is set in
              check_for_done(dirp->d_name)
           if dirp isn't null after it reads another entry, it sleeps and
              waits
           for a new file to appear that may be the last file, whose
              name indicates

      falls out of loop once last written file is found:
        starts timer:
           rewinds pointer to beginning of directory
           countFiles(dirp, dp); counts # of files
           [Why are the above 2 instructions inside of the timer??? dumb]
           loops while not done:  (finally meat and potatos)
              rewind pointer again
              [currTimeStep: global int, initialized to 0,incremented by 50]
              while dirp = readdir(dp) != NULL:
                 if parse_file_name(dirp->d_name) is true:
        end timer

 3. check_for_done(char *filename):
      parses file name to check the extension; if ext = "done", return 1,
         else 0

 4. parse_file_name(char *filename):
      parses file name, checks for correctness wrt type of file,
         then checks 'tmp'
      (field to see if it equals currTimeStep;
       if yes, increment nodecnt and return 1, else return 0)