Hi Karsten,

Overall this looks like an excellent talk. Here is more information.

1. Between January - February 3 lucky applications will be chosen on the
ORNL machines to do really BIG science. GTC and Chimera and S3D are the top
3 of these.  For GTC, we will be studying the meoscale dynamics, structure
formation and heat and particle transport in CTEM Turbulence in DIII-D
L-Mode discharges. Ultimately we want to validate the results against
experimental fluctuation spectra comparing the BES data and look at zonal
flow patterns compared with BES images.

This science needs to be first verified, so we will run 2 different versions
of GTC, the classic version, and a shaped version that PPPL has been working
on for the last 3 years. Each version will run on 1/2 of the machine
simultaneously . We will be generating approximately 200MB per process, on
approximately 17.5K process per run (35K total). This data will be generated
every hour, and the entire simulations will take about 120 hours during a 6
day span. Since we are studying the mesoscale dynamics, we will not be able
to go out for very late times because we would need to switch the code from
a delta-f method to a full-f method, and come up with a better collision
model, and this would truly require more than a petascale computer to run
this problem. So we will just look for smaller timescales.

Numbers: 3.5TB/hour/code = 7TB/hour total * 120 hours = 840TB. Of course, we
only have 250TB to store the data. What do we do???
Another problem. If we run with MPI-IO, we will hopefully spend 10% of our
time in I/O, which is very expensive!

Possible solutions:
A. data reduction before we hit disk. We might have no choice and have to
save just one geometric region, which reduce the total amount saved by about
60 (but still keeping 2 full restart/analysis files), those reducing the
total data size to a meager 27 TB.
B. Data compression... (Just don't know how to do this right now, we are
looking at turbulence, from a monte-carlo like code, so it's not trivial to
do).

Another important point. The computer will still be very new in this stage,
and we might have a bug at first. We will have to fully monitor the
simulation, so that we can trust this!

Cost of this run will be about $1M per application. Not cheap by any
standards, so we better not run for 60 hours, and then notice there is a
problem!


Chimera numbers have changed. They now say that they will produce about
500GB every 15 minutes. This still puts them at about 280 TB for their
simulation. I'm sure they will need good diagnostics to run, and having this
automated would be a great benefit.

S3D will produce about 100GB every hour, or about 14TB of data.


Good luck!

Scott