Alex Szalay sold me on the concept of Amdahl's number, which is a ratio between the number of operations and the amount of I/O. For example, an Amdahl number of 1.0 corresponds to a machine that can do 1 bit of I/O / sec per 1 operation / second. So, the ratio is a comparison of rates that represents the I/O balance of a machine and, thus, is scale-invariant in some sense.
I have touted that the petabyte capacity database here at Hopkins has a high Amdahl number as if it is meritorious. The number is about 0.6, which is an order of magnitude higher than typical HPC systems. This is great, but leads to questions along the line:
Why do I care about the Amdahl number?
A nice discussion of this issue was covered by Oliver Ratzenberger in the XtremeLargeMPP blog.
This question has become much easier to answer based on instrumenting applications. You can measure the Amdahl number of an application, which reflects the balance of how much I/O it does relative to the computation. The in-depth analysis and scans done by scientists on the Sloan Digital Sky Survey and the JHU Turbulence Database have very high Amdahl numbers (>0.5). Most computational and multi-scale simulations do a small amount of I/O, have low Amdahl numbers (<0.1) and, thus, run just fine on most parallel supercomputers.
If you match the Amdahl number of the machine and application there should be no wasted resources. This is true only in a time averaged sense; I/O and computation do not always happen concurrently and different portions applications have different ratios. It's also worth noting that the Amdahl number of an application or workload is independent of the architecture on which it is run.
However, I want to put forth that supercomputing applications have co-evolved with HPC architectures and that the mix of I/O and computation represents the set of applications that will run well on the hardware deployed at HPC facilities, rather than the natural balance of I/O and computation. For example, it is typical that in a multi-scale simulation, it might take 5 times as long to store a time-step as to compute (solve the equations) a timestep.
Discussion