RAJA Performance Suite Output¶
This section describes the contents of output files generated by the Suite.
When the Suite is run, several output files are generated that contain data
describing the run. By default the files be placed in the directory where the
executable is invoked, and the file names will contain the prefix
RAJAPerf- and a string indicating the contents.
Note
You can provide command-line options to place the output files in a different directory and/or give them a different file name prefix. Such options and syntax are described in the Suite help output:
$ ./bin/raja-perf.exe -h
Currently, there are five output files generated that provide information described below. All output files are plain text files. Other than the checksum file, all file contents are in ‘csv’ format for easy processing by common tools for generating plots, etc.
Timing – execution time (sec.) of each loop kernel and variant run.
Checksum – checksum values for each loop kernel and variant run to ensure that they are producing correct results. Typically, a checksum difference of ~1e-10 or less indicates that results generated by a kernel variant match a reference variant.
Speedup – run time speedup of each kernel variant with respect to a reference variant. The reference variant can be set with a command-line option. If not specified, the first variant of a kernel that is run will be used as the reference. The reference variant used is noted in the file.
Figure of Merit (FOM) – basic statistics about speedup of a RAJA variant vs. baseline for each programming model run. When the execution time of a RAJA variant differs from the corresponding baseline variant by more than some tolerance, this is noted in the file with
OVER_TOL. The default tolerance is 10% and can be changed via a command-line option.Kernel – basic information about each kernel that is run, which is the same for each variant of a kernel that is run. Kernel information is described in more detail in the next section.
Kernel information output¶
Information about kernels run when the RAJA Performance Suite executes is
placed in the RAJAPerf-kernels.csv file (unless the file prefix name is
changed by the user). This information is reported for rank zero when running
with multiple MPI processes. When running with more than one MPI rank,
information can be easily aggregated across all ranks if needed. For example,
the total aggregate problem size is the number of ranks times the problem size
shown in the kernel information.
Information reported in the file for each kernel is:
Name – full kernel name, format is group name followed by the kernel name, separated by an underscore.
Feature – RAJA features exercised in RAJA variants of kernel.
Problem size – Size of the problem represented by a kernel. Please see section below.
Reps – Number of times a kernel runs in a single pass through the Suite.
Iterations/rep – Sum of sizes of all parallel iteration spaces for all loops run in a single kernel execution.
Kernels/rep – total number of loop structures run (or GPU kernels launched) in each kernel repetition.
Bytes/rep – Total number of bytes read from and written to memory for each repetition of kernel.
FLOPs/rep – Total number of floating point operations executed for each repetition of kernel. Currently, we count arithmetic operations (+, -, *, /) and functions, such as exp, sin, etc. as one FLOP. We do not currently count operations like abs and comparisons (<, >, etc.) in the FLOP count. So these numbers are rough estimates. For actual FLOP counts, a performance analysis tool should be used.
Notes about problem size¶
This section describes how the Suite calculates problem sizes and the rationale behind it.
The concept of problem size is subjective and can be interpreted differently depending on the kernel structure and what one is trying to measure. For example, problem size could refer to the amount of data needed to be stored in memory to run the problem, or it could refer to the amount of parallel work that is possible, etc.
The Suite uses three notions of problem size for each kernel: default, target, and actual. Default is the problem size defined for a kernel and the size that is run if no run time options are provided to run a different size. Target is the desired problem size to run based on default settings and alterations to those if input is provided to change the default. Actual is the problem size that is run based on how each kernel calculates it based on defaults and run time input.
We employ an admittedly loose definition of problem size for each kernel, which depends on the kernel structure. Of all loop structures (e.g., single loop, nested loops, etc.) that are run for a kernel (note that some kernels run multiple loops, possibly with different sizes or loop structures), problem size refers to the size of the data set required to generate the kernel result. The interpretation of this and the definition of problem size for each kernel in the suite is determined by the kernel developer and team discussion.
Here are a few examples to give a better sense of how we determine problem size for various kernels in the Suite.
Vector addition:
for (int i = 0; i < 0; i < N; ++i) {
c[i] = a[i] + b[i];
}
The problem size for this kernel is N, the loop length. Note that this happens to match the size of the vectors a, b, c and the total amount of parallel work in the kernel. This is common for simple, data parallel kernels.
Matrix-vector multiplication:
for (int r = 0; r < N_r; ++r) {
b[r] = 0;
for (int c = 0; c < N_c; ++c) {
b[r] += A[r][c] + x[c];
}
}
The problem size if N_r * N_c, the size of the matrix. Note that this matches the total size of the problem iteration space, but the total amount of parallel work is N_r, the number of rows in the matrix and the length of the vector b.
Matrix-matrix multiplication:
for (int i = 0; i < N_i; ++i) {
for (int j = 0; j < N_j; ++j) {
A[i][j] = 0;
for (int k = 0; k < N_k; ++k) {
A[i][j] += B[i][k] * C[k][j];
}
}
}
Here, we are multiplying matrix B (N_i x N_k) and matrix C (N_k x N_j) and storing the result in matrix A (N_i X N_j). Problem size could be chosen to be the maximum number of entries in matrix B or C. We choose the size of matrix A (N_i * N_j), which is more closely aligned with the number of independent operations (i.e., the amount of parallel work) in the kernels.
Caliper output files¶
If you’ve built RAJAPerf with Caliper support turned on, then in addition to the outputs mentioned above, we also save a .cali file for each variant & tuning run, such as: Base_OpenMP-default.cali, Lambda_OpenMP-default.cali, Base_CUDA-block_128.cali, etc.
Also, by using the –variants and –tunings flag when running, you can specify which variant/tunings to run.
There are several techniques to display the Caliper trees (Timing Hierarchy)
2: Caliper’s Python module caliperreader:
import os
import caliperreader as cr
DATA_DIR = os.getenv('HOME')+"/data/default_problem_size/gcc"
os.chdir(DATA_DIR)
r = cr.CaliperReader()
r.read("RAJA_Seq.cali")
metric = 'avg#inclusive#sum#time.duration'
for rec in r.records:
path = rec['path'] if 'path' in rec else 'UNKNOWN'
time = rec[metric] if metric in rec else '0'
if not 'UNKNOWN' in path:
if (isinstance(path, list)):
path = "/".join(path)
print("{0}: {1}".format(path, time))
You can add a couple of lines to view the metadata keys captured by Caliper/Adiak:
for g in r.globals:
print(g)
You can also add a line to display metadata value in the dictionary r.globals
For example print out the OpenMP Max Threads value recorded at runtime:
print('OMP Max Threads: ' + r.globals['omp_max_threads'])`
or the variant represented in this file:
print('Variant: ' + r.globals['variant'])
Note
The script above was written using caliper-reader 0.3.0, but is fairly generic. Other version usage notes may be found at the link below
3: Using the Hatchet Python module for single files:
import hatchet as ht
DATA_DIR = os.getenv('HOME')+"/data/default_problem_size/gcc"
os.chdir(DATA_DIR)
gf1 = ht.GraphFrame.from_caliperreader("RAJA_Seq.cali")
print(gf1.tree())
4: Using the Thicket Python module for multiple files:
import thicket as th
DATA_DIR = os.getenv('HOME')+"/data/default_problem_size/gcc"
os.chdir(DATA_DIR)
th1 = th.Thicket.from_caliperreader(["RAJA_Seq-default.cali", "Base_Seq-default.cali", "Base_CUDA-block_128", "Base_CUDA-block_256"])
print(th1.tree())