Post-Processing

The data produced when Running Testplans, that is, all output produced by Hooks, Testcases, and the system commands these are executing, is collected and organized in a directory referred to as the testrun. In addition to collecting command output, then cijoe can also be instructed to collect artifacts produced by tools, such as log-files, process profiles, results from system-probes etc.

cijoe collects all of this, nothing is filtered out of hidden. Thus the data generated can be overwhelming when browsing through a testrun.

Thus, to produce a humanly readably representation of a testrun, then a Reporter is provided which augments the testrun with HTML formatted report.

Similarly, the Extractor, provides an interfaces to extract the key metrics of interesting.

The reporter, and the other post-processing tools are, when integrated in a CI/CD pipeline, executed after the cij_runner is done. However, one can run the tools while cij_runner is executing to get a preview of how the execution is progressing.

Note

This section lacks documentation of usage and how to extend the post-processing capabilities. Your PR improving this is most welcome.

Reporter

The cij_reporter command-line tool:

usage: cij_reporter [-h] [--output OUTPUT] [--template TEMPLATE] [--force]

cij_reporter - CIJOE Test Runner

optional arguments:
  -h, --help           show this help message and exit
  --output OUTPUT      Path to test output (default:
                       /home/safl/git/cijoe/docs)
  --template TEMPLATE  Path to report template (default:
                       /home/safl/.local/share/cijoe/templates/report.html)
  --force              Overwrite possibly existing report (default: False)

The reporter uses the Jinja templating-engine. The jinja-template given to cij_reporter by default produces a HTML-formatted report. However, you can provide your own jinja-template to produce a report in the textual format of your choosing, you could even just produce text-based summary to dump directory on the command-line.

Extractor

usage: cij_extractor [-h] [--output OUTPUT] --extractor {fio_json_read}
                     [{fio_json_read} ...]

cij_extractor - CIJOE Test data extractor

optional arguments:
  -h, --help            show this help message and exit
  --output OUTPUT       Path to test output (default:
                        /home/safl/git/cijoe/docs)
  --extractor {fio_json_read} [{fio_json_read} ...]
                        Name of, or path to, one or more extractor(s)
                        (default: None)

Analyser

usage: cij_analyser [-h] [--output OUTPUT] --preqs PREQS

cij_analyser - CIJOE Test Analyser

optional arguments:
  -h, --help       show this help message and exit
  --output OUTPUT  Path to test output (default: /home/safl/git/cijoe/docs)
  --preqs PREQS    Path to performance requirements yaml file (default: None)

Plotter

usage: cij_plotter [-h] [--plot {line}] [--output OUTPUT]
                   [--x-scale {linear,log,symlog,logit}] [--x-label X_LABEL]
                   [--y-scale {linear,log,symlog,logit}] [--y-label Y_LABEL]
                   [--title TITLE] [--show {0,1}] [--save {0,1}]
                   [--save-fmt [{png,pdf} ...]] [--label-fmt LABEL_FMT]
                   [--ctx {iodepth,bs}] [--metric {iops,bwps,lat}]

cij_plotter - CIJOE Metric Plotter

optional arguments:
  -h, --help            show this help message and exit
  --plot {line}         The plot to produce (default: line)
  --output OUTPUT       Path to test output (default:
                        /home/safl/git/cijoe/docs)
  --x-scale {linear,log,symlog,logit}
                        Scale of the x-axis (default: linear)
  --x-label X_LABEL     When None is given, a label is generated using plot-
                        data (default: None)
  --y-scale {linear,log,symlog,logit}
                        Scale of the y-axis (default: linear)
  --y-label Y_LABEL     When none is given, a label is generated using plot-
                        data (default: None)
  --title TITLE         When None is given, a title is generated using plot-
                        data (default: None)
  --show {0,1}          Show the plot produced by the plotter (default: 0)
  --save {0,1}          Store the plot produced by the plotter (default: 1)
  --save-fmt [{png,pdf} ...]
                        The format to store the plot in (default: ['png'])
  --label-fmt LABEL_FMT
                        Provide a label format-string using jinja2 template
                        syntax (default: None)
  --ctx {iodepth,bs}    The context attribute to use as 'metric-as-a-function-
                        of' (default: iodepth)
  --metric {iops,bwps,lat}
                        The metric to plot (default: iops)