Scripts#

There are multiple scripts in the resources of cijoe. These can be run directly with the cli tool.

# Run a script directly
cijoe --config cijoe-example-core.default/cijoe-config.toml \
	core.example_script_default

You can also create your own cijoe scripts locally and run them which can be run in the same manner. Let’s start by running the script produced by cijoe --example:

# Run a local script directly
cijoe --config cijoe-example-core.default/cijoe-config.toml \
	cijoe-example-core.default/cijoe-script.py

When running, an output directory is populated with log files, statefiles, command-output files, and artifacts produced or otherwise collected by the script.

Content Overview#

Let’s take a look at the example script provided by cijoe --example:

"""
cijoe example script
====================

The script is a modified "Hello, World!" example. It repeatedly prints a message a set
number of times and allows parameterization of the message content.

The purpose of this script is to demonstrate how to run commands and supply input to the
script using a configuration file, environment variables, command-line arguments and 
workflow step arguments.

An example of using the core infrastructure of cijoe:

* cijoe.run(command)

  - Error-handling; checking return-code
  - Output processing state.output()

Input is given to scripts via configuration-files, environment variables and from
workflow-step-arguments, this is demonstrated as the first thing in the script.

cijoe also has primitives for transferring data:

* cijoe.get(src, dst)

  - Transfer 'src' directory or file from **target** to 'dst' on **initiator** 

* cijoe.put(src, dst)

  - Transfer 'src' directory or file from **initiator** to 'dst' on **target** 

These are not used in the example code below, but you can experiment and try adding
them yourself.
"""

import logging as log
from argparse import ArgumentParser, Namespace

from cijoe.core.command import Cijoe


def add_args(parser: ArgumentParser):
    """Optional function for defining command-line arguments for this script"""
    parser.add_argument(
        "--repeat",
        type=int,
        default=1,
        help="Amount of times the message will be repeated",
    )


def main(args: Namespace, cijoe: Cijoe):
    """Entry-point of the cijoe-script"""

    # Grab message from the configuration-file
    message = cijoe.getconf("example.message", "Hello World!")

    # When executed via workflow, grab the step-argument
    repeat = args.repeat
    if repeat < 1:
        log.error(f"Invalid step-argument: repeat({repeat}) < 1")
        return 1

    log.info(f"Will echo the message({message}), repeat({repeat}) times")

    # Now, execute a command that echoes the 'message' 'repeat' number of times
    for _ in range(1, repeat + 1):
        err, state = cijoe.run(f"echo '{message}'")
        if "Hello" not in state.output():
            log.error("Something went wrong")
            return 1

    log.info("Success!")

    return 0

As you can see, then it is just a regular Python script making use the batteries included with Python in the form of the argparse and logging.

What makes the above a cijoe script, is the convention for the main() function and the use of the cijoe module. The following subsections go through how the cijoe-isms are used.

Command Execution#

The following method is available to execute commands.

cijoe.run(command, evars, ...)

This is similar to the Python builtin subprocess.run(), with the major behavioral differences:

You can change where the command is executed!
- Via the configuration-file; see Transport Configuration.
You cannot change where the Python code is executing!
- cijoe runs Python / script logic on the initiator but not on the target

You can handle all the logic in Python without requiring Python and its dependencies on the target system. Keep this in mind when writing your scripts to ensure that your Python code focuses on logic, control flow, and text processing. Avoid using shell-related helpers like shutil since the Python code will not be re-targeted only the commmand.

Command Output#

The result from cijoe.run(...) is the tuple:

(err: int, state: CommandState)

That is, the error / returncode of the command and a command-state object. A couple of things to take note of:

command output is always a combination of the stdout and stderr output streams into a single command output stream
command output is always captured and written to file on the initiator
You can tell the cijoe command-line tool to also dump the command output to stdout on the initiator, in realtime, with the argument -m / --monitor

To handle the command output in your script, then you can conveniently read the entire output via a helper on the CommandState object:

state.output()

It will return the output in a UTF-8 decoded form with errors replaced. This work well when working with textual output from commands. If you want to work with non-decoded output from the command, then you can read and process the command-output file instead of using the output() helper.

Logging#

When using the cijoe command-line tool, then there is a option for logging -l / --log-level. The content of this log comes from calls to the Python built-in logging module. Thus, if you want logging in your scripts, then you can just do as the example does, e.g.:

import logging as log

...

log.info("Status information on something")
log.error("Something went very wrong!")

Log statements are printed to stdout in the shell where cijoe is running. When writing cijoe scripts, it is recommended to use logging for any printed output. This ensures that regular output is reserved for command output, such as when using the -m or --monitor options.

Getting Configuration Values#

cijoe provides a convenient function for retrieving values from the configuration file. It can be used as follows:

value = cijoe.getconf("example.max.value")

log.info(f"Max value is capped at ({value})")

You can override configuration file values using environment variables. For example, if the environment variable EXAMPLE_MAX_VALUE is set, its value will be used instead of the value specified in the configuration file.

Script Arguments#

Some cijoe scripts have command-line arguments, which are evaluated with the argparse library. These arguments are added by defining a function add_args(parser: argparse.ArgumentParser) in the cijoe script file.

from argparse import ArgumentParser

def add_args(parser: argparse.ArgumentParser):
   parser.add_argument(
     "--message",
     type=str,
   )

Note that the argparse documentation advices against using the bool() function as type converter. The cijoe scripts that has boolean cli-arguments use the choices parameter and a custom store action to convert the strings "true" and "false" to True and False, respectively.

from argparse import ArgumentParser, _StoreAction

def add_args(parser: ArgumentParser):
   class StringToBoolAction(_StoreAction):
      def __call__(self, parser, namespace, values, option_string=None):
         setattr(namespace, self.dest, values == "true")

   parser.add_argument(
      "--run_local",
      choices=["true", "false"],
      default=True,
      action=StringToBoolAction
   )

Adding Python packages#

Python projects are commonly installed in virtual environments (venv) and it is recommended that cijoe is installed using pipx, since when doing so then cijoe will be installed in a venv, however, it is made available on your shell / terminal, such that you can simply invoke the cijoe command-line tool without first activating a venv or in other ways “enter” the venv. This is very convenient.

However, in case your script requires some Python package which is not readily available, how to add it? This can be done by injecting it into the cijoe venv provided by pipx. Here is an example, of adding matplotlib:

# Add a Python package to the cijoe venv provided by pipx
pipx inject cijoe matplotlib