I once created a cross-platform test runner for CLI applications, using an embedded Lua interpreter to execute the application then check its output, filesystem changes, etc. It wasn't very good, and doing it again but better has been in the back of my mind ever since. However, I have come to think that the best way to do system tests is often to use the same test runner as for the code; you're more likely to run them if you have to opt-out rather than remember to run a separate test suite, you don't have to learn a new test language/framework, and you don't have the overhead of another application in the build system.

Below is something I wrote in Rust yesterday to test one of my applications; the only dependency is rand for temporary file name generation. It's in the tests directory, so Cargo runs them as integration tests. My filename is cli.rs and tests are prefixed with cli_ so I can immediately identify them as system tests.

use std::{
    fs::{File, read_to_string, remove_file},
    io::Write,
    path::PathBuf,
    process::{Command, Output},
    env,
    str,
};

use rand::{
    distributions::Alphanumeric,
    Rng,
    thread_rng,
};


const APP_NAME: &str = "../../target/debug/app-name";

/// Retrieve a path to a non-existent file in a temporary directory.
fn temp_file() -> PathBuf {
    let mut rng = thread_rng();
    let path = env::temp_dir();

    let file = loop {
        let name: String = (&mut rng).sample_iter(Alphanumeric)
            .take(4)
            .map(char::from)
            .collect();

        let mut file = path.clone();

        file.push(name);
        file.set_extension("txt");

        if ! file.exists() { break file; }
    };

    println!("* temporary file path: {:?}", file);
    file
}

/// Create a temporary file with the provided data.
fn temp_file_with(content: &str) -> (PathBuf, File) {
    let path = temp_file();
    let mut file = File::create(&path).unwrap();

    file.write_all(content.as_bytes()).unwrap();

    (path, file)
}

/// Execute a command and return its output.
fn exec(command: &str, args: &[&str]) -> Output {
    Command::new(command)
        .args(args)
        .output()
        .expect("Failed to execute process")
}

The one thing I don't like is hard-coding the path to the application in the code.

We have three functions:

• temp_file() gives us a path to a non-existent file in a temporary directory, letting us do whatever we might need with a file that's safe to work with. Notice the println!() call -- if a test fails, Cargo prints the test's standard output to the console, which will include the path to our temporary file. Because it will probably fail before we run our cleanup code (as long as we don't always run such cleanup), we'll have the file available for inspection if needed.
• temp_file_with() creates a temporary file and fills it with content.
• exec() runs a program with whatever arguments are given and returns its standard output and standard error.

It's designed to be used like:

#[test]
fn test_my_application() {
    let (path, file) = temp_file_with("\
    Line 1\n\
    Line 2\n\
    Line 3\n\
    ");

    let output = exec(APP_NAME, &["-opt1", "-opt2", path.to_str().unwrap()]);
    let output = str::from_utf8(&output.stdout).unwrap();

    assert!(output.contains("Some text"));

    let text = read_to_string(&path).unwrap();
    assert!(text.contains("Line 2"));

    remove_file(path);
}

 Possible improvements:

• Pass the (Optional) file's extension to temp_file().
• After so many loop iterations while generating the filename, increase the number of characters in that name.
• Obtain the executable's path from cargo-metadata and store it via std::sync::Once (or once_cell::sync::OnceCell on stable). This is likely not worth it without a large number of tests.
• Create a way to always remove a file, even after test failure.