Skip to content

test: characterize CLI behaviour - #103

Open
shayyz-code wants to merge 2 commits into
mainfrom
test/90-characterize-cli
Open

shayyz-code wants to merge 2 commits into
mainfrom
test/90-characterize-cli

Conversation

@shayyz-code

@shayyz-code shayyz-code commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR records how the poo binary behaves in v0.1.11, so the upcoming CLI refactor has to change it on purpose. It adds a cli module to the language_specs test binary (tests/language_specs/cli.rs) with 18 spec_cli_* specs. Each spec runs env!("CARGO_BIN_EXE_poo") in a fresh temp directory that is deleted afterwards. The interpreter and the conformance harness are unchanged.

Behaviour the specs record:

  • A successful program exits 0 with an empty stderr. A top-level return adds Result: <Debug> to stdout, such as Result: Int(3) or Result: Vector([Int(1), Int(2)]). Without a return, and for an empty file, nothing extra is printed.
  • Parse, lexer, runtime, and Io errors go to stderr as <Kind> error: in '<path>': <message> (Io: failed to read '<path>': ...) and exit 1. Every one of these specs also asserts that the exit code is not 101 (a panic).
  • Output written before a runtime error stays on stdout.
  • With no arguments, the binary prints Usage: <program> <file_path> to stderr and exits 1.
  • --help and --version are read as file paths, which gives an Io error and exit 1. So is --speed when it comes before the file path.
  • --speed after the path adds \nInterpreted within: <Duration Debug> to stdout. The spec checks the shape of that line, not the number. On an error the line is not printed.
  • Extra unknown arguments are ignored without any message.
  • spec_cli_runs_recursive_function is #[ignore = "#45: ..."]. Today the binary panics with exit 101 instead of printing Result: Int(120).
  • spec_cli_reports_non_utf8_argument_without_panicking is unix-only and #[ignore = "#110: ..."]. Today a non-UTF-8 argument such as b"\xff.poo" panics in std::env::args() with exit 101; the spec expects exit 1 with an Io or usage error.

Assertions check stable substrings. They do not depend on the absolute temp path or on timing values.

Issue

Closes #90

Verification

  • cargo test --test language_specs cli:::

    running 18 tests
    test cli::spec_cli_reports_non_utf8_argument_without_panicking ... ignored, #110: CLI panics with exit 101 on non-UTF-8 arguments
    test cli::spec_cli_runs_recursive_function ... ignored, #45: interpreter panics with exit 101 on recursive calls
    test cli::spec_cli_reports_missing_file_as_io_error ... ok
    test cli::spec_cli_prints_usage_without_arguments ... ok
    test cli::spec_cli_reports_parse_error_on_stderr_with_exit_code_1 ... ok
    test cli::spec_cli_speed_flag_appends_timing_line ... ok
    test cli::spec_cli_reports_directory_path_as_io_error ... ok
    test cli::spec_cli_ignores_extra_unknown_arguments ... ok
    test cli::spec_cli_keeps_output_printed_before_runtime_error ... ok
    test cli::spec_cli_runs_program_and_exits_zero ... ok
    test cli::spec_cli_reports_runtime_error_on_stderr_with_exit_code_1 ... ok
    test cli::spec_cli_treats_help_flag_as_file_path ... ok
    test cli::spec_cli_treats_version_flag_as_file_path ... ok
    test cli::spec_cli_speed_flag_omits_timing_line_on_error ... ok
    test cli::spec_cli_speed_flag_before_path_is_treated_as_file_path ... ok
    test cli::spec_cli_prints_top_level_return_value_in_debug_form ... ok
    test cli::spec_cli_prints_nothing_extra_without_top_level_return ... ok
    test cli::spec_cli_error_cases_exit_with_code_1_not_101 ... ok
    
    test result: ok. 16 passed; 0 failed; 2 ignored; 0 measured; 58 filtered out
    
  • cargo test --test language_specs cli::spec_cli_reports_non_utf8 -- --ignored fails as expected: exit 101 from the env.rs unwrap.

  • cargo test --test language_specs cli::spec_cli_runs_recursive_function -- --ignored fails as expected. The binary exits 101, and stderr shows the panic at src/type_inference.rs:117:36: Undefined function: fact.

  • just check passes (fmt, check, clippy, tests including the conformance corpus, mdbook build and test, dist generate --check, cargo deny check).

  • Each commit is a self-contained Conventional Commit and passes just check

Documentation and Visual Impact

None. Only tests changed. The module list in tests/language_specs/main.rs now includes cli.

🤖 Generated with Claude Code

shayyz-code and others added 2 commits October 14, 2026 14:59
Add spec_cli_* specs that run the poo binary and pin down its v0.1.11
exit codes, stdout, and stderr: Result output for top-level return,
parse, runtime, and Io errors exiting 1, the usage message, --help and
--version read as file paths, the --speed timing line, and ignored
extra arguments. Error paths assert exit 1 rather than the panic code
101. Recursion is recorded as an ignored known failure for #45.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assert only the stable prefix of the missing-file Io error, check the
lexer case for its exact message, give each argument case a real stderr
needle, and include stderr in every contains assertion. Record the
non-UTF-8 argument panic as an ignored unix-only known failure for #110.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shayyz-code
shayyz-code force-pushed the test/90-characterize-cli branch from a30efa7 to 32c7984 Compare October 14, 2026 08:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: characterize CLI behaviour

1 participant