Skip to content

Add CTest-based CLI smoke tests and fix CLI WAV sample-rate handling#1350

Draft
olilarkin wants to merge 1 commit into
feature/cli-backend-linuxfrom
codex/add-test-suite-for-cli-target
Draft

Add CTest-based CLI smoke tests and fix CLI WAV sample-rate handling#1350
olilarkin wants to merge 1 commit into
feature/cli-backend-linuxfrom
codex/add-test-suite-for-cli-target

Conversation

@olilarkin

Copy link
Copy Markdown
Member

Motivation

  • Provide automated smoke/regression coverage for the headless CLI target so changes to the CLI/API are caught by CI.
  • Exercise real CLI binary behavior (parameters, file I/O, generated audio, MIDI) instead of only unit-level checks.
  • Ensure generated WAV output uses the requested sample rate when --sr is provided.

Description

  • Enable CTest in the top-level build by adding include(CTest) to CMakeLists.txt so tests can be discovered and run.
  • Add a new test directory Tests/CLI with CMakeLists.txt that builds IPlugCLITest-cli and registers two tests: a build fixture and a Python smoke test run of the executable.
  • Add a Python smoke/regression script Tests/CLI/cli_smoke_test.py that exercises --info, --params/--param, --set/--set-name/--set-norm, --save-params/--load-params, generated text/WAV output (--sine + --output-txt/--output), --process-file, and MIDI render flows with --midi/--render.
  • Hook Tests/CLI into the existing test list by adding add_subdirectory(CLI) to Tests/CMakeLists.txt.
  • Fix CLI WAV output sample-rate handling by setting outputSampleRate when --sr is parsed in IPlug/CLI/IPlugCLI_main.cpp so generated WAV headers reflect the requested sample rate.

Testing

  • Built the example CLI plugin and verified the binary with cmake -S Examples/IPlugCLITest -B build-cli-test and cmake --build build-cli-test --target IPlugCLITest-cli, then exercised commands like --info, --set, --sine --output-txt and --sine --output which returned expected outputs (exit code 0 and correct files written).
  • Exercised the new test suite with cmake -S Tests/CLI -B build-cli-suite -DCMAKE_BUILD_TYPE=Debug and ran ctest --test-dir build-cli-suite --output-on-failure, and all registered tests passed.

Codex Task

@claude

claude Bot commented May 9, 2026

Copy link
Copy Markdown

Code Review

Overview

This PR adds a CTest-based smoke test suite for the CLI target and fixes a real bug: outputSampleRate was never set when --sr was parsed without an --input file, so WAV headers generated by --sine --output always had the default 44100 Hz regardless of the requested rate. The fix is correct and minimal.


Bug Fix (IPlug/CLI/IPlugCLI_main.cpp)

Correct. The one-liner outputSampleRate = static_cast<int>(sampleRate) in the --sr branch mirrors the identical assignment that already exists in the --input/--process-file branch (line 446). The test test_generated_audio_outputs directly validates this fix by asserting sample_rate == 1000 in the WAV header.


CMake (CMakeLists.txt + Tests/CMakeLists.txt + Tests/CLI/CMakeLists.txt)

Issues:

  • enable_testing() is redundant in Tests/CLI/CMakeLists.txt. include(CTest) at the root already calls enable_testing() for the whole tree. Calling it again in a subdirectory is harmless but unnecessary. When the file is used as a standalone build it is needed, but in that case the root include(CTest) is absent — so the two modes have inconsistent behavior. Consider guarding with if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR).

  • cmake_minimum_required + project in Tests/CLI/CMakeLists.txt conflict with subdirectory use. CMake silently ignores cmake_minimum_required/project inside add_subdirectory, but having them there signals intent to also build standalone. The standalone path (cmake -S Tests/CLI) requires iPlug2 to be findable independently, which won't work in a fresh checkout without extra -DIPLUG2_DIR flags. The PR description says this was tested standalone, but there is no documentation or wrapper script for it. Either document the standalone invocation explicitly or remove the standalone design if it is not intended.

  • Fixture ordering may cause a race on parallel CTest runs. The FIXTURES_SETUP / FIXTURES_REQUIRED mechanism should handle this, but IPlugCLITest.CLI.Build invokes cmake --build on ${CMAKE_BINARY_DIR} which is the build directory of the test project, not necessarily the directory that owns IPlugCLITest-cli when built as part of the main tree. Verify this resolves correctly when ctest is run from the main build tree.


Python Smoke Test (Tests/CLI/cli_smoke_test.py)

Correctness:

  • The bug fix is directly exercised: test_generated_audio_outputs asserts sample_rate == 1000 in the WAV header. Good.
  • test_parameters normalizes param 0 to 0.5 and expects value == 50.0, implicitly assuming a linear [0, 100] Gain range. If the example plugin ever adds a curve or changes the range, this breaks silently. A comment naming the assumption would help.

Style / minor issues:

# Deprecated in Python 3.9+; use the built-in tuple[] syntax instead
from typing import Tuple
def read_wav_params(path: Path) -> Tuple[int, int, int, int]:

Replace with -> tuple[int, int, int, int] (requires from __future__ import annotations already present, so this is safe on Python 3.7+).

def assert_close(actual: float, expected: float, tolerance: float = 1.0e-9) -> None:
    if not math.isclose(actual, expected, rel_tol=tolerance, abs_tol=tolerance):

Passing the same value as both rel_tol and abs_tol is unusual — for values near zero rel_tol is irrelevant, and for large values abs_tol may be too tight. Separate them or pick one appropriate strategy. For parameter round-trip at 12.5, abs_tol=1e-6 alone is more readable.

Test coverage gaps:

  • No negative/error-path tests (e.g., missing executable, invalid param index, unreadable input file). These would catch regressions in CLI error handling.
  • No validation of the generated audio content — only the WAV header is checked. Even a simple sanity check (non-zero RMS, or that a 100 Hz sine at 1000 Hz SR has the right period) would catch DSP-level regressions.
  • test_midi_render only checks that 10 samples are written; it does not verify any audio was actually rendered (samples could all be zero).

Hardcoded plugin metadata:

assert info["name"] == "IPlugCLITest"
assert info["version"] == "1.0.0"
assert info["manufacturer"] == "AcmeInc"
assert [param["name"] for param in params] == ["Gain", "Frequency", "Attack", "Decay"]

These couple the test tightly to IPlugCLITest's current config.h. That is intentional (it is a regression test), but a comment saying so avoids confusion when someone updates the example.


Summary

Area Verdict
Bug fix Correct and well-tested
CMake integration Mostly sound; standalone-vs-subdirectory mode is slightly inconsistent
Python smoke tests Good coverage of the happy path; minor style issues; no error-path or audio-content tests
Conventions (2-space indent, C++17, etc.) Python file follows PEP 8; C++ change is one line and matches surrounding style

The core fix and test infrastructure are solid. The items above are mostly polish — the most actionable ones are the enable_testing() guard and adding at least one negative-path test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant