For Developers of TEN framework

Test Suites in the NYRA Framework

The NYRA framework includes three types of test suites:

1. Unit Tests

2. Smoke Tests

3. Integration Tests

NYRA uses gtest as the testing framework for unit tests and smoke tests, and pytest as the testing framework for integration tests.

Unit Tests

• Directory: tests/nyra_runtime/unit

• Place new C++ unit test cases in tests/nyra_runtime/unit.

After compiling successfully, go to the output directory (e.g., out/linux/x64/tests/standalone/ depending on OS and CPU) and run:

./nyra_runtime_unit_test

Smoke Tests

• Directory: tests/nyra_runtime/smoke

• Place new smoke test cases in tests/nyra_runtime/smoke.

Navigate to the output directory (e.g., out/linux/x64/tests/standalone) and run:

./nyra_runtime_smoke_test

Loop Testing

You can repeatedly run tests multiple times to catch intermittent or timing-related issues:

Only Unit Tests

failed=0
for i in {1..100}; do
  if [ $failed != 0 ]; then
    echo "error occurred:" $failed
    break
  fi
  ./nyra_runtime_unit_test
  failed=$?
done

Only Smoke Tests

failed=0
for i in {1..100}; do
  if [ $failed != 0 ]; then
    echo "error occurred:" $failed
    break
  fi
  ./nyra_runtime_smoke_test
  failed=$?
done

Both Unit and Smoke Tests

failed=0
for i in {1..100}; do
  if [ $failed != 0 ]; then
    echo "error occurred:" $failed
    break
  fi

  ./nyra_runtime_unit_test
  failed=$?

  if [ $failed != 0 ]; then
    echo "error occurred:" $failed
    break
  fi

  ./nyra_runtime_smoke_test
  failed=$?
done

Integration Tests

• Directory: tests/nyra_runtime/integration

• Integration tests are black-box tests for the NYRA framework, validating real-world scenarios.

• Typical structure:

  tests/nyra_runtime/integration/
    ├── test_1/
    │    ├── test_case.py
    │    └── ...
    ├── test_2/
    │    ├── test_case.py
    │    └── ...
    └── ...

Run integration tests with pytest:

cd path/to/out/
pytest tests/nyra_runtime/integration/

Testing Tricks

Linux

You can slow down the NYRA runtime to expose timing-related bugs by limiting CPU usage and/or setting the CPU affinity:

sudo apt install util-linux
cpulimit -f -l 50 -- taskset 0x3 ...

• taskset 0x1: Use only 1 CPU core for the program.

• taskset 0x3: Use only 2 CPU cores for the program.

Reducing available CPU resources can help reveal race conditions or timing flaws.