Skip to content

Experiment Execution Guide for PANTHER¤

This guide explains how to set up, run, and interpret experiments in PANTHER. Learn about the core components, execution flow, and result analysis.

Overview¤

Core Components¤

ExperimentManager¤

  • Orchestrates the setup, execution, and teardown of experiments.
  • Key methods:
  • run_tests: Executes the test cases.
  • _initialize_test_cases: Loads and prepares test cases from configuration.

TestCase¤

  • Represents an individual test within an experiment.
  • Key methods:
  • deploy_services: Deploys required services.
  • execute_steps: Executes the test steps defined in the configuration.
  • validate_assertions: Verifies test outcomes.

Setting Up an Experiment¤

  1. Prepare Configuration:
  2. Use the provided sample configuration file: 
    cp config/experiment_config.yaml.example config/experiment_config.yaml

  3. Update the configuration file to define your test cases, services, and assertions.

  4. Build Docker Images:

  5. Build Docker images for the required services: 

    panther build-docker-images

  6. Validate Configuration:

  7. Ensure the configuration is valid before execution: 
    panther validate-config --config config/experiment_config.yaml

Running an Experiment¤

  1. Execute the experiment: 

    panther run-experiment --config config/experiment_config.yaml

  2. Monitor the logs for real-time updates:

  3. Logs are stored in the directory defined in the configuration (outputs/logs).

  4. Results are automatically saved in the outputs/ directory.

Example Scenario: Testing QUIC¤

Configuration¤

  • Test QUIC client-server communication using Picoquic.

Sample YAML: 

tests:
  - name: "QUIC Server-Client Test"
    description: "Testing QUIC communication with Picoquic."
    network_environment:
      type: "docker_compose"
    services:
      quic_server:
        name: "quic_server"
        implementation:
          name: "picoquic"
          type: "iut"
        protocol:
          name: "quic"
          version: "rfc9000"
          role: "server"
        ports:
          - "4443:4443"
    steps:
      wait: 100
    assertions:
      - type: "service_responsive"
        service: "quic_server"
        endpoint: "4443/health"
        expected_status: 200

Execution¤

  1. Build Docker images for Picoquic: 
    panther build-docker-images
  2. Run the experiment: 
    panther run-experiment --config config/experiment_config.yaml

Interpreting Results¤

  1. Output Directory:

  2. Results are saved in the outputs/ directory.

  3. Log Files:

  4. Located in outputs/logs.

  5. Contain details about test execution and errors.

  6. Success Criteria:

  7. Validate against assertions defined in the configuration.

Troubleshooting¤

  1. Missing Services:

  2. Ensure all services are defined in the configuration.

  3. Validation Errors:

  4. Run the validation command to debug configuration issues.

  5. Docker Issues:

  6. Verify Docker is running and accessible.

For further assistance, consult the Main README or contact the development team.