Deployment overview using Ansible playbooks

Can we add the Ansible playbook execution diagram in the docs?

Once the cloud instances have been successfully provisioned, the Netprobe can now be deployed via Ansible.

Prerequisites Copied

• Compatible Ansible/Python in control and target machines (compatibility matrix)
• Resources: either in localhost or cloud storages (+ access permissions for virtual machines if resources are in cloud)
  • Netprobe
    • Netprobe tarball
    • Netprobe setup file - Gateway Demonstrator - are GW files for internal demo only, including LICD?
    • Gateway tarball
    • Gateway setup file
    • License Daemon tarball
• Cloud CLI tools (installed automatically during bootstrap):
  • AWS CLI for S3 access
  • Azure CLI for Azure Blob Storage access
  • Google Cloud SDK for GCS access

SSH configuration for Ansible Copied

Configure startup.yml to define the SSH configurations required for Ansible to securely connect to your target servers.

# Ansible SSH configurations (required)
ansible_user: "netprobe_user"
ansible_ssh_private_key_file: "../../../resources/id_rsa"
ansible_ssh_common_args: "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
# Connection timeout configuration
connection_timeout: 300

Bootstrap for Python and cloud CLI installation for Ansible Copied

Run bootstrap.yml to set up the Python runtime environment and system dependencies on your target servers. You can also configure this to install cloud-specific CLI tools. This preparation ensures Ansible functions properly to automate the Netprobe installation process.

# Set to true to enable additional Python installation for Ansible compatibility
perform_bootstrap: false

# OS-specific package options (uncomment the appropriate line for your target OS):
# RedHat
# bootstrap_packages: "ansible python3 python3-pip python3-setuptools python3-six python3-devel"
# Ubuntu
bootstrap_packages: "python3 python3-pip python3-setuptools python3-six python3-dev"
# Pip packages (same for all OS versions)
pip_packages: "ansible six setuptools pyyaml jinja2 cryptography pyOpenSSL boto3 botocore"

python_path: "/usr/bin/python3"

bootstrap: |
    set -e
    if command -v dnf > /dev/null 2>&1; then
      echo "Using dnf package manager"
      sudo dnf install -y {{ bootstrap_packages }}
    elif command -v yum > /dev/null 2>&1; then
      echo "Using yum package manager"
      sudo yum install -y {{ bootstrap_packages }}
    elif command -v apt-get > /dev/null 2>&1; then
      echo "Using apt package manager"
      sudo DEBIAN_FRONTEND=noninteractive apt-get update -y && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q {{ bootstrap_packages }}
    else
      echo "No supported package manager found"
      exit 1
    fi
    echo "Installing essential Python modules via pip..."
    sudo {{ python_path }} -m pip install --upgrade {{ pip_packages }}

    {{ additional_bootstrap | default('') }}

Cloud storage authentication Copied

Before running any Ansible operation, ensure you have access to the cloud storage (whether in AWS, Azure, or Google Cloud) where your resources are stored.

AWS Copied

For Amazon Web Services integration, the Ansible playbooks support multiple authentication methods for S3 access. The authentication follows the AWS SDK credential precedence chain.

Required S3 permissions Copied

Regardless of authentication method, the credentials must have the following S3 permissions:

• S3 Object Access (s3:GetObject) - for downloading deployment resources
• S3 Bucket List (s3:ListBucket) - for listing bucket contents

Supported authentication methods Copied

• Instance Profile (Default - Recommended) Uses the IAM instance profile attached to the EC2 instance. No credential parameters needed.

  Prerequisites:

  • Running on an EC2 instance with attached instance profile
  • Instance profile has required S3 permissions: s3:GetObject, s3:ListBucket

  Configuration:

  resources:
    source:
      aws:
        region: us-east-1  # Optional region parameter
        # No other parameters needed - uses instance profile automatically
  

  Use this when: Default behavior for EC2 instances with attached instance profiles.

• Named AWS Profile Uses a named AWS CLI profile configured on the target machine.

  Prerequisites:

  • AWS CLI configured with named profile on target machine
  • Profile has required S3 permissions

  Configuration

  resources:
    source:
      aws:
        profile: non_default_profile  # Name of AWS CLI profile
        region: us-east-1            # Optional region parameter
  

  Use when using specific AWS CLI profiles for credential management.

• Session Token (Temporary Credentials) Uses temporary AWS credentials with session token.

  Prerequisites

  • Valid AWS access key, secret key, and session token
  • Temporary credentials have required S3 permissions

  Configuration

  resources:
    source:
      aws:
        sessiontoken:
          accesskey: AKIAIOSFODNN7EXAMPLE
          secretkey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYzEXAMPLEKEY
          token: AQoEXAMPLEH4aoAH0gNCAPyJxz4BlCFFxWNE1OPTgk5TthT+FvwqnKwRcOIfrRh3c/L
        region: us-east-1
  

  Use for temporary access or when using AWS STS assumed roles.

Google Cloud Platform (GCP) Copied

The Ansible playbooks support multiple authentication methods based on the Google Cloud Ansible collection. The authentication type is configured using the auth_kind parameter in your GCP resource configuration.

Required service account permissions Copied

Regardless of authentication method, the service account must have the following IAM roles:

• Storage Object Viewer (roles/storage.objectViewer) - for accessing cloud storage objects
• Storage Bucket Viewer (roles/storage.bucketViewer) - for listing storage buckets

Supported authentication methods Copied

• Application Default Credentials (auth_kind: "application")
  Uses Google Cloud Application Default Credentials (ADC). No credential parameter needed.

  Prerequisites

  • GOOGLE_APPLICATION_CREDENTIALS environment variable set to service account key file path, OR
  • Virtual machine has a service account attached (recommended for GCP compute instances)

  Configuration

  resources:
    source:
      gcp:
        auth_kind: "application"
        # No credential parameter required
  

  Recommended for deployments on GCP compute instances with attached service accounts.

• Machine Account (`auth_kind: “machineaccount”)
  Uses the default service account of the compute instance.

  Prerequisites

  • Running on a GCP compute instance
  • Instance has a service account attached

  Configuration:

  resources:
    source:
      gcp:
        auth_kind: "machineaccount"
        credential: "service-account@project-id.iam.gserviceaccount.com"  # Service account email
  

  Use when you want to explicitly specify the service account email for a compute instance.

• Service Account Key File (`auth_kind: “serviceaccount”)
  Uses a service account JSON key file for authentication.

  Prerequisites:

  • Service account JSON key file accessible on the target machine

  Configuration:

  resources:
    source:
      gcp:
        auth_kind: "serviceaccount"
        credential: "/path/to/service-account-key.json"  # Path to JSON key file
  

  Use for deployments from non-GCP environments or when using explicit service account keys.

• Access Token (`auth_kind: “accesstoken”)
  Uses a Google Cloud access token for authentication.

  Prerequisites

  • Valid Google Cloud access token
  • Token management (renewal) handled externally

  Configuration

  resources:
    source:
      gcp:
        auth_kind: "accesstoken"
        credential: "ya29.c.Kp6B9Q..."  # Access token string
  

  Use for temporary access or when integrating with external token management systems.

Unsupported authentication method

service_account_contents - inline service account JSON content is not supported for security reasons.

Netprobe configuration Copied

Use the following configuration to deploy the Netprobe:

# =============================================================================
# NETPROBE CONFIGURATION
# =============================================================================

# =============================================================================
# ENVIRONMENT SETUP
# Configuration for setting up the environment for initial netprobe usage
# =============================================================================

# List of packages needed by netprobe to run
packages:
  # Method to install the package, available options at the moment are 'system', 'pip', and 'shell'
  - method: system
    packages: [
      # JAVA package if java-plugins or c2-plugins will be used, sample packages below
      #{ name: openjdk-21-jre-headless },
      #{ name: java-21-openjdk },
      # RedHat 8/9 packages
      #{ name: openssl-libs }, { name: glibc }, { name: libnsl2 }, { name: libxcrypt }, { name: zlib }, { name: libstdc++ }, { name: libgcc }, { name: libtirpc },
      #{ name: krb5-libs }, { name: libcom_err }, { name: keyutils-libs }, { name: libselinux }, { name: pcre2 }, { name: libcurl }
    ]

# Workspace directory (required)
workspace_dir: "/opt/workspace"

# Environment variables (required)
env_vars:
  # Component log file (required, for rollback mechanism)
  LOG_FILENAME: "{{ workspace_dir }}/logs/{{ component }}/{{ component }}.log"
  MAX_LOG_FILE_SIZE_MB: "10"
  # Gateway information (required)
  GATEWAY_HOST: "0.0.0.0"
  GATEWAY_PORT: "7039"
  # Netprobe information (required)
  NETPROBE_PORT: "7036"

# =============================================================================
# RESOURCE CONFIGURATION
# Configuration where resources come from when installing or updating the system
# =============================================================================

resources:
  # Remove source to use localhost as default
  #source:
    # Valid configurations are gcp, azure, aws
    #aws:
      # Default authentication is instance profile (no parameters needed)
      # Optional region parameter, defaults to instance metadata
      #region: us-east-1
      # Alternative: Named AWS CLI profile
      #profile: non_default_profile
      # Alternative: Session token for temporary credentials
      #sessiontoken:
      #  accesskey: accesskey
      #  secretkey: secretkey
      #  token: token
    #azure:
    #  resource_group: resource_group
    #  storage_account_name: storage_account_name
    #  account_key: account_key
    #gcp:
      # Default authentication is application, see Ansible readme.md for more details
      #auth_kind: application
      #credential: "please see Ansible readme.md if needed"
  # Cloud storage container where resource is located
  # container: geneos
  # Path where resource is located (required)
  path: ../../resources/netprobe.tar.gz
  # Netprobe executable file (required)
  executable: netprobe.linux_64
  setupfile:
    # Cloud storage container where resource is located (optional, defaults to resources.container)
    #container: geneos
    # Path where resource is located in (required)
    path: ../../resources/netprobe.setup_template.xml

# =============================================================================
# UPDATE OPERATIONS
# Configuration specific to update operations
# =============================================================================

# List of files to import over during Netprobe updates (only used for update operation)
import_config:
  - dir: "collection_agent"
    patterns:
      - 'collection-agent.yml'
      - 'logback.xml'
  - dir: "Workflow"

# =============================================================================
# VERIFICATION
# Configuration for verification after every operation
# =============================================================================

# List of patterns for netprobe verification after every operation
verification:
  # Initialization delay after update before logs verification (optional, defaults to 5 seconds) 
  #initialization_delay: 5
  # Log strings to indicate system behavior (required)
  # List of strings are patterns used for Extended Regular Expressions
  success:
    - "SelfAnnouncing Connected to Gateway"
  errors:
    - "ERROR.*FATAL.*"

# =============================================================================
# DIAGNOSTICS
# Configuration for diagnostic operations (only needed on fetch_archive operation)
# =============================================================================

# Diagnostic configuration (only needed on fetch_archive operation)
diagnostic:
  # A list of files to archive for diagnostic purpose
  files:
    # Netprobe log files, as configured in env_vars.LOG_FILENAME
    - "{{ env_vars.LOG_FILENAME | dirname }}*"
    # Collection agent log files and log configuration
    - "{{ workspace_dir }}/{{ component }}/collection-agent.log"
    - "{{ workspace_dir }}/{{ component }}/collection_agent/collection-agent.log"
    - "{{ workspace_dir }}/{{ component }}/collection_agent/logback.xml"
    # Collection agent configurations
    - "{{ workspace_dir }}/{{ component }}/collection-agent.yml"
    - "{{ workspace_dir }}/{{ component }}/collection_agent/collection-agent.yml"
  dest: "./"

Configure package dependencies Copied

Configure the Netprobe deployment Copied

Installing the Netprobe Copied

The Netprobe can be installed either on a remote host or a local machine by running the following Ansible commands.

• Remote host installation

  ansible-playbook -i "<hosts>," netprobe.yml
  

  Important

  Replace <hosts> with a comma-separated host list. A comma is needed at the end of the list.

• Local machine installation

  ansible-playbook -i "localhost," -c local netprobe.yml
  

  Important

  A comma is needed at the end of localhost.

Running these commands will perform the following:

1. Install all required packages.
2. Set up the Netprobe system service.
3. Install the Netprobe.
4. Run the Netprobe.

Ansible-supported operations Copied

The Ansible playbook supports the following operations:

Specify your desired operation in the command by appending -e "operation=<value>.

For example: ansible-playbook -i "hosts," netprobe.yml -e "operation=update".

Custom commands Copied

Update Netprobe Copied

ansible-playbook -i "<hosts>," netprobe.yml -e "operation=update"

The update operation downloads and installs the Netprobe package configured in resources.container (if from Cloud Storage) and resources.path. A backup copy of the current version will be stashed for a potential rollback operation. The list of files in import_config will be imported to the new package after installation. Post verification will be performed based on log patterns configured in verification.success and verification.error.

Rollback Netprobe Copied

ansible-playbook -i "<hosts>," netprobe.yml -e "operation=rollback"

The rollback operation will only work after a previous update operation. A copy of the current Netprobe installation will be stashed with suffix _failed for further analysis. Post verification will be performed based on log patterns configured in verification.success and verification.error.

Update Netprobe setup file Copied

ansible-playbook -i "<hosts>," netprobe.yml -e "operation=setupfile"

The setupfile operation downloads the Netprobe setup file configured in resources.setupfile.container (if from Cloud Storage) and resources.setupfile.path. Post verification will be performed based on log patterns configured in verification.success and verification.error. User can rerun the setupfile operation on a working version in case of errors.

Update environment variables and reinstalls required Netprobe packages Copied

ansible-playbook -i "<hosts>," netprobe.yml -e "operation=environment"

The environment operation rewrites environment variables configured in env_vars and reinstalls packages listed in packages. Post verification will be performed based on log patterns configured in verification.success and verification.error. User can rerun the environment operation on working values and packages in case of errors. (eg. Gateway host: env_vars.GATEWAY_HOST)

Fetch diagnostic files Copied

ansible-playbook -i "<hosts>," netprobe.yml -e "operation=fetch_archive"

The fetch_archive operation will archive files listed in diagnostic.files and archive them in the format netprobe_diagnostics_<host_name>_<YYYY-mm-dd>_<HHMM>. This operation does not stop the service to ensure continuous monitoring during diagnostic collection.

Uninstall Netprobe Copied

ansible-playbook -i "<hosts>," netprobe.yml -e "operation=uninstall"

The uninstall operation deletes everything inside workspace_dir and removes the systemd service file.