This article describes the results of performance benchmark tests performed using Siemplify 5.6.0 software in October 2021. The tests were conducted using Siemplify cloud (SaaS) deployment in two deployment sizing scenarios:

  • Simple deployment for small and medium customers
  • Advanced deployment for big customers 

Benchmarking Process

The performance benchmark process was executed using automated tests that injected and ran 1000 alerts with an automated playbook to completion.

Ingestion Process

The ingestion process used the Demo Connector, which creates Alerts with the following predefined characteristics:

  • Number of Events
  • Number of Entities
  • Alert size

The Alerts were ingested using batch ingestion methods. The benchmarking process was executed using an automated test, which allows for batch incident parsing, mapping, classifying, ingestion, and execution of a specified playbook. The test was measured by the total time it took from the first alert received to closing the cyber case.

Each ingested alert included 5 security events and 10 entities with total 5K size. One playbook was executed per each alert with 10 steps.  

Results

Ingestion Results

Alerts per dayTime to process a single Alert (in ms)
1,000197.36
5,000198.56
10,000199.01
20,000199.65
50,000230.11
60,000256.23
100,000381.73
120,000570.11

Playbook Results

Simple Deployment

The deployment supports up to 30K Playbooks per day with 300K Playbook Actions.

Playbooks per dayTime to process a single Playbook (in ms)
1,0002749.13
5,0002856.77
10,0002987.61
20,0003521.49
30,0005972.57

Advanced Deployment

The deployment supports up to 65K Playbooks per day with 650K Playbook Actions. 

Playbooks per dayTime to run a single Playbook (in ms)
40,0003873.89
50,0004256.61
60,0005506.80
65,0007133.28

NOTE: The data specified in the Performance benchmark were processed without data compression. The results might vary based on many factors, including the Alerts size, system configurations, Playbook complexity, data settings, and the type of actions performed.