Skip links
Crest Data
Automating Data Parsing and Bidirectional Movement for Security Analytics and Eliminating Manual Hurdles

Automating Data Parsing and Bidirectional Movement for Security Analytics and Eliminating Manual Hurdles

Author
Published on:
Published in: Security
Case Study

Automating Data Parsing and Bidirectional Movement for Security Analytics and Eliminating Manual Hurdles

Automating Data Parsing and Bidirectional Movement for Security Analytics and Eliminating Manual Hurdles

 Executive Summary

Until recently, it has been difficult for the customer to ingest and analyze security data. They had to manually set up S3 bucket connections and develop notebooks to capture and parse critical data, including AWS CloudTrail, AWS VPC logs, and Syslogs. Additionally, to conduct analytics, the customer had to manually build jobs and queries via the user interface, resulting in extensive manual activities and reliance on direct access to the instance to run the queries regularly.

Crest Data overcame these challenges by creating custom notebooks for collecting and parsing data from S3 buckets, thereby eliminating manual work for the customer. These notebooks also facilitate bi-directional data flow, enabling the customer to ingest data from the customer to Splunk and from Splunk to customer tables. Moreover, Crest developed a Splunk app for the customer that provides new commands that allow administrators to perform queries and run jobs from Splunk, removing the need to access the instance directly and streamlining security operations.

About the Customer

The customer is a cloud data intelligence platform used by many enterprises to accelerate innovation with a unified data engineering, data analytics, and data science platform. It was the first to introduce the “lakehouse” model, which combines the performance and governance of a data warehouse with the cost and scalability of a data lake.

Customer Challenge

The cloud-based data intelligence platform had operational challenges when ingesting and analysing security telemetry. The customer had to manually set up S3 bucket integrations and create notebooks from scratch to just ingest and parse critical data types (AWS CloudTrail, AWS VPC logs, Syslogs). Data processing and analytics were tedious tasks that involved manual creation of jobs and queries via the user interface. These manual processes led to a high cost in terms of time and effort, and delayed the insights security teams could get from their data.

Customer Solution

In response to the operational problems, Crest Data created an integration solution to automate data processes and enable connectivity between the cloud-based data intelligence platform and the client’s SIEM.

Key components of the solution include:

  • Automated Data Ingestion & Parsing: Crest Data developed custom notebooks to automate the ingestion and parsing of key security data, such as AWS CloudTrail, AWS VPC logs, and Syslogs from S3 buckets into the platform. This saves users from having to develop these integrations themselves.
  • Bidirectional Data Movement: Data can be pushed and pulled between environments, enabling users to push data to Splunk for ingestion and pull data from Splunk into platform tables for further analysis.
  • Custom Integration App: Crest developed a new app that allows administrators to manage platform tasks directly from their SIEM environment, minimising the need to access the instance.
  • Custom Commands: Three special custom commands have been developed to expedite processes.
    • databricksquery: Enables users to query data stored in platform tables directly from the SIEM.
    • databricksrun: Allows the submission of one-time runs without having to create a formal job on the platform.
    • databricksjob: Permits the execution of pre-configured jobs and notebooks through the SIEM command line.

Outcomes 

The integration with the cloud data intelligence platform led to several outcomes in terms of security and reduced technical complexity:

  • Automation of Manual Tasks: The development of dedicated notebooks successfully automated the ingestion and parsing of key logs (such as AWS CloudTrail, AWS VPC logs, and Syslogs) from S3 buckets. This eliminated the need for the customer to previously set up integrations and create notebooks.
  • Minimized Manual Work: The automated data ingestion and parsing process eliminated the need for manual effort in data preparation and analysis.
  • Streamlined Centralized Management: Creating a Splunk app allowed administrators to run queries and jobs from the SIEM. This was achieved through the use of the custom commands databricksquery, databricksrun, and databricksjob.
  • Reduced Instance Dependency: One of the primary outcomes was the reduced dependency on direct access to the platform instance. Security teams can now perform routine data processing and job execution without needing to navigate the platform’s native user interface.
  • Improved Data Connectivity: The solution enabled bi-directional data flow, enabling the ability for companies to push data of interest to Splunk for analytics, while pulling Splunk data into platform tables for processing.

About Crest Data

Crest Data is a data and AI-driven technology solutions provider for enterprises and technology innovators across cybersecurity and cloud security, helping them move faster and more securely. We help companies automate their security processes with sophisticated integrations that automatically ingest and interpret key telemetry. Our capabilities for developing custom SIEM solutions enable security operations to centralize the management of complex data environments, saving time and eliminating the need to access data platforms directly.