Standardizing Security Data Onboarding with GOLD Parsers to Maximize Threat Visibility
Executive Summary
A leading cybersecurity telemetry platform encountered a major issue with its in-built parsers, which had poor parsing performance as they did not support all event types across all log sources. Moreover, the existing parsers were custom-made for specific customers, resulting in multiple versions that demanded maintenance work by the partners. The customer felt a strong need for GOLD parsers that provide comprehensive coverage, ensuring maximum capture of information from raw logs and enabling mapping to the Unified Data Model (UDM) to fully leverage the platform’s rules and threat-detection capabilities.
To overcome such hurdles, Crest Data teamed up with the customer to use and develop Chronicle GOLD parsers that standardized data onboarding and data parsing of huge volumes of data. The team created various parsers for various vital log sources, such as Windows, Zeek, PAN, Cisco, and Office 365, and created effective ingestion systems, such as syslog and APIs. Crest Data was able to efficiently normalize and map raw logs to UDM fields, enabling faster onboarding of more than 150 platform end-customers, resulting in enhanced searchability and threat detection while improving the platform’s overall scalability and cost-efficiency.
About Customer
The customer is a global leader in information technology, specializing in Internet-related services and products. As part of their major cloud ecosystem, they offer a specialized cybersecurity telemetry platform dedicated to threat hunting and threat intelligence.
Customer Challenge
The customer’s platform ingests security telemetry data that must be normalized into a Unified Data Model (UDM) format using in-built parsers. However, these parsers posed various limitations:
- Low Parsing Rates: The built-in parsers were inefficient due to the lack of support for all event types of all log sources.
- Maintenance Overhead: Most of the available parsers were custom-made and specific to a particular customer. There were several versions of the same parsers, which required maintenance work from the partners.
- Coverage Gaps: There was an immense need to have “GOLD Parsers” that ensured comprehensive coverage in both breadth (all types of events a given source of logs can generate) and depth (the specific fields in those events).
- Underused Functionality: The platform was not able to fully leverage its advanced capabilities, like detection rules and threat intelligence, without the capability to extract the full information out of raw logs and map it precisely to the UDM.
Customer Solution
In an attempt to overcome the shortcomings of current parsers and achieve inclusive data coverage, Crest Data partnered with the customer to create and deploy GOLD parsers. This solution standardized the onboarding process for high volumes of data and ensured high parsing rates across multiple log sources.
The most important technical aspects of the solution were:
- GOLD Parser Development: Crest has developed different high-quality GOLD parsers to support a wide variety of log sources, such as Windows, Zeek, PAN, Cisco, and Office 365.
- Log Source Analysis: The team performed in-depth analysis of various security telemetry and log sources to get a deep understanding of each type of data.
- Advanced Ingestion Mechanisms: Crest developed effective ways of collecting information, using syslog through Chronicle Forwarder, Ingestion APIs, and other third-party applications.
- UDM Normalization and Mapping: The solution entailed the normalization of raw logs carefully and mapping user context and information to the Unified Data Model (UDM) fields.
- Rule and Script Definition: To optimize threat detection capabilities of the platform, Crest described detection rules and complex parsing scripts in UDM fields.
- Lifecycle Integration: Crest engaged in the entire product release lifecycle and ensured strict adherence to quality standards of parsers before they were rolled out.
Outcomes
The collaboration between Crest Data and the customer led to the substantial streamlining of operations and enhancements in the security capabilities of the platform:
- Quick Onboarding: Creation of GOLD parsers led to more rapid onboarding of all security telemetry to the platform.
- Wide Customer Impact: The solution increased parsing rates across multiple log sources for more than 150 customers of the platform.
- Improved Threat Detection: Through comprehensive mapping of a greater number of UDM fields, Crest Data ensured the creation of accurate and improved detection rules.
- Enhanced Visibility and Search: The enhanced coverage of security telemetry resulted in enhanced search, data view, and general threat detection.
- Operational Efficiency: The customer benefited substantially in terms of scalability, cost-efficiency, and complete data coverage.
- Standardized Data Ingestion: The project helped standardize data onboarding and parsing strategies for huge volumes of data from all the log sources.
About Crest Data
Crest Data is a data and AI-first product engineering and technology solutions provider with deep expertise in cloud and AI, cybersecurity, observability, data analytics, and workflow automation. In this case study, Crest Data successfully leveraged its security data engineering and specialized parser development capabilities to enable the customer to standardize and scale the onboarding of massive security telemetry datasets through the creation of high-quality GOLD parsers, supported by robust ingestion mechanisms, meticulous UDM field mapping, and the implementation of advanced detection rules for over 150 customers.
With 1,200+ experts and a track record of 5,500+ successful projects across 150+ global customers, and backed by strong partnerships with Google, AWS, Microsoft, Datadog, Dynatrace, ServiceNow, and NetApp, Crest Data delivers outcome-focused solutions that strengthen security, improve platform reliability, and enable sustainable digital growth.
