Threat Intelligence Integration Framework to Accelerate Incident Detection
Executive Summary
The customer wanted to enhance its capabilities to gather threat data from multiple threat sources. However, they faced a unique challenge: each threat source represented data in a different format, making normalization essential prior to ingestion. With a huge number of new threat sources to integrate, the customer required a standardized integration development process to efficiently handle huge volumes of complex and diverse data formats and ensure the information could be effectively normalized.
About the Customer
The customer offers an API-first cloud-native intelligence management platform. Using custom API solutions, it takes third party data and puts it to work, helping security teams speed up detection, response to incidents, collaborations, and investigative work.
Customer Challenge
As a premier threat-sharing and cloud-native intelligence management platform, the customer was facing the problem of inability to source data from various third-party sources and normalize it to help security teams to accelerate detection, incident response and investigation work. However, the inconsistent and fragmented nature of data posed significant problems for the customer.
Some of the problems identified are as follows:
1. Data Inconsistency: The data from each threat source was in inconsistent format making it difficult to ingest it directly without proper normalization.
2. Huge scale of integration: Since the new data sources that needed integration were huge, it became difficult to manually do integration of each data source.
3. Absence of Standardization: A properly defined integration development process was needed to standardize the normalization of data collected from a variety of sources.
Proposed Solution
Crest Data developed a standardized integration solution based on the customer’s framework to tackle the problems of inconsistent data sets and a wide variety of data source types. This helped establish a robust development process to source the data, normalize it and ingest into the customer’s platform.
Some of the core points of the solution are as follows:
- Development of Integrations: Crest Data developed robust more than 25 integrations categorized into Closed source and Open source types.
- Multi-step Data Ingestion Workflow: Crest Data formulated a multi-step data ingestion workflow to ensure a streamlined normalization of data.
- Source Analysis: The team analyzed various third-party data sources to identify from where Indicators of Compromise (IOC) data can be sourced for ingestion.
- Automatic Extraction: Custom integration development to pull data from a variety of sources using REST APIs and normalize it.
- Submission to the Platform: Submission of normalized IOC data pulled from various inconsistent data sources into the customer’s platform to perform various operations like threat detection, incident response and investigative work.
This integration framework allowed the customer to function as an intelligence management platform empowering their security team to make strategic decisions and perform various operations based on this data.
About Crest Data
Crest Data is a data and AI-first product engineering and technology solutions provider with deep expertise in cloud and AI, cybersecurity, observability, data analytics, and workflow automation. In this case study, Crest Data applied its CloudOps and DevSecOps capabilities to help the customer migrate from on-prem infrastructure to a secure, scalable, and cost-efficient AWS environment, supported by infrastructure automation and proactive monitoring.
With 1,200+ experts and a track record of 5,500+ successful projects across 150+ global customers, and backed by strong partnerships with AWS, Google, Microsoft, Datadog, Dynatrace, ServiceNow, and NetApp, Crest Data delivers outcome-focused solutions that strengthen security, improve platform reliability, and enable sustainable digital growth.
