EclecticIQ Platform Technology Overview

Unified, normalized, de-duplicated and enriched Cyber Threat Intelligence, based on open-source OASIS standards STIX and TAXII.



Based on STIX and TAXII standards.

Written in Python. Uses Elastic, PostgreSQL and Neo4j.

Powerful visualization and graphing tools, plus comprehensive browse and search.

Create your own workflows, extensions and custom components.

Send real-time CTI information into IT security controls.

Ready for on-premise deployment, into your environment.

Overview

EclecticIQ Platform Deep Dive

Technical Design

Modular. Extensible. Easy to run.

Inspired by OASIS CTI standards STIX and TAXII, EclecticIQ Platform is a multi-tier, distributed Linux-based system built in Python using the latest cutting-edge technologies and mature open-source components.

Internals: REST APIs. Redis in-memory data structures. JSON data format built with Flask. Celery framework for asynchronous task queuing. Gunicorn WSGI HTTP server. NGINX high-performance HTTP server.

Data Layer: Built with 'polyglot persistence' that switches between PostgreSQL for main data storage and Elasticsearch and Neo4j for specialized use cases including full-text search and graph traversal.

CTI Data Model: Entities are based on EclecticIQ's interpretation of STIX 1.2, resulting in a precisely-crafted set of patterns, idioms and decisions informed by experience working with real-world data and clients. 

Observables Data Model: A separate data model for Observables supports data fusion and greater depth of connections between disparate data points. 

Graphical User Interface (GUI): A fully-developed GUI expands the role of the CTI analyst.

Command Line Interface (CLI): Supports simple integration and scripting tasks.

REST API: Supports integration with enterprise platforms, third-party solutions and scripting applications.

Python API: Build your own extensions for incoming/outgoing feeds and enrichments.

Data Processing

Normalized, de-duplicated and enriched data.

With consistency and flexibility, EclecticIQ Platform normalizes and transforms CTI information from multiple sources and formats into one unified data model. Built-in content transformers and a flexible Python API allow you to define your own mappings and content transformations.

Ingestion: Takes in structured sources, including industry and proprietary standards (see "Standards" section below), along with unstructured sources such as plain text, and PDF documents.

Normalization: Transforms CTI information from multiple sources and formats into one unified data model. From a "single pane of glass" management console, analysts have complete data consistency, absolute alignment across data sources, types and formats.

De-duplication: Consolidates data using a package-based approach, which examines the contents of the entire content package to discover identical matches within individual fields; and an entity-based approach, which compares external reference IDs.

Enrichment: Additional processes discover correlations, aggregate similar records, and draw in details using external-facing APIs to intelligence community data sources.

Versioning: To manage the life cycle of CTI, EclecticIQ Platform includes built-in support for entity versioning, a concept inspired by STIX 1.2.

Deployment

On-premise deployment.

All-in-one deployment, on your infrastructure: All components, internal and external, run on a single node. We provide a set of RPM packages, powered by standard or custom builds of CentOS or RedHat 7. 

All-in-one deployment, on the cloud: Same as above, using your choice of cloud provider (although not recommended for production, Amazon AMI is also available for test and development).

Distributed deployment: We support a distributed deployment model, giving you the flexibility to optimize components across multiple nodes. 

Scalability and Extensibility

Scale up, down, out and in.

Don't outgrow (or overdo) your Threat Intelligence Platform. Whether in terms of functionality, network throughput or storage capacity, EclecticIQ Platform has the flexibility to match your precise needs. 

Scale up/down (vertically): Upgrade, add or subtract resources in a single-node, all-in-one deployment.

Scale out (horizontally): Add nodes to a distributed deployment to achieve specific performance metrics.

Extend functionality: Add content transports, parsers, seralizers and enrichers. 

Integrate with IT security controls: Includes built-in SIEM integrations (Splunk, IBM QRadar, HPE ArcSight) and connectivity with Snort signatures, Yara signatures and CybOX patterns.

Standards

Automate workflow with industry standards 
and proprietary APIs.

EclecticIQ Platform maintains data internally using EclecticIQ JSON formats, and is fully compatible with industry standards, simple formats such as CSV, and a full range of proprietary APIs for both input and output. 

Industry standards:

  • ArcSight Common Event Format (CEF)
  • Malware Attribute Enumeration and Characterization (MAEC)
  • OASIS CTI standards: STIX, TAXII, CybOX
  • OASIS Customer Information Quality (CIQ) standard

Proprietary APIs: 

  • Anubis Cyberfeed
  • CISCO AMP ThreatGRID
  • Exodus Intelligence
  • Farsight DNSDB API
  • FireEye iSIGHT API
  • Flashpoint
  • Fox-IT
  • Group-IB
  • Intel 471
  • Wapack Labs Threat Recon

Open-source leadership

Creating essential tools for the CTI community.

One of EclecticIQ’s top priorities is to promote adoption of STIX and TAXII, the OASIS standards for CTI. As part of this effort, we have created two open source projects: 

Cabby

TAXII client implementation developed in Python as a command line tool. 

Features include: 

  • Support for all TAXII services according to TAXII specification (v1.0 and v1.1)
  • Version agnostic: Abstracts specific implementation details and returns entities that will work in any version of the standard.
  • Stream parsing: High-bandwidth “Poll Service” messages are parsed on the fly, lowering memory requirements and reducing the time until content is available to the end user.

Visit Cabby project

OpenTAXII

Robust Python implementation of TAXII Services that delivers a rich feature set and a developer-ready API. OpenTAXII implements all TAXII services according to TAXII specifications (version 1.0 and 1.1)

Additional functionality includes:

  • Customizable APIs
  • Authentication 
  • Flexible logging

Visit OpenTAXII project


Resources

STIX 2.0 - Build your own intelligence

White paper

STIX 2.0 - Build your own intelligence

This White Paper introduces STIX 2.0, compares it with the existing STIX 1.2 architecture and shows the future for STIX 2.x.

Download White Paper
EclecticIQ Platform - Product data sheet

Data sheet

EclecticIQ Platform - Product data sheet

Data sheet of EclecticIQ Platform containing Technical Architecture and Technical Details, including Incoming Feeds, Enrichment Services, Outgoing Feeds, Analytic Toolkit, and Administrator tools.

Download data sheet

More about EclecticIQ Platform

Subscribe to our newsletter

By signing up you will receive our quarterly newsletter on Cyber Threat Intelligence. Read the latest issue of our newsletter here.

Thank you!