Collaborative Threat Intelligence Sharing and Automated Information Exchange - Insights from the JTAN Project Experience

| July 1, 2024

The JTAN (Joint Threat Analysis Network) Project, co-funded by the European Union’s CEF program, addresses the critical need for efficient and effective threat intelligence sharing among cybersecurity teams. As cyber threats grow in complexity and scale, the ability to quickly exchange and analyze threat data across organizations has become essential for maintaining robust security.

This document outlines the practical methods and tools developed and tested by the JTAN Project to facilitate this exchange. These include MISP Synchronization for real-time data sharing, REST APIs for automated integration with other systems, and real-time streaming solutions for immediate response scenarios. Additionally, advanced techniques like privacy-aware caching using PSS and Bloom Filters are explored to enhance data sharing while maintaining confidentiality.

The strategies and tools detailed here are designed to be reusable and adaptable, particularly for networks of CSIRTs (Computer Security Incident Response Teams), SOCs (Security Operations Centers), and other dispersed groups involved in cybersecurity operations. By implementing these approaches, organizations can improve their collective situational awareness and better defend against emerging cyber threats.

Data Exchange Mechanisms and Strategies

By reviewing various open data exchange mechanisms and strategies, the JTAN project has tested and established a robust and versatile framework for sharing threat intelligence. This framework effectively caters to a diverse range of needs and scenarios in the field of cybersecurity. Additionally, these strategies are reusable and particularly beneficial for networks of CSIRTs, SOCs, or any dispersed group involved in operational security.

(Push/Pull) MISP Synchronization

This method involves the synchronization of threat intelligence information across multiple MISP (Threat Intelligence Sharing Platform) instances. It allows various organizations to share and collaborate on cyber threat data in real-time, ensuring that the latest threat intelligence is available across all synchronized MISPs.

(Pull cache) MISP Caching Synchronization

As an extension of MISP Synchronization, this strategy involves caching shared data to enhance the performance and reliability of data exchange. This approach is particularly beneficial in scenarios where MISP instances or caching sources are too large or out-of-context for direct and complete ingestion into MISP. Caching enables the correlation and availability of references, allowing users to determine if an indicator has been observed elsewhere without needing access to the entire dataset.

(OpenAPI) MISP REST API

Utilizing the RESTful API provided by MISP, this strategy allows for programmatic access to the MISP platform. It enables automated data exchange and integration with other systems or tools, enhancing the flexibility and scalability of threat intelligence sharing.

(Stream) Streaming using CocktailParty or N6

This approach is more dynamic, featuring real-time data streaming. Tools such as CocktailParty and N6 facilitate the streaming of threat intelligence, thus providing a continuous flow of information. This method is crucial in scenarios that demand an immediate response to evolving threats or the processing of a continuous data stream.

(Privacy-aware cache) PSS and Bloom Filter for Sharing and Data Exchange

PSS and Bloom Filters represent advanced techniques utilized for efficient data sharing and exchange. PSS enables scalable and decentralized data searching, while Bloom Filters offer space-efficient probabilistic data representation. This combination ensures the sharing and searching of large volumes of threat data with minimal resource consumption. Additionally, a notable aspect is the capability to share information while maintaining the confidentiality of the actual data.

API Documentation References

Tool API Reference(s)
MISP MISP OpenAPI Documentation
MISP Playbooks
N6 N6 Stream API
IntelMQ IntelMQ API
AIL AIL API
FlowIntel FlowIntel API
PSS PSS Metaformat
Graphoscope Graphoscope API Usage
Taranis-NG Taranis-NG Core ReST API
CocktailParty CocktailParty
MWDB MWDB REST API and MWDBLIB

Interoperability Review via MISP

The existing API integrations were tested to ensure that the tools commonly used in JTAN and the CSIRT community are interoperable with the MISP standard REST API. If the tools did not support a REST API, evaluations were performed on available compatible streaming protocols within JTAN, such as STOMP.

Tool API Reference(s)
MISP ✔️ Core testing protocol via REST API acting as a reference for the other tools
N6 ✔️ N6 Stream API relies on STOMP and can interconnect with IntelMQ and CocktailParty.
IntelMQ ✔️ The different IntelMQ models (Expert bot, API collector, API output, Expert) are supported to connect to MISP. Acting as a streaming tool, IntelMQ integrates N6 support.
AIL ✔️ Can push to MISP any investigation or item in AIL. MISP galaxies and taxonomies are also supported.
FlowIntel ✔️ Via FlowIntel Connector or Analyzer, data can be pushed to MISP and use the MISP modules as an analyzer. MISP galaxies and taxonomies are also supported.
PSS ✔️ PSS Metaformat is compatible with MISP standard feed format.
Graphoscope ✔️ MISP REST API accessible via MISP plugins
Taranis-NG ✔️ Presenter can create MISP standard JSON format and Publisher can create MISP events.
CocktailParty ✔️ Streaming via STOMP compatible with N6 CocktailParty
MWDB ✔️ via karton-misp-pusher

JTAN Project

The JTAN Project is co-funded by the European Union’s CEF program.

The Action will establish a Joint Threat Analysis Network, an open collaboration group of European computer security incident response teams (CSIRTs) focused on collecting, sharing, and analyzing technical, operational, and strategic threat intelligence. The purpose of this collaboration is to combine the unique advantages of different teams to obtain comprehensive situational awareness and actionable information to effectively defend constituencies in each Member State, from critical infrastructure operators targeted by state-sponsored actors to individual citizens affected by cybercrime. The main part of the Action addresses gaps in the Cyber Threat Intelligence (CTI) tooling that is currently used by national-level CSIRTs in Europe. By strengthening individual tools and interconnecting them, the beneficiaries will achieve a new level of common situational awareness and benefit from shared knowledge and tooling.

JTAN References