Update: HPL R&D work on Big Data for Security

At HP Labs we are making good progress in our R&D work on “Big Data for Security”, aiming at identifying new security threats and issues from large amounts of collected data logs.

We have already identified a few key threats happening within organisations, including last-generation malware infections as well as misconfigured devices. We are actively looking for trials with HP customers, whilst aiming to transfer our new technologies in next generation HP SW security solutions.

Our current work focuses on massive amount of DNS data collected from DNS servers and aiming to subsequently correlate with other types of data logs.

We leverage key HP HAVEn assets to provide integrated real-time and historical analytic solutions along with compelling visualization. Specifically we currently use HP ArcSight Logger, HP ArcSight ESM, HP Vertica and HP TippingPoint RepSM as the underlying “big data infrastructure” to build our security analytical framework.

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

Big Data for Security @ HP Labs: Key Milestone Achieved

In the Big Data for Security R&D project, at HP Labs, we achieved an important milestone. We delivered our first, fully working prototype (and related demonstrator) illustrating how  it is possible to analyse Big Security data to identify potential (new) security threats and issues of relevance to organisations.

We focused, as a case study, on DNS events: DNS logs are usually huge, due to the very large amount of DNS queries (and replies) performed per second. As a consequence, companies usually fail in logging this type of information or they restrict the collection/retention to very small time periods. On the other hand, DNS Infrastructure is critical and can be used to launch attacks and/or for criminal intents.

Hence, being able to analyse DNS logs (potentially in conjunction with other logs) is key to identify attacks and misbehaviours.

Our demonstrator analyses DNS logs (currently only DNS queries, in the near future also DNS replies) and provides insights about potential security threats and issues. This is achieved via Historical (Security) Analytics and Visualization capabilities developed at HP Labs.

We fully leverage current HP Software and Security (HAVEn) solutions, Including HP ArcSight Logger, HP ArcSight ESM, HP Vertica and HP RepSM.

In the coming months we aim to:

·         Refine this solution by including advanced anomaly detection functions, trend analysis and machine learning, coupled with compelling visualization;

·         Process a wide range of data types, beyond DNS logs (e.g. web proxy logs, IPS logs, vulnerability scanning logs, user access logs, etc.)  along with related analytics;

·         Process and analyse unstructured data, by leveraging HP Autonomy;

·         Leverage distributed analytics solutions (including Hadoop) and advanced statistical tools (e.g. R).   

This is work in progress. We are currently showcasing this solution to HP customers and partners to gather additional requirements and feedback. More to come in the coming months.

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

 

Big Data for Security: On Using DNS Logs for Security Threat Detection

I am particularly interested in the area resulting from the intersection of the following topics: big data for security, big data analytics, distributed programming and data analysis solutions, security and cloud.

 

In particular I am interested in public case studies, business cases and trials involving the usage of (large amounts of) DNS data to detect new security threats and issues.

 

Here are some key related work and approaches:

·         EXPOSURE: Finding Malicious Domains Using Passive DNS Analysis;

·         Large scale DNS analysis

·         Identifying malicious domains in large DNS resolver datasets using Hadoop

·         Emerging trends around big data analytics and security: Panel

·         Botnet detection by monitoring group activities in DNS traffic

·         DNS Based Security Incidents Detection in Campus Network

·         DNS-based Detection of Scanning Worms in an Enterprise Network

·         Cache poisoning detection method for improving security of recursive DNS

·         Preserving the big picture: Visual network traffic analysis with tnv

  

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---