Big Data for Security @ HP Labs: Key Milestone Achieved

In the Big Data for Security R&D project, at HP Labs, we achieved an important milestone. We delivered our first, fully working prototype (and related demonstrator) illustrating how  it is possible to analyse Big Security data to identify potential (new) security threats and issues of relevance to organisations.

We focused, as a case study, on DNS events: DNS logs are usually huge, due to the very large amount of DNS queries (and replies) performed per second. As a consequence, companies usually fail in logging this type of information or they restrict the collection/retention to very small time periods. On the other hand, DNS Infrastructure is critical and can be used to launch attacks and/or for criminal intents.

Hence, being able to analyse DNS logs (potentially in conjunction with other logs) is key to identify attacks and misbehaviours.

Our demonstrator analyses DNS logs (currently only DNS queries, in the near future also DNS replies) and provides insights about potential security threats and issues. This is achieved via Historical (Security) Analytics and Visualization capabilities developed at HP Labs.

We fully leverage current HP Software and Security (HAVEn) solutions, Including HP ArcSight Logger, HP ArcSight ESM, HP Vertica and HP RepSM.

In the coming months we aim to:

·         Refine this solution by including advanced anomaly detection functions, trend analysis and machine learning, coupled with compelling visualization;

·         Process a wide range of data types, beyond DNS logs (e.g. web proxy logs, IPS logs, vulnerability scanning logs, user access logs, etc.)  along with related analytics;

·         Process and analyse unstructured data, by leveraging HP Autonomy;

·         Leverage distributed analytics solutions (including Hadoop) and advanced statistical tools (e.g. R).   

This is work in progress. We are currently showcasing this solution to HP customers and partners to gather additional requirements and feedback. More to come in the coming months.

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

 

On Policy Decision Support for Big Data

When dealing with big data (inclusive of hybrid and unstructured one), it is very hard to understand the implications and impact of defining (security, business, sharing, privacy, etc.) policies on this data.
Which data is actually affected by the policies? Are these policies comprehensive? Are there corner cases that are not covered? Further complexity is introduced by the fact that analytics can be performed on big data, whose outcomes and implications are unknown at priori, as well.
Decision support tools are required to help policy makers to explore the implications of defining policies on big data and related analytics. In my view, these tools must provide synthetic visualization of big data as well as real-time feedback on the implications of defining specific policies and related constraints.
I am looking for:

- Tools providing synthetic, visualization of big data as well as potential analytics

- Public documentation, approaches, case studies, solutions, etc. describing how businesses currently cope with the consequences of defining (security, business, privacy, sharing, etc.) policies on big data



--- Posted by Marco Casassa Mont (here and here) ---

--- NOTE: use this mirror blog if you prefer posting on an external blog site ---

--- NOTE: my original HP blog can be found here ---

Frameworks for Graphical Visualisation of Policies

I am looking for public information about case studies, frameworks and/or approaches to graphically visualise policies. In particular, how to convey data sharing policies by using graphical metaphors.


Policies are increasing getting more and more complex: not everybody can makes sense of them and/or translate them into practical/actionable terms. On the other hand, they are used in many digital contexts (web services, enterprise, B2C, cloud, etc.) to dictate constraints, SLAs, expectations and obligations.

I am interested in exploring how graphical visualisation can help to:

• convey them to end-users in a more intuitive way ...

• enable better reasoning about their meaning and implications

• allow administrators to translate them into enforceable activities/constraints



--- Posted by Marco Casassa Mont (here and here) ---

--- NOTE: use this mirror blog if you prefer posting on an external blog site ---

--- NOTE: my original HP blog can be found here ---