Update: HPL R&D work on Big Data for Security

At HP Labs we are making good progress in our R&D work on “Big Data for Security”, aiming at identifying new security threats and issues from large amounts of collected data logs.

We have already identified a few key threats happening within organisations, including last-generation malware infections as well as misconfigured devices. We are actively looking for trials with HP customers, whilst aiming to transfer our new technologies in next generation HP SW security solutions.

Our current work focuses on massive amount of DNS data collected from DNS servers and aiming to subsequently correlate with other types of data logs.

We leverage key HP HAVEn assets to provide integrated real-time and historical analytic solutions along with compelling visualization. Specifically we currently use HP ArcSight Logger, HP ArcSight ESM, HP Vertica and HP TippingPoint RepSM as the underlying “big data infrastructure” to build our security analytical framework.

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

On HP Vertica Analytics Platform 7

HP recently announced the release of the HP Vertica Analytics Platform 7: http://www.vertica.com/hp-vertica-analytics-platform-7-crane/

Aiming to leveraging it in our HPL “Big Data for Security” R&D work.

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

My Tweets of the Week (18-24 November 2013)

My tweets of the week (18-24 November 2013) at https://twitter.com/MCasassaMont:

·         How NASA Manages Big Data - http://www.informationweek.com/big-data/hardware-architectures/how-nasa-manages-big-data-/d/d-id/899791 … #bigdata #in

·         Working on next generation HPL "Big Data for Security" R&D solutions. #security #bigdata #in

·         Are large scale Man in The Middle attacks underway? - https://isc.sans.edu/forums/diary/Are+large+scale+Man+in+The+Middle+attacks+underway+/17075 … #security #in

·         More Data Scientists, or Fewer Complex Big Data Applications? - http://www.wired.com/insights/2013/11/more-data-scientists-or-fewer-complex-big-data-applications/ … #bigdata #in

·         Gartner analyst slams OpenStack, again - http://www.networkworld.com/news/2013/112013-gartner-openstack-276187.html?hpg1=bn … #cloud #in

·         2014 - Malware to reduce but risks to rise - http://www.scmagazineuk.com/2014--malware-to-reduce-but-risks-to-rise/article/321783/ … #security #in

·         HP announces Vertica 7 'Crane' update for better big data insights - http://www.v3.co.uk/v3-uk/news/2307924/hp-announces-vertica-7-crane-update-for-better-big-data-insights … #in

·         Warning! Targeted Internet misdirection on the rise - http://www.computerworld.com/s/article/9244173/Warning_Targeted_Internet_misdirection_on_the_rise … #in

·         NCA warns UK of mass CryptoLocker ransomware attacks - http://www.scmagazineuk.com/nca-warns-uk-of-mass-cryptolocker-ransomware-attacks/article/321576/ … #in

·         The three universal questions companies ask about big data - http://www.citeworld.com/consumerization/22693/andrew-mcafee-big-data-three-questions … #in

·         HP: 90% of Apple iOS mobile apps show security vulnerabilities - http://www.networkworld.com/news/2013/111813-hp-ios-vulnerabilities-276063.html?hpg1=bn … #in

·         10 reasons the browser is becoming the universal OS - http://www.infoworld.com/d/applications/10-reasons-the-browser-becoming-the-universal-os-230812 … #in

·         IT Security Faces Big Data Skills and Resource Gap - http://www.informationweek.com/security/database-security/it-security-faces-big-data-skills-and-resource-gap--/d/d-id/899770 … #in

·         What big data projects often lack: a return - http://www.federaltimes.com/article/20131117/IT03/311170002/What-big-data-projects-often-lack-return … #in

·         Cyber attack emergency service launched - http://www.scmagazineuk.com/cyber-attack-emergency-service-launched/article/321222/ … #in

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

Big Data for Security @ HP Labs: Key Milestone Achieved

In the Big Data for Security R&D project, at HP Labs, we achieved an important milestone. We delivered our first, fully working prototype (and related demonstrator) illustrating how  it is possible to analyse Big Security data to identify potential (new) security threats and issues of relevance to organisations.

We focused, as a case study, on DNS events: DNS logs are usually huge, due to the very large amount of DNS queries (and replies) performed per second. As a consequence, companies usually fail in logging this type of information or they restrict the collection/retention to very small time periods. On the other hand, DNS Infrastructure is critical and can be used to launch attacks and/or for criminal intents.

Hence, being able to analyse DNS logs (potentially in conjunction with other logs) is key to identify attacks and misbehaviours.

Our demonstrator analyses DNS logs (currently only DNS queries, in the near future also DNS replies) and provides insights about potential security threats and issues. This is achieved via Historical (Security) Analytics and Visualization capabilities developed at HP Labs.

We fully leverage current HP Software and Security (HAVEn) solutions, Including HP ArcSight Logger, HP ArcSight ESM, HP Vertica and HP RepSM.

In the coming months we aim to:

·         Refine this solution by including advanced anomaly detection functions, trend analysis and machine learning, coupled with compelling visualization;

·         Process a wide range of data types, beyond DNS logs (e.g. web proxy logs, IPS logs, vulnerability scanning logs, user access logs, etc.)  along with related analytics;

·         Process and analyse unstructured data, by leveraging HP Autonomy;

·         Leverage distributed analytics solutions (including Hadoop) and advanced statistical tools (e.g. R).   

This is work in progress. We are currently showcasing this solution to HP customers and partners to gather additional requirements and feedback. More to come in the coming months.

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

 

On cyber attacks leveraging DNS: where are the numbers?

DNS is a critical infrastructure: it is often leveraged to launch attacks and/or for criminal intents. Recent news further demonstrates that the impact and consequences of such cyber attacks could be extremely serious.

I am looking for public statistics, reports, analysis, etc. that quantify the amount of cyber attacks  that directly or indirectly exploit DNS infrastructure. Any help (links, documents, etc.) would be really appreciated.

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

My Tweets of the Week (02-07 September 2013)

My tweets of the week (02-07 September 2013) at https://twitter.com/MCasassaMont:

·         Aussie startup Packetloop turns big data APT forensics into Arbor Networks success - http://www.cso.com.au/article/525707/aussie_startup_packetloop_turns_big_data_apt_forensics_into_arbor_networks_success/ … #in

·         How Big Data Is Transforming The Mobile Industry - http://www.businessinsider.com/local-mobile-makes-big-data-real-2013-9 … #in

·         What's Next for IPS? - http://isc.sans.edu/forums/diary/What+s+Next+for+IPS+/16508 … #in

·         Looking for public statistics about cyber attacks that exploit/leverage DNS infrastructure and related capabilities #in

·         3 Lessons CMOs Take Away From IT's Flawed Approach to Big Data - http://www.cio.com/article/739057/3_Lessons_CMOs_Take_Away_From_IT_s_Flawed_Approach_to_Big_Data … #in

·         10 defining moments in tech-labor history - http://www.infoworld.com/slideshow/117920/10-defining-moments-in-tech-labor-history-226080 … #in

·         Preparing for HP customer meeting. Presenting and demonstrating our R&D work on Situational Awareness and Big Data for Security #in

·         Exploring the Big Data Stack - http://www.datamation.com/data-center/exploring-the-big-data-stack.html …

·         Big Data's Big Puzzle - http://www.forbes.com/sites/christinemoorman/2013/09/03/big-datas-big-puzzle/ … #in

·         10 hard-earned lessons of a lifetime in IT - http://www.infoworld.com/slideshow/116989/10-hard-earned-lessons-of-lifetime-in-it-225919 … #in

·         Big data security - http://www.it-director.com/technology/big-data/content.php?cid=14440 … #in

·         A Closer Look into the Future Big Data Ecosystem [INFOGRAPHIC] - http://smartdatacollective.com/bigdatastartups/141626/closer-look-future-big-data-ecosystem-infographic … #in

·         Tor Use Uptick - http://isc.sans.edu/  #in

 

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

 

On Big Data for Security and Analytics Technologies

I found an interesting report, by Enterprise Strategy Group, called “The Evolution of Big Data Security Analytics Technology” providing an overview of the market landscape.  In my view, two key areas are open to research and innovation:

 

·         Real-time big data security analytics

·         Asymmetric big data security analytics

From a security perspective, whilst an increasing number of new tools, solutions and frameworks are emerging in these areas, there are a few key challenges that need to be addressed.  They relate to the quality and effectiveness of big data analytics: how to make sense of big (security) data and provide meaningful insights in order to identify new threats and security issues; how to capture this knowledge into repeatable and simplified capabilities that can be used by a wide range of stakeholders.

 

These are going to be key R&D aspects to be addressed in the future, in addition to the well-known big data issues i.e. dealing with velocity, volume and variety of data.

 

In particular HP and HP Labs are in a unique position to make progress here, also thanks to opportunity to leverage key HP assets in the space of security and big data management i.e. HP ArcSight, HP Vertica and HP Autonomy.  

 

 

 

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---

 

 

My Tweets of the Week (05-09 August 2013)

My tweets of the week (05-09 August 2013) at https://twitter.com/MCasassaMont:

 

·         3 Ways To Develop A Big Data Strategy - http://www.forbes.com/sites/centurylink/2013/08/08/3-ways-to-develop-a-big-data-strategy/ … #in

·         RAM wars: RRAM vs. 3D NAND flash, and the winner is...us - http://www.computerworld.com/s/article/9241488/RAM_wars_RRAM_vs._3D_NAND_flash_and_the_winner_is...us … #in

·         Enterprise whales leave R&D to the startup minnows - http://www.theregister.co.uk/2013/08/09/you_will_be_assimilated/ … #in

·         IT's new role as big data stewards - http://www.techrepublic.com/blog/big-data-analytics/its-new-role-as-big-data-stewards/ … #in

·         The 7 Steps Of Big Data: How To Make It Work - http://www.forbes.com/sites/netapp/2013/08/07/7-steps-of-big-data/ … #in

·         How big data platforms should work - http://www.infoworld.com/t/big-data/joyent-cto-how-big-data-platforms-should-work-224299?page=0,0 … #in

·         The Need for Speed with Big Data - http://www.businessweek.com/articles/2013-08-06/the-need-for-speed-with-big-data … #in

·         Detect the undetectable: Start with event logs - http://www.infoworld.com/d/security/detect-the-undetectable-start-event-logs-224173 …? #in

·         Banks seek big data, analytics and security start-ups for tech accelerator - http://www.zdnet.com/banks-seek-big-data-analytics-and-security-start-ups-for-tech-accelerator-7000018986/ … #in

·         Malware-as-a-service blossoms in Russia, vendor research finds - http://www.networkworld.com/news/2013/080513-lookout-malware-272528.html?hpg1=bn … #in

·         DMARC: another step forward in the fight against phishing? - http://isc.sans.edu/diary/DMARC+another+step+forward+in+the+fight+against+phishing+/16297 … #in

·         10 big data trends changing the face of business - http://www.infoworld.com/slideshow/113454/10-big-data-trends-changing-the-face-of-business-223969#slide1 … #in

·         Big Data strategy essentials for business and IT - http://www.ft.com/cms/s/0/e1a68da8-fb98-11e2-8650-00144feabdc0.html#axzz2b5Eqopic … #in

·         A strategic approach to cloud integration - http://www.networkworld.com/news/tech/2013/080213-cloud-integration-272502.html?hpg1=bn … #in

·         There's a tide of unstructured data coming - start swimming - http://www.theregister.co.uk/2013/08/05/unstructured_data/ … #in

 

 

--- Posted by Marco Casassa Mont (here and here)  ---

--- NOTE:  use this mirror blog if you prefer posting on an external blog site  ---

--- NOTE:  my original HP blog can be found here  ---