This video contains a discussion about big data and AI in detecting threats.
- [Instructor] What is big data? Well, you may have heard the term, but recently it seems to have become the buzz word inside cyber security. Big data is a term that's used to describe large and complex data sets that can't be analyzed using standard data processing applications and, therefore, specialized tools have been developed and fielded in our industry. There are a lot of challenges with big data, namely how are you going to conduct data analysis, data capture, data search, data storage, and data privacy for these large data sets that contain just so much information? For example, if I wanted to collect every purchase that has been made on Amazon.com for the past 10 years, that would be an enormous amount of data and it might break a standard database because of the sheer volume of records, but using these big data techniques, it wouldn't be a problem.
One of the most commonly used big data platforms is Apache Hadoop which is a collection of open source software that's used to facilitate the networking of many servers to solve problems involving these huge, big data sets. Hadoop doesn't require a defined structure like a database would with SQL. In fact with Hadoop, an individual server could fail, but the cluster could continue the entire computational process as if nothing ever happened. While the future is bright for big data, there are some issues that we as security professionals should be concerned with including the fact that organizations simply don't understand big data well enough yet.
It becomes really easy to introduce new vulnerabilities that we simply aren't expecting. Also, big data platforms rely heavily on open source software and that may contain malicious code like back doors or use default credentials. Many of these big data servers have not been properly hardened yet or even reviewed and this makes authentication of users or data access from other locations not nearly as secure as they could be. Also, because expertise in big data is lacking, you may find that poor coding practices occur, such as lack of input validation, that becomes commonplace in some of these applications.
With all that being said, big data is essential to the security monitoring of large enterprise networks. Our existing structured databases simply cannot hold enough information for our analysts to be able to consolidate all of our logs from every endpoint and server into a single repository. For that reason, it is important for our organizations to continue to look at big data solutions as we start moving forward with designing and building out our new security operations centers. Another key technology that must be embraced moving forward is the use of artificial intelligence and machine learning.
Our human analysts simply cannot search through all of our logs for unknown vulnerabilities with enough speed or accuracy to be agile enough to stop the threat actor. Instead, we must begin to rely on machine learning technologies to sift through this huge volume of data and then flag what might be important and present it to an analyst for review and decision. Artificial intelligence has become important for defenders as well, but attackers are also starting to use it for their exploitation. For example, Carnegie Mellon University has developed an AI named Mayhem that's able to search out a code base and look for zero-day exploits, determine if that exploit might work, and then conduct an attack autonomously.
This artificial intelligence program was created by security researchers at the University as part of a red teaming exercise that was happening under a government grant, but it does demonstrate the capability that could be used against our networks if this type of technology becomes commonly used by threat actors in the future.
- Best practices for conducting research
- Current threats and threat models
- Emerging social media platforms and their threats
- Integrating research into business functions
- Security activities across the systems and software development lifecycles
- Adapting solutions to meet business needs
- Collaborating with programmers, sales staff, facilities managers, and others
- Providing guidance to senior management