As security threats continue to worsen and evolve, real-time visibility into big data could be the answer to keeping networks and endpoints secure without having to allocate untenable amounts of human resources to the problem, according to ESET.

ESET senior research fellow Nick FitzGerald says, “Businesses have expanded their security approach to a point where they have sensors and security applications throughout the network, feeding back massive amounts of data from the endpoints, network, servers, and perimeter. All that data needs to be aggregated, correlated, and analysed so that the business can make the right security decisions. Unified threat visibility and management is the goal for organisations looking to improve their security posture.

Reports suggest that it can take an average of 146 days to detect a cyber breach and up to 81 percent of reported intrusions aren't detected by internal security processes but by news reports, law enforcement notifications, or external fraud monitoring. This is far too long for businesses to detect and remediate threats. They need to move faster to combat breaches but are hampered by a lack of resources and visibility.

While there is potentially plenty of security data available in the enterprise, there are significant challenges in analysing that data effectively.

For example, network defenders monitor and detect threats as they attempt to pass through the network. Network packet capture creates terabytes of data every few hours, which makes it practically impossible to separate the useful data in real time.  Endpoints also generate data, albeit with a different structure and nature, which means it needs to be analysed differently from network data.

Availability data is different again; it monitors the system health of the enterprise and can easily give false positives on harmless activity. Separating the useful data from the extraneous is complex, and it's not practical or reasonable to expect the IT security team to be able to do this manually.

“It seems logical to simply aggregate all the data but even then, turning that data into insights that can be acted on is a huge task. The time it takes to analyse the data adds to a business's mean time to detection (MTTD) and mean time to remediation (MTTR)," FitzGerald says.

“Using a big data solution can help by deploying clusters of machines that can scale and ingest the data. Businesses need to be able to search unstructured data to gain useful information. This should be augmented by devices that can eliminate useless data and identify false positives, thus usefully reducing the amount of data without human intervention. Once that's done, it's essential to include reporting capabilities so that IT and security personnel can quickly and easily understand what actions to take to secure the enterprise."

“Visibility is the holy grail for network security, along with being able to access, analyse, and act in real time.”

By gathering all the relevant data and analysing it using big data techniques, businesses can gain the unified visibility they need in real time, so they can act faster to remediate and mitigate threats to the network. Doing so effectively and efficiently could be a game-changer for cybersecurity.