Turning Alert Fatigue Into Alert Awareness

Wim Remes, founder at Wire Security, has witnessed up-close the daily avalanche of security alerts that has buried the teams he has managed under tens of millions of extremely urgent, and largely irrelevant, warnings each day.

While the most trivial of those security alerts can be handled with automation tools, one of Remes’ teams battled a persistent backlog of thousands of security events and could not keep up.

“The entire team was trying to work down the backlog full-time,” he says. Companies that fail to keep up, as he has seen, are unlikely to successfully stop the barrage of cyberattacks targeting their data and systems, at least not for long.

This isn’t a problem that’s going to get better anytime soon. The high number of alerts has been made worse during the COVID pandemic, as enterprises made most work remote. If not adequately managed, the constant ringing of alarm bells simply wears analysts out.

[Read also: How to Respond to a Cyber Breach]

“The human toll of the last year has been tremendous,” says Martin Fisher, director of information security and chief information security officer at Northside Hospital in the Atlanta area. “We are jumpy. Every alert becomes a minor incident-response investigation. That level of effort is exhausting. People are burning out.”

Many security tools issue alerts that don’t provide enough context to understand what’s happening. Sometimes it’s the tool’s fault. Sometimes it’s a lack of staff training. “If this were an easy problem, it would have been solved long ago,” says Chris Blow, director of offensive security at Liberty Mutual.

To bring down alerts to a level humans and systems can manage, companies must make the leap from alert fatigue to alert awareness. They can do this by fine-tuning security tools and sensors, focusing on the alerts that matter, limiting analyst fatigue with automation, bolstering staff training and proactively strengthening their security posture.

Tune your security devices

This is the obvious step, but it’s not necessarily an easy one. Security tools that are used to monitor networks and systems are finicky. Tune them too tightly and they become positively chatty with too many alerts; tune them too loosely and security teams risk missing real attacks.

You can’t just set them and forget them, says Blow at Liberty Mutual. Yet that’s how many enterprises treat them. He says the most successful companies have processes in place that involve everyone necessary for handling alerts, including the relevant application and system owners, as well as those who work in the SOC. They meet weekly or more frequently to keep up with fast-changing threats.

Prioritizing alerts, and making sure everyone understands those priorities, goes a long way toward solving alert fatigue.

Remes of Wire Security recommends creating a tight feedback loop between the environment that security tools monitor and the configuration of the tools that issue alerts. “Your security monitoring tools need to be tuned to the alerts you want to see,” he says. To do this, Remes recommends using a risk-based framework that enables enterprises to apply cyber risk assessment decision-making to the devices, applications, networks, and users they want to protect.

Prioritize alerts

Not all event detections are the same. While alerts in the backlog need to be analyzed, not all of them need to be investigated immediately. “Prioritizing alerts, and making sure everyone understands those priorities, goes a long way toward solving alert fatigue,” says Blow.

To better set priorities, Blow says it’s essential to consider factors such as the source of the alert, the volume of alerts, the size of the event’s potential impact, the progress of the possible attack, and the resemblance of an incident to previous incidents. Fisher of Northside Hospital also prioritizes sensitive areas like patient safety and experience, as well as malware and ransomware that threaten business operations.

Automate the right alert response

Some alerts can be addressed by machine. For instance, there’s no need for a priority alert to human teams for a user who is locked out of their account after too many failed login attempts and then needs a password reset. But while one or two failed logins don’t need to be a high priority, says Blow, “if the alert triggers multiple times over a day, it could then be escalated.”

[Read also: 5 Ways Your Company Can Avoid becoming a Headline]

Other areas that can be automated include phishing attack analysis. “When someone reports a phishing email, you can analyze the data and interact with your mail infrastructure to remove similar emails from other users’ mailboxes,” Remes says.

But be careful about automating bad processes, advises Remes. “What you don’t want to happen is that your automated processes are not understood,” he says. “Going from understanding what is happening, to understanding what a team needs to do, to automating those processes is not a fast trajectory,” he says.

Train staff on the latest security tools

“The number of tools has grown exponentially,” says Blow.

Too many businesses try to deploy products and hope that solves their security risk. The tools end up poorly managed, if they get used at all. Instead, organizations may need to invest in the professional services they need to get their tools up and running effectively—and to make sure they have the right amount of training for security staff.

“Too many organizations just ignore services and adequate training, and it shows in their implementation,” Blow says.

Get ahead of the threats

As the alert backlog begins to clear, it’s crucial that a security operations team proactively scan the network for possible signs of compromise before alerts are ever triggered. If a particular criminal group has been targeting your industry (such as the recent attacks on healthcare), then you should search for indications they are already inside your own IT environment.

[Read also: Tool Sprawl Threatens Post-Pandemic Security]

“When we are successful at detection, we don’t even need the alert because we can just block whatever technique they’re using,” says Blow.

Beating the alert to the punch sounds like an unattainable goal, and without proper training and services and platforms it probably is. But it’s a goal worth setting.

How did Remes’ overwhelmed team eventually work through its backlog? It optimized the environment as much as possible, and divided and conquered the backlog over time. “We have five analysts,” he says. “Every analyst takes one day a week to focus on the alerts backlog. That means each analyst gets four days a week to focus on other things.”

It worked. The team’s backlog is now down from 3,000 to 80 alerts. But like everyone else’s, keeping the backlog down is a constant battle.