Why You Need A Centralized Crisis Management Center For Cyber Threats And Emergencies

Emergency preparedness and IT alerting expert Jeff Trask joins Danielle Ricci, Sr. Director of Marketing for AlertFind, to discuss the benefits of creating a unified alerting center for both emergency and IT notifications.

In our upcoming webinar, he’ll discuss how companies can use this approach to protect their business critical operations, decreasing their response time and ensuring they can immediately every employee.

Here are some highlights from their conversation:

Danielle Ricci: So Jeff, tell me about the unified alerting center you set up when you formerly worked at a Fortune 500 company.

Jeff Trask: We were an all-hazard shop. So we did everything. We did everything from the basics like service outages, infrastructure outages, to outages caused by cyber incidents.

When you think about setting up a response system like that, you really have to look at it from an all-hazards standpoint. We took a unique approach and said “In the private sector, your biggest business impact is an IT outage, regardless of the cause."

Whether it's a human error, whether it's just a failure of the infrastructure application, external forces, etc., that’s where you can have direct impact to your reputation, financial impact and productivity impact.

So we looked at our alerting from an all-hazards approach. We had mechanisms in place to escalate all of those types of outages, whether it be from the customer, from IT, from the business or the cyber security team. So all those mechanisms were in place to allow for a proper escalation for an incident.

Danielle Ricci: I know a lot of companies have two different systems. They have the IT alerting system that lives within the IT organization, and then emergency notification system which is run by the emergency preparedness team. Why is it traditionally set up that way?

Jeff Trask: I think it all depends on the way both functions mature within an organization. So in some organizations, your emergency preparedness lives in facilities, security or risk management. And then in some organizations it lives in IT. So depending on where that function matures, that’s where that functionality is going to live.

Same thing with your IT alerting. IT is obviously going to live in that department, and they feel the need sometimes to develop their own alerting/engagement/paging-type systems. So depending on where that falls within an organization, it could live in multiple locations.

In some organizations, it's all together. And that's where it makes sense to have one solution that's going to handle all those incidents. Generally, the IT organization has a 24-hour presence. So this type of alerting system tends to gravitate toward those 24/7 presences or functions just because most people don't want to be on call, with their finger on an alerting system button. They want to give that functionality to somebody who has the training and the tools, the connectivity and the availability.

Shortening the response time is really your ultimate goal with any type of incident. On the IT side, it's tracking the time to restore service. On the all-hazards business-continuity side, it’s how quickly can we restore those critical business functions. So, in general, those two are obviously related.

Danielle Ricci: So does moving to a centralized alerting center also deliver cost savings?

Jeff Trask: There are obviously cost efficiencies, when you have two different alerting systems and you want to move to one platform, or integrate them both into one platform.

Danielle Ricci: Are there any particular industries or types of companies that you think benefit from this approach?

Jeff Trask: No. I think, honestly, any private sector entity can benefit from this approach. You just mentioned that having two systems leads to duplication of effort and cost. From a private-sector standpoint, again, IT is so integral to critical business functions.

I mean, when we do business continuity planning, we look at critical systems, personnel, facilities, key vendors. And obviously, IT falls into two categories, right? It falls under critical systems supporting your business functions and your vendor-hosted applications, that kind of thing. Or it could be private infrastructure, cloud services, etc. So, it doesn't matter what type of business it is. It could apply to anybody.

Danielle Ricci: So how do you staff this center? What types of people and skillsets are you looking for?

Jeff Trask: Honestly, the thing I look for when I staff a center like this is somebody who's good at facilitation, communication and resource management. Because what are they doing, right? They're communicating that an outage has occurred. They're letting people know so they can take action or engage in protective action. They can engage and facilitate getting the response, the right people engaged. They can re-prioritize work.

So if they can't use a particular application, they’ll notify the employees so that they can use something else or go to a manual mode in order to keep things moving. They get all the right people engaged. They're working the incident and driving it to resolution. So they're a facilitator and they're a resource manager.

Danielle Ricci: Do they need additional industry-specific skills in order to tackle whatever the problem is?

Jeff Trask: So if you can find somebody who's a good facilitator, communicator and resource manager, that's ideal. Obviously, you try to find somebody who has a working knowledge of the organization, who knows basically who the players are and who to engage. But other than that, I think those are the skills that I'm looking for.

Danielle Ricci: And I would think, too, because they have to do all these things, they need to be a good relationship builder. They need to have good relationships -- both with IT and then with security or facilities management.

Jeff Trask: Right - they need good relationships with IT, facilities, security and the business units.

Danielle Ricci: How many people do you need?

Jeff Trask: It varies based on the organization. It depends on what your volume is, how many incidents you have, right? So if you're a combined shop where you're all hazards and you have a lot of systems in play, you might need half a dozen people.

If you're a smaller shop, I'd say at least two. I think the working best practice is two. Because if you're running a 24/7 shop, you don't want to have somebody alone overnight or if something happens from a health and safety standpoint or if things just get really crazy. You always want to have two people in there.

Danielle Ricci: What’s a typical day like in the alerting center?

Jeff Trask: When there's nothing going on, they're really just kind of monitoring things, keeping an eye on the systems using whatever monitoring or alerting tools they have. They're constantly listening, so they have phones there that are hotlines that people can call to report problems. Maybe they're keeping an eye on what the help desk is doing in terms of their call volume.

Are they seeing any trends? They're looking at external situational awareness, resources, state and local information sources, the news, right? If there's any natural disasters, any protests or bomb threats, whatever, in relation to their location.

They're gathering data. Somebody's going to call. An incident's going to happen. Somebody's going to report it and immediately, they're going to communicate out, right? Let people know what's going on to the best of their ability. They’ll communicate what it is, where it is, and what people can do to take a protective action.

So in the event of an IT outage, they need to know what system is down? What is the outage? Is it a total outage or a partial outage? What's the name of the system and are there any workarounds at this time? Can you use another system instead?

So you try to get that information out. And then, as the incident matures, you continue to communicate to provide additional information. Meanwhile, you're paging out all the appropriate resources to respond to that incident.

And at the same time, you're sending out communications to the business and the leadership. Letting them know what's going on depending on how big it is. And then, they just keep going through that cycle, right? Engaging additional resources, troubleshooting and then, keeping that communication line active until the incident is resolved and everything is restored to normal and they communicate a closure.

Danielle Ricci: We’ve talked about designing your plans, your emergency response plans, doing training, exercising those plans. How does your alerting center play into that?

Jeff Trask: When you do a drill or exercise, it's always important to exercise all elements. So you might have an emergency operations center set up for the really big incidents, but it has to start somewhere. That first call has to come through your alerting center and let the organization know what's going on.

Things like this escalate, so if they're dealing with some kind of issue at that ground level, they might be able to, within their own resources at the alerting center, resolve that incident without having to escalate it.

Or it could be something that's really big or causing a big impact on the business and then, at that point, you need so many additional resources, you activate an emergency operations center. So there's that escalation that happens. So I think it's important to test it right from the first phone call all the way through the mobilization of the EOC through the closure of the incident.

Danielle Ricci: So just like you would escalate an IT incident, you can actually escalate your alerting hub.

Jeff Trask: Just like in a city, you have a 911 center. Somebody's still going to call in and say, "Hey, the tornado touched down," and at that point, the center is going to initiate internal alerting mechanisms that are going to formalize an EOC and engage additional resources.

Meanwhile, those first responders are still going to be working the actual issue. Just like in an IT situation that’s causing a huge financial impact or is keeping employees from doing their job. The alerting center can be dealing with those issues while the EOC you create handles the bigger issues.

Danielle Ricci: Are there any common obstacles that companies need to be looking for? And how can they resolve those?

Jeff Trask: One of the biggest obstacles are silos. If you have one alerting function living in one organization and the other alerting function living in another, it’s going to be hard to function well.

People don't like change, they like to have ownership over things and they don't like to give up control. So that's always a challenge.

Cost is another big area. If you don't have a really mature organization, and you have to stand up the center, get the right tools and technology, and hire the right staff, that’s going to be costly.

You make a business case. Regardless of what the incident is, you look at your time to resolution. And if you can implement a process that results in a quicker time to resolution, say you go from hours to minutes, think about how much money you saved in lost productivity.

What if you’re in manufacturing? If your systems are down, you can't produce product, you can't make money, right? Most of the time when you do a cost-benefit analysis, the cost of staffing a center like that, or equipping a center like that, is far lower than what you’re losing from downtime and lost productivity. By handling the small incidents quicker, and of course the big incidents quicker, you’re going to save money.

Danielle Ricci: So we've talked a little bit about staffing in terms of personnel. What about any specific tools or equipment that they need to budget for as well?

Jeff Trask: Yeah, I think obviously you want to give them the right IT resources, you want to give them the right situational awareness tools, and you want to give them the right alerting tools.

It's all going to vary by the organization, but you have to do an assessment and figure out what tools are going to help you have insight into your system. There's tools for monitoring network performance, there's tools for monitoring storage capacity, applications, cyber threats.

The best thing somebody can do is just do that kind of assessment, figure out what tools are available and get them implemented. IT tools, alerting systems, monitoring tools, and then you've got your situational awareness, right? Like newsfeeds and alerts from the CDC, NOAA and other federal and state agencies that tell you about your natural and man-made disasters.

Danielle Ricci: Are there any resources that you'd recommend for companies who want to implement these types of centers? Any place you can go for blueprints, roadmaps?

Jeff Trask: There’s no easy blueprint. You've got to do what's best for your organization. I think you have to look at what you're trying to accomplish and what approach would be the best for your organization.

Look at the goals of what you're trying to accomplish, and then look at how your company is structured and what resources you have, and then do your best to build a model that works.

Join Jeff and Danielle as they discuss how creating a unified emergency and IT alerting strategy can improve response times and protect business operations in our new webinar, “Are Cyber Threats And Emergencies Separate Parts Of Your Alerting Strategy? Why You Need A Centralized Crisis Management Center,” at 2 p.m. EDT on Thursday, June 7.