Tag Archives: Incident Management

Understanding How Your Help Desk and NOC Services Connect

This post about the Help Desk and NOC was initially published on CX Expert. It has been updated to reflect current market trends and information.

Perhaps, there is no other obstacle that causes more concern to managed services providers than how they are going to manage, monitor and provide services to their clients. For companies that are getting into managed services or cloud, one obstacle must be tacked. This obstacle is what you will do about your help desk and NOC. You must determine the best practices to follow, the cost, and many other essential things.

Origin of the NOC

The help desk and NOC is the heart of managed services providers. It is capable of providing redundancy, physical security and an area that is secure for collaboration for technicians to be able to manage and monitor customer environments.

Every managed service provider (MSP) service delivery model needs the NOC. However, many people don’t know where the idea of the NOC came from and how the early MSPs came to rely on them. With the advent of cloud computing and other business process and technological advances, it is essential to know how different factors influenced the NOC for the cloud provider and MSP today.

The idea of the NOC is not new – it has been around for a very long time. Initially, it was a creation of telecommunications, and it was used to monitor and manage telecommunications networks. Technicians sit and receive information in real-time inside the NOC. The NOC’s physical configuration allows for an intimate, secure and safe place for technicians to collaborate and discuss with other technicians on issues and problems that would otherwise be unsafe or unsuitable around other non-approved personnel.

MSPs came into existence in the mid-1990s. During this time, nearly all companies had a business plan. The plan included a 27/7 operational NOC that was physically secure from which to deliver their managed services. These configurations, procedures, and tools used in each of the MSPs were different, but the presence of the physical NOC was an essential and consistent characteristic.

The Help Desk and its Importance

A lot of people confuse the terms help desk and NOC when they serve two different and vital functions. The main reason behind the confusion is easy to understand when you look back at how they both came into being and how they have been split apart gradually in modern time managed services conditions.

We have already discussed the role of NOC, but it is important to look at how the help desk fits in the equation. Older NOCs have the help desk integrated into them to maximize the benefits of security, redundancy and collaborative work environments. The simplest and easiest way of distinguishing and defining the two models is system-based work, and the NOC performs network around the management and monitoring of objects that are under management with the MSP. On the other hand, the help desk is more responsible for interfacing with end-users, and it is also a customer-facing department. It helps to respond to problems and get solutions.

It is easy to see why the two areas seem to be similar. Each company should make these choices on their own, but it is important to understand that there is a difference between the two and they service unique and important functions within managed services practice.

The Interaction Between the Help Desk and the NOC

Now that we know the difference between help desk and the NOC, we must look at how they should interact with each other in managed services practices. It is important to acknowledge that there are different ways in which help desk and the NOC interact and co-exist.

Helpdesk Existing Within the NOC

Having a help desk and a NOC should be evident. All MSP’s logical and physical security controls can be addressed in one physical space. The physical access to workstations, how technicians log in to systems belonging to clients and change management are important to control and can be monitored effectively and be enforced if the NOC has the help desk residing into it.

There are more configurations where the MSP can create an entire floor of their work premises, where the help desk teams and the NOC teams work, complete with secure access, albeit in sections within the secure area. The main purpose of mingling the help desk and the NOC is to take advantage of the process efficiencies and security. If you take the security of your operation into account in the same way, there is no need to build out different facilities. Instead, you can just build it once.

There are many other benefits to have the configuration. Any interaction between the help desk and the NOC is naturally easier when they are both located within the same secure area. This can help with redundancy, and the continuity of the business plans should anything happen to the facility.

The Help Desk Existing Outside the NOC

In larger MSP environments, you will find the help desk outside the NOC. This is common, but it will depend on your unique situation. In most cases, you will find large companies with the help desk located outside the NOC because the help desk team won’t fit within the NOC. Typically, MSPs that need to have multi-time zone help desks or multilingual helpdesk do not manage the MSP centrally, but they have several help desk facilities that are located in different locations.

It is not always needed to build full NOC around all the help desk areas. Therefore, help desks exist outside the NOC. Regardless of how you operate and configure your NOC and help desk, there should be enough controls that deal with how the two elements in your business interact with one another. There should be documentation on the handling of trouble tickets, the handling of the connectivity within the MSP and how the redundancy of power is handled within the MSP organization to maintain operational effectiveness.

Building a NOC or Outsourcing

This post was initially published on CX Expert. It has been updated and revised based on current technology and changing business requirements.

Regardless of which side of the issue you are, there are a few people who would dispute the fact that the managed services model of facilities offers the most control and security for the client. The reason behind building a NOC should be evident.

If the NOC is built and operated correctly, it will provide a secure and enclosed space for the managed service provider (MSP) to be able to view its managed services operations. You can think of it as a mission control center where managed service providers can view the IT network of the client and manage all the activities that take place across an MSP’s entire network of clientele. The NOC has come to represent one of the characteristics of MSPs for many years. Companies can still rely on software such as Freshdesk CRM to ensure good results.

For the NOC to be effective, it must possess certain features. The features tend to be similar when you compare them side by side. For the MSPs who are starting out, there are cost-effective methods to help them achieve each of these essential features.

Security

It is necessary for a NOC to be secure. However, most people might find it difficult to understand what this means. A lot of smaller MSPs have visions of expensive and large NOCs. These preconceived notions cause smaller businesses concern and many never undertake to build their NOC.

A NOC does not have to be expensive and cost millions of dollars for it to be effective. It is essential to have a NOC enclosed. It should be limited to the employees who can access client networks. Also, the facility should have security controls to ensure that only the authorized employees can enter the NOC. This simply means that you must have locked in place and have systems or people to watch the technicians who come in, work and leave the facility.

If a business is or deals with credit cards for example, the requirement is quite similar and the PCI or PA DSS certifications require a secure facility.

Redundancy and Backups

It is worthless to build a NOC on the sand. The NOC represents the security, stability, and integrity for those MSPs who possess one as well as the clients who rely on them. The NOC should be well-built.

This means that it should take into consideration the natural order of things. This includes how unforeseen problems can happen to throw an MSP’s entire world into problems. Some natural occurrences such as storms, earthquakes and power disruptions can cause havoc on an unsuspecting managed service provider.

Backup power and redundant internet connectivity are just a few of the features that will make a NOC effective and viable.

Sharing of the minds

A good NOC should be able to do more than provide physical and network security. It can offer a managed service provider with something more important than the technological benefits. This means that the MSP’s technicians working in a central place should collaborate to solve problems.

Physically proximate technicians to one another can work together and solve issues concerning client and share information that can help in educating the MSP. As a managed service provider, one of the most important traits is expertise and education related to business and technology. Being in the same room is more superior to sharing information through the email.

Building a virtual NOC

A lot of MSPs choose to forgo building a NOC that is physical in favor of building a virtual NOC. Maybe you don’t know what is involved in constructing a virtual NOC because it is very natural. For you to answer this crucial question, it is important to consult with the UCS which is the Unified Certification Standard because this is the template that exists and addresses how the MSPs should employ a virtual NOC in an environment of managed services.

The certifications of managed services and audits outline the use of virtual NOC and the operation guidelines for technicians as they travel. Operating a virtual NOC and MSP comes with the challenge of maintaining adequate security. As the virtual NOC technicians leave or enter the office, traveling to client sites and going to other locations, keeping physical security, accurate logs and maintaining managed services of high standards can be difficult. Therefore, the MSP should implement certain protocols that will ensure the integrity of client data. Using encryption technologies, poison pill technology, strong authentication and other forms of common sense should be cheap and small MSPs should be able to afford them.

This means that a technician who takes a laptop that was used to monitor a client should be guarded both outside the MSPs office and inside. The managed service provider should, therefore, have a mechanism for auditing the work done by its technicians irrespective of their location. In short, audit procedures and security employed in running a NOC that is physical should also be used to operate a virtual NOC. The redundancy backups and physical security are the only things missing which only a physical NOC can offer.

Partner vs. Build

This is an important issue that consumes the attention of a lot of young entrants to the managed services profession. There is the opportunity to build a virtual NOC that is robust although there are some MSPs who still decide to partner with existing managed service providers in order to leverage that manage services provider’s NOC. Partnering with another MSP to access a physical NOC should not be taken lightly. It is important to know all the facts and make informed decisions for your practices.

Young MSPs come into the partnerships for the reason of having nothing to do with the availability of a physical NOC. It is normally a reason for lack of confidence and business process. However, MSPs should weigh the decision because the wrong choice can lead to unwanted repercussions. It is difficult to advise everyone on the right path in the long run because companies have unique situations. However, if a business decides to partner with a managed service provider who has a NOC, then it is important to choose a partner that has passed the audits.

How to Build a NOC

This post was initially published on CX Expert. It has been updated and refreshed before being republished here.

A network operations center (NOC) is the central point of monitoring for your network. It helps to ensure uptime in your business. NOCs are not limited to networks. They can provide visibility into systems management, virtualized infrastructure, IT security and many more. In most cases, only large companies have resources necessary to create a NOC infrastructure that is effective. However, even small and medium businesses can gain visibility into the performance and availability of their networks by creating their own NOCs or using tools that NOCs use.

 For you to create your NOC, you don’t need to have a modern room that is full of expensive and high technology gear for network surveillance. It is possible to create your NOC just about anywhere and have the ability to know where and how network issues occur in time for troubleshooting. We will look at some essential capabilities to build into your network management system and get a NOC field of vision.

Centralize Alert M

When dealing with a growing network with a lot of devices from different manufacturers, alerts can be useful. You should be able to receive them in a single and central location for actionable and easy access and insight.

Alerts may include but are not limited to:

  • performance metrics,
  • availability statistics,
  • errors message,
  • hardware thresholds, and a host of other relevant factors.

The primary challenge with alert management lies in receiving alerts on time and managing them at a centralized level to help in comparing alerts, tracking alert history, eliminating false positives and deducing alert functions.

Group your Network Elements

It is possible to have network hardware in different models, types, versions, from different and unique makers in different locations. They are compatible with different platforms. It is therefore difficult to get an understanding of the network issues given the medley of network devices. The best solution is to create logical groups of your devices. This will allow you to monitor your devices as a group and not disparate entities. You can also create static groups to help you add network nodes manually. You can also add network nodes automatically, but this is based on a pre-defined condition.

You can get a logical understanding of the situation by using grouping devices for monitoring networks. You can also use grouping to set parent-child dependencies between your network elements, and this can allow you to eliminate any redundant alerts and be able to understand the impact of any faulty device on its dependents.

Customize Your Network Diagnostics

You may find it hard and frustrating that even though you have the ability to get network performance and health, you are not capable of dissecting the information as fast as needed. This ties directly to the KPIs you are measuring but in many cases, this problem arises due to the dashboard view that is used to see the network performance data.

You need to use a web-based dashboard because it is the best and can be viewed from anywhere. The ability to customize the dashboard will also allow you to understand your data easier and faster because you will see the things that need attention fast such as:

  • The top interfaces by traffic
  • The top errors and discards
  • The top interfaces by traffic
  • Top interfaces that face maximum percent utilization

Map device topology

You are sometimes faced with the work of searching for the reason that caused your network to go down without any clue if you are a network administrator. However, it is straightforward to pin the problem on a map and trace the source of the problem. You just have to use the mapping topology. This will help you to monitor the availability by looking at a map. You can use the following steps to do it.

  • Discover the nodes of your network such as interfaces, network devices, and servers
  • Place the nodes of your network on a custom map
  • Connect the elements of your network on the ARP table data to get a graphical depiction of both virtual links and physical links.

Unify Management Platforms

The budget is not the only thing involved but also the operational expertise, and management overhead are also needed to run different platforms for network management for different requirements. You can cut back on time and money if you unify the management platform.

Unifying your platforms can also give you a comprehensive view of the NOC functions. Therefore, it is important to look for a solution that can stand alone and still be compatible with other management modules for systems management, network configuration management, and virtualization management. You can simplify your operations if you have the same management platform. It also allows you to customize your interface and doesn’t require a lot of work to manage your NOC.

Make sure you can access the network performance monitoring data of your organization from the comfort of your workstation because this can be the most effective NOC that any network admin could ever have. Also, it is crucial to have the right NOC dashboard to get a comprehensive view that is always available to you. This clearly shows how the network devices are doing and the things that are causing your network downtime. You don’t need a chief network engineer when designing your NOC.

Conclusion

If your company manages multiple networks, then you should be aware of the challenges involved when it comes to monitoring them at once. The data from your clients and your data is private, and your networks should be running without any delays.

Unless you have a time in your business that can handle this high level of network management, you may want to outsource professional support. You can use NOC engineers and technicians to monitor the health of your infrastructure, the capacity of your infrastructure and the capacity of your infrastructure.

With all this important information, they can make informed decisions for your business and adjust the systems to optimize the productivity and performance of your organization. They will send out alerts in case of any issues based on the type, severity, and level of expertise needed to solve the issues and any other things your NOC team specifies. After resolving the issue, you can alter things in your system and monitor your system to prevent the problem from recurring.

The Impact To Your Business of “Unplanned Work”

You can’t feel it, you can’t see it, you can’t touch it, but unplanned work is silently killing your business. How many times have you finished your day at work, very exhausted yet unable to cross anything off your list of high-priority to do things? This can make you feel robbed. It is impossible to tell where your time went to. This is a sign that you are falling victim to unplanned work.
 
Unplanned work is not evident in our metrics of performance, so it is difficult to analyze. However, the impact of unplanned work is great. It can mask dependencies and block and stall important priorities. Risk will accumulate across your system if you receive more work and start it late. The risk will continue to yield more risks in the form of unknown dependencies, neglected work, too much WIP and conflicting priorities. This can jeopardize the ability of your organization to deliver predictably.
 
 

Effects of Unplanned Work

You can encounter unplanned work in 2 different flavors:
  • The requirements that you have to change in the middle of a project
  • The defects you find during testing. 

You can deal with both of them if you have an agile team that is run well. It is important to understand the cause of unplanned work if you want to make your project cheap and predictable.

 
It is also important to understand that unplanned work steals valuable time from planned work. Most people see unplanned work as a norm instead of seeing it as a significant problem. 

When solving a problem, you should ensure that the problem will not occur again, but if you solve the symptoms of a problem rather than the problem, then the problem is likely to occur again. The main problem that is left unattended will lead to more issues which will lead to the allocation of more resources, and this can worsen things.
 
Every organization, department, team, and individual should measure the amount of unplanned work that is being performed. You can even forget about solving problems that come later and get a sense of how much time is allocated to work that isn’t adding any value to the business and rather spent on fighting the status quo. 
 
Let us consider release management as a concrete example to explain unplanned work. You take a piece of software and deploy it to the market and spend a lot of time trying to diagnose why a certain release didn’t work as expected. The time you spend trying to diagnose the problem is unplanned work. You can easily solve some of these issues through deployment, continuous integration and automated testing. You can be proactive and resolve a lot of (future) unplanned work as you try to diagnose and debug why things didn’t work as you had expected.
 
It is difficult to get rid of unplanned work, but we can easily learn how to plan for it. The only concern is how to plan and tackle unplanned work. Before we go deep and discuss how you can handle and mitigate unplanned work, it is important to discuss incident management, problem management, and post-mortems because they can help plan for the unplanned.
 

Incident Management and Problem Management

Problem management and incident management are key components of the Information Technology Infrastructure Library service model, and they have been created to provide a more streamlined service to consumers.
 

Incident Management

Incidents are things related to customer contacts. They can be account updates, information request, and issue reporting. There are different methods that incidents can be reported through, and this includes email, phone, and chat. Incidents can also be generated by automatic monitoring tools. There are different incidents that come through/from different sources. They would get routed to your tool for incident management. This could be something simple for smaller teams, but larger teams may need enterprise level tools or in-house customer-built applications.
 
Your business should have the responsibility of reviewing information about different incidents and check if there is a solution available to the customer. An example of an incident is when a customer wants to change their account password. The helpdesk will receive the incident, get the necessary information and make sure the customer has passed all the security checks, facilitate the changing of the password and inform the customer that the password has been changed and then close the incident. Some incidents can be managed with automatic tools while others have to be managed manually.
 

Problem Management

Incident management is repetitive in nature and can get tedious. This can exhaust the more skilled employees in your organization so if you have such employees in your organization, consider moving them into managing problems.
 
Problem management is deeper than incident management. This is where a single problem causes multiple incidents from multiple clients or customers. Problem management needs the best people. The role of these people is to find out why a certain problem happened and find the best ways to fix it and prevent the problem from happening again.
 

Post-mortem

A post-mortem is usually performed after a project has been concluded. The process helps to determine and analyze different elements of the concluded projects that were successful or unsuccessful. The main purpose of project post-mortem is to inform improvements in processes to mitigate future risks. This helps to promote best practices in an organization. Post-mortem helps to manage risks in an organization.
 

Mitigating unplanned work

As we had already seen, unplanned work is time-consuming, expensive and can negatively impact other projects in your organization. 
 
It can also drain all your skilled resources. This can be the rarest and the most important resources you have in your organization. This is why unplanned work hurts more. Two main methods you can use to mitigate unplanned work are:
  • Widening the bottleneck to moving configurations down and upstream without having to tie down your constrained skill base.
  • Increasing communication flow between producers and developers to relevant changes.

 

When you widen the bottleneck, it turns hours of work into a few minutes. If something goes wrong, you can always roll back and mitigate the impact of unplanned work. You can widen the bottleneck by creating a build of configurations, automating activities of migration of configurations and creating other jobs.
 
When the flow of communication is increased, your developers will know any changes to production that impacts their activities directly. They will be able to run comparisons instantly against critical systems and see if they have to consider specific changes as they perform their work. This helps to prevent failed deployments or reduce the likelihood of failed deployments. 
 

Customer relationship management

This is somewhat self-explanatory. Customer relationship management (CRM) is a strategy used by a business to execute business objectives and meet the requirements of their customers. 
 
The service strategy can be used to improve the customer relationship strategy within a business and ensure that the business can create value for its shareholders and customers by contributing to the value. The strategy ensures that businesses are able to organize their operations in an appropriate manner to deliver services that will enable the success of customers. 
 

Document Objectives

 
The business objectives are the results you want to maintain and achieve as you run your business. Every business should have clear and attainable goals. 
 
The goals may include productivity, profitability, customer service, core values, employee retention, growth, change management, marketing, maintain financing, competitive analysis and more.
 

Document Requirements

If you want to arrive at a solution that will ensure the continuity of your business, you need to map out every application; server, data, and software solution set in your environment. You should assign a downtime tolerance that is business-driven for each of the requirements. Map out the application interdependencies as you do this.
 
 

Test Your DR Planning

 
Disaster Recovery is never a onetime event. This is a constant process, and it is important for disaster recovery to keep up with the changes in the environment and evolving service-level agreements. The truth is that the data center is rapidly changing and it is almost impossible for the change control processes and operations to keep up. It is also very hard to conduct disaster recovery tests with enough frequency to be meaningful. This leads to most companies considering disaster recovery and tools for monitoring to allow analysis that is a near-real-time of the disaster recovery processes and setup.
 

 

Every business should conduct disaster recovery tests to determine the people who need more training and also know the disaster recovery processes that need to be refined. Some of the main disaster recovery things that a business should have include monitoring, environment awareness, hardware, and software independence and work from a knowledge base.