A major PaaS provider offering a non-stop compute platform needed to automate their way around recurring issues to continue guaranteeing their millisecond data loss and recovery SLAs, giving them time to diagnose and remove the underlying problems.

I assisted one of the engineers from a major PaaS provider the other week so he could get back to having a good nights sleep.

The company he works for offer a non-stop IBM Power PC compute platform and he needed to automate activities his engineers were doing so his platform could be relied on to continue guaranteeing the millisecond data loss and recovery SLAs it was designed for.  They knew they needed time to diagnose and remove the underlying problems and so needed a fast and reliable way to fix the issues as they occurred in the meantime.

This blog describes what was done in this specialist environment but provides a great example of applying remediation automation in any environment. This service provider happens to offer an IBM PowerPC Platform as a Service to banks, utilities, telcos and the like making use of two clusters in two data centres, cross site replication provides High Availability and zero dataloss failover. The engineers use NMIS, opEvents and opConfig for managing the whole environment. NMIS is used to collect statistics, state and events from the IBM Power PC Cluster instances, NMIS also collects data from the member servers, the Fibre Channel and Ethernet switching and the SAN. Having NMIS meant visibility across the entire environment and all the way down to OS health particularly the underlying daemons, services etc on the Power PC cluster. In this case making use of NMIS’s Service management and plugin capabilities to monitor the IBM systems.

The team were making use of NMIS’s server management functions to collect state and performance data from several Nagios plugins for the PowerPC servers. NMIS and opEvents were successfully alerting the team to the fact that SVC replication was failing occasionally by sending notifications via SMS and Email to the right teams. The team were responding to these by following a process to restart the SVC service on the machines, of course this was usually in the middle of the night! They needed a way to automate this remediation task quickly so here is what they did in about 20 minutes to complete the work and without spending money.

First they read the Docs on Automating Event Actions in opEvents 

Next they looked at the SVC replication events in opEvents by looking at the details tab for one of the previous events. It was decided they only wanted this triggered if the alert was “Service Down” rather than “Service Degraded” and they only wanted it to happen if the service was down on the Primary site not if it was down on the Secondary site. In the details tab of the event they noted the following event attributes:
event => Service Down
element => svc_lun_inservice
host => primary_cluster.example.com

Next they tested a shell script they had used for restarting the svc service remotely, this was simply three remote ssh commands they had been issuing by hand, they placed it on the NMIS server in:
/usr/local/nmis8/bin/restart_svc.sh

Final piece of the puzzle was to call the script when the event details matched.

So editing the EventActions.nmis file
/usr/local/omk/conf/EventActions.nmis

The following was added:

1. A script action – added to the ‘script’ section of EventActions.nmis (copying one of the example scripts as an example) they created the follwing:
'script' => {
'restart_svc' => {
arguments => 'node.host',                           # this script takes the nodes IP or hostname
exec => '/usr/local/nmis8/bin/restart_svc.sh',
output => 'save',                 # save a copy of the script's stdout which echos what it is doing
stderr => "save",
exitcode => "save",
max_tries => 2,
},

2. A matching rule – added near the top of EventActions.nmis in the appropriate location (again the easiest was copying and editing one of the existing entries).
'20' => {
IF => ' event.event eq "Service Down" and event.element eq "svc_lun_inservice" and node.host eq "primary_node.example.com" ' ,
THEN => 'script.restart_svc ()', # note this name matches the the name in the script section above
BREAK => 'false'
},

Finally they needed a way to test it without actually doing anything or breaking anything. So they first edited the restart script slightly so it only echoed the key commands it would issue on the server. They then found a copy of the service down event in /usr/local/nmis/logs/event.log copied it to the clipboard changed the datestamp(epoch) to now and appended it back to the log
"echo {the editied event line} >> event.log"
and watched opEvents for the event to appear.

They then looked at the actions section of the event in the GUI and were able to see the script had fired and were able to see everything the script had done, from it’s output to it’s exit code to it’s stderr messages.

Finally they changed the script so it actually ran the commands and went home.

That night the svc replication failed – they didn’t get emailed or smsed about this time though as the system repaired itself immediately and before the 5 minute escalation time had passed. Job done.

In the meantime after a month of further diagnostics with the network carrier, the data-centre provider the SAN vendor and a team of others they found the underlying issue for the LUN replication, a timeout problem related to cross site Fibre Channel and the additional backup loads happening at night.  Couple of timers changed and all good.

Get the E-book here

With a growing dependence on the internet, many small and medium-sized organisations are opting out of managing their own networks and handing the reigns over to Managed Service Providers.  Managed services is a fast-growing industry that is expected to reach $193.34 billion with 12.5% CAGR by 2019 and Opmantek has long been recognised as a leading self-hosted RMM software provider to some of the biggest players in the industry.

In recent times, there has been a shift in the market to Software as a Service (SaaS) purchasing and many vendors now offer cloud-based solutions as a ‘simple’ software play for MSP’s. However, we have found that our customers require flexibility, and cloud is a one size fits most service that is not capable of supporting all network devices.  Every day we are getting more and more enquiries from MSP’s looking to regain control of their network by taking back ownership of their network management system.

Here are the top reasons our customers prefer on-premise or private cloud hosted Opmantek software over SaaS.

100% visibility and control

SaaS-based remote management systems are often a good option for micro MSP’s because of the ease of deployment and monthly subscription payments for each monitored instance that make budgeting easy until your client base starts to grow. The devices under management become more obscure as the size of the networks you are managing increase.  That’s when you start to lose network visibility, the event, health and performance data starts to lose quality and additional licensing costs associated with more complex network architectures begin to emerge.

Opmantek Software can be deployed in the cloud or on-premise but because you retain ownership of the database and have access to the source code at the core of NMIS, you maintain architectural, device and cost control.

Flexible integration

It is unlikely that any business will be implementing network management software for the first time when they come to search for a solution.  It is highly likely that they will already have a number of products performing different functions within their network environment and they may not wish to, or be able to, replace them all at once.

Opmantek supports a huge range of integration options through REST APIs (HTTP(S)), batch operations, information provided in JSON files or CSV forms.

Service Now API integration is tried and tested in both directions for incident feeds and CMDB asset feeds as well.

Opmantek offers end to end solutions, you can implement them in stages (or only implement some of the features) and be assured that whatever other platforms you are working with, you will have the ability to integrate and feed data between your systems and maintain complete visibility in your Opmantek dashboards.

Tailored Business Requirements and Ownership of data

One size fits all rarely works for large network environments, but it is this cookie cutter system approach that is standard with SaaS RMM software providers.

“Configuration over customization” has long been a core value of our development team – rather than creating new versions of software for individual customers, we co-create software features with our customers and make them available and configurable for the hundreds of thousands of organisations that have used Opmantek’s software over the years.  Almost every conceivable business requirement is catered for using configurations. Fault configuration, for example, can be adjusted in hugely flexible ways with our modelling policy engine, enabling one to make use of any metadata about a device to selectively apply fault and performance management.  opEvents enables event management, correlation, escalation, network remediation automation, network fault analysis automation to be configured in an infinitely flexible manner to suit customer difference and your NOC.

Unlimited Scalability

With the number of connected devices within every organization increasing exponentially over recent years the ability to grow and scale to meet the needs of a business, without eroding profits for managed service providers is becoming more and more critical.  As Managed Service Businesses grow, many SaaS providers force their users to compromise on the richness of data and troubleshooting capabilities by requesting paid upgrades to store more historical data.

At Opmantek we provide unparalleled scalability and ease of scaling.

We are highly efficient in our polling so you gain the most from each poller with thousands of devices per server instance even with full-featured polling. We also enable you to use as many pollers as you wish using opHA, we haven’t seen a limit of how many devices or elements one can manage on a single system. MongoDB provides cost-effective storage of as much data as you choose to store, making trending information and machine learning capabilities far more effective.

We are regularly approached to replace other major vendors products because they are unable to scale, enormously costly to scale or they lose functionality as they do.

Want to learn more about how Opmantek’s RMM solutions can increase your network visibility, deliver unmatched automation and save money for your Managed Service organization?  Click here to request a personalised demo from one of our engineers today!

How using the Gartner cyber security CARTA model can help secure customer data

The disclosure by hotel chain, Marriott, that the personal details of up to 500 million guests may have been compromised is a cyber security wake-up call for companies that store customer details—including in the cloud.

The potential theft of millions of passport details  ̶ reported on Friday, 30 November  ̶  could prove expensive. According to US magazine, Fortune, Marriott will offer to reimburse customers the cost, if fraud has been committed and customers need new passports.

For companies that store customers’ financial and personal details, the breach highlights two key issues that need to be addressed in corporate cyber security policies.

First, cyber prevention requires vigilance. The Marriott breach was detected more than two years after it first occurred. This is a sobering thought for chief information officers. Just because your systems and people have not detected a breach, that doesn’t guarantee that a breach hasn’t occurred.

The second issue is agility. Cyber security is a continuous arms race between cyber security professionals and attackers. The cloud is now extending that arms race into new dimensions. To stay secure, companies have to be fast-paced and stay pro-active. This involves a change in mindset.

Proactive mindset the key to cyber prevention

But what practical steps should your company take to avoid a similar breach? Most important is, don’t wait for a cyber security alert: look into new ways of detecting any breaches that may already have occurred.

And don’t rest easy. If you are a major corporate, it is safest to assume you are constantly being attacked—and that some attacks will succeed.

Four-step process to mitigate risk

To mitigate and manage similar cyber security risks, we recommend a cyber response process built around four key steps:

1. Prevention. Review firewalls and update controls to comply with the latest threat assessments. This includes a rigorous assessment of cloud-systems security.
2. Detection. Understand the control tools that identify attacks, and continually review them as you move more functions and data into the cloud.
3. Remediation. Work out now how you will respond if you discover a breach. This includes a customer-communications strategy.
4. Restoration. Figure out how you can restore a secure environment quickly if you discover that your data  ̶  or your customers’ data  ̶  has been compromised.

This four-step process is built on a methodology put together by Gartner, called the ‘Continuous Adaptive Risk and Trust Assessment’ (CARTA). Gartner provides a great 60-minute introduction to this approach, accessible with registration.

To stay secure, though, the key will always be vigilance. As companies move more functions and databases into the cloud, malware designers will refine their attacks. A continuous re-assessment of cyber prevention tactics will prove the most effective strategy in this ongoing cyber arms race. ​Talk to Roger and his team of experts today on +61 2 9409 7000 to find out more about protecting your business.

Globalisation, new technologies and digital business models are transforming the supply chain. Many businesses rely on organisations and individuals in different regions or countries to own the processes, materials or expertise used to provide a product or service.

However, malicious individuals or groups are increasingly aware that any supply chain is only as strong as its weakest link. If just one participant in a supply chain is lax about security, all businesses and individuals involved may be at risk.

Malicious parties may exploit weaknesses to steal valuable intellectual property, disrupt the creation or delivery of products and services, or threaten businesses or individuals for financial gain.

The United States National Institute of Standards and Technology (NIST) highlighted the importance of a cyber-secure supply chain in its Cybersecurity Framework. The latest version of the Framework – which provides voluntary guidance for organisations to better manage and reduce cyber-security risks – incorporates additional descriptions about how to manage supply chain cybersecurity.

Furthermore, a recent KPMG report points out “organisations that understand and manage the breadth of their interconnected supply chains and their points of vulnerability and weaknesses are better placed to prevent and manage issues.”

So what measures businesses can take to reduce cyber-security risks to their supply chains? Here are some steps that business owners and managers may consider taking:

• Provide security expertise and resources to all participants in their supply chain.
• Review participants’ processes for addressing technology vulnerabilities that attackers may exploit.
• Check participants’ processes and technologies for dealing with infections by malicious software (malware).
• Determine whether background checks are conducted on all workers involved in the business’s supply chain.
• Review processes used to ensure all components used in the supply chain are legitimate and free of malware or vulnerabilities.

By implementing these and other measures through a comprehensive supply chain cyber security plan – that is itself part of an integrated approach to cyber security and physical security – businesses can minimise the risk of infiltration and compromise by attackers. If you would like to learn more, please contact us at info@firstwave.com.au.