Leveraging OpEvents & OpConfig to Automate Operational Changes

Purpose

This article will provide an example of using opEvents to trigger opConfig to make an operational change.

Use Case

If an interface starts registering input errors, we want to automatically shift traffic off the circuit in order to maintain transmission quality.

Related Pages

Before attempting this configuration the admin should be familiar with the following wiki articles.

Sequence Overview

  • NMIS polls a router with an SNMP query.
  • The router returns an ‘interface input error’ counter value that has increased; thus triggering a pre-defined threshold.
  • NMIS generates an ‘input error’ alert that is processed by opEvents.
  • opEvents has a predefined action rule matching on node, interface and input errors.  This action will fire an opConfig ‘Configuration Set’.
  • The associated opConfig Configuration Set will increase the OSPF cost on the associated interfaces, thereby causing the router to select another path if available.

Configuration

NMIS

Be default NMIS has the necessary configuration for alerting on input errors.  This is done with the NMIS thresholding system.  The thresholds for the different alerting levels may be adjusted in the appropriate section of /usr/local/nmis8/models/Common-threshold.nmis.  The levels below represent a percentage of input error packets as compared to good packets.
/usr/local/nmis8/models/Common-threshold.nmis
'pkt_errors_in' => {

‘item’ => ‘ifInErrorsProc’,

‘event’ => ‘Proactive Interface Error Input Packets’,

‘title’ => “Input Error Packets”,

‘unit’ => ‘packets’,

‘select’ => {

‘default’ => {

‘value’ => {

‘fatal’ => ‘0.5’,

‘critical’ => ‘0.25’,

‘major’ => ‘0.1’,

‘minor’ => ‘0.05’,

‘warning’ => ‘0.02’,

}

}

}

},

opEvents

By default, opEvents processes the NMIS event log.  All event are evaluated by /usr/local/omk/conf/EventActions.nmis.  If an event matches a rule the appropriate actions will be taken.  EventActions.nmis is also where we define the scripts that opEvents can fire.  The first step is to define the scripts that will shift traffic off a link that’s running input errors.  Since we want to shift all traffic off of this link we will need to run scrips for both ends of the circuit.  Notice the reference to a configset; these will be defined in the opConfig section.


Changes to /usr/local/omk/conf/EventActions.nmis require that the omkd service be restarted.


/usr/local/omk/conf/EventActions.nmis
'script' => {

‘bnelab_p2_fa0_0_route_not’ => {

arguments => ‘act=push_configset name=bnelab-p2_fa0-0_route_not at=now+1minute nodes=bnelab-p2’,

exec => ‘/usr/local/omk/bin/opconfig-cli.exe’,

output => ‘save’

},

‘bnelab_rr1_e1_2_route_not’ => {

arguments => ‘act=push_configset name=bnelab-rr1_e1-2_route_not at=now+1minute nodes=bnelab-rr1’,

exec => ‘/usr/local/omk/bin/opconfig-cli.exe’,

output => ‘save’

},

},


With the scripts defined let’s add the matching rule to the policy section.
/usr/local/omk/conf/EventActions.nmis
'policy' => {

’10’ => {

IF => ‘event.any’,

THEN => {

’10’ => {

IF => ‘event.node eq “bnelab-rr1” and event.element eq “Ethernet1/2” and event.event eq “Proactive Interface Error Input Packets”‘,

THEN => ‘script.bnelab_rr1_e1_2_route_not() and script.bnelab_p2_fa0_0_route_not()’,

BREAK => ‘false’

},

opConfig

The next step is to define the config sets.  Config sets are opConfig talk for the configuration commands you’d like to be run on the router.  Because this step is complicated, yet very repeatable, I’ve supplied this script:  writeConfigSet.sh.  Run the script and it will prompt you for the commands you want to be run on the router and install the config set in opConfig.  In order to verify config sets use the opConfig GUI, from the top menu bar select views, then Configuration Set Overview.

Here is what our example config set looks like.
{

“name”: “bnelab-rr1_e1-2_route_not”,

“commands”: [

“int e1/2”,

“ip ospf cost 9999”,

“exit”

],

“post-commands”: [“write mem”]

}

Testing and Verification

Generate Input Errors

There are several different kinds of input errors but the easiest kind to create in a lab environment are giants.  This is done by having mismatched MTU’s on either side of the same circuit; then sending packets that are too big from the side with the larger MTU.

Example Link - 500
In this example we’ll send giants from bnelab-p2 like so:
bnelab-p2#ping 10.248.2.6 size 1530 repeat 1000 timeout 0 

On benlab-rr1 we’ll see the error counters increment.
bnelab-rr1#show int e1/2 | inc error|giants

0 runts, 4073 giants, 0 throttles

4073 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 output errors, 0 collisions, 1 interface resets

Observe Input Error Event in NMIS

After the next NMIS collect cycle for bnelab-rr1 we should see an event similar to the following:
18-May-2018 13:30:20 bnelab-rr1 Proactive Interface Error Input Packets Fatal Ethernet1/2 p2 Bandwidth=10 Mbps: Value=12.37689 Threshold=0.5

Observe Input Error event in opEvents

Next, find the input error event in opEvents.

Notice the actions taken and scripts sections.  Based on this we know the script was successful and what time the config change has been scheduled for.

Confirm Successful Configuration Push in opConfig

From the opConfig GUI, navigate to the top menu bar select Views, Configuration Change History.  Find and select the config push that relates to our test event.

 

 

 

 

 

 

Uncategorized

Why is the Cybersecurity Insurance Market Surging?

​The global cybersecurity insurance market is growing quickly and is expected to reach more than $20 billion within the next few years. Businesses are increasingly seeking insurance coverage as the number of breaches affecting sensitive customer or corporate data or disrupting systems and operations, increases. Rising reliance on technology to connect with consumers, partners and other stakeholders, the takeup of cloud, mobile, Internet of Things and other models, and the growing sophistication of cyber attacks are all combining to increase cybersecurity risk.

In this environment, board members and senior management teams are shouldering greater responsibility – and in many cases liability – for cybersecurity. For directors and managers, this means dumping traditional perspectives that cybersecurity is the province of the technology team and taking a proactive, organisation-wide approach to minimising cyber risk.

Discharging this responsibility means implementing a comprehensive cybersecurity program that incorporates a range of measures to reduce risk. According to the Australian Government’s business.gov.au website, these measures may include updating all passwords to passphrases that swap letters for symbols and use different spelling; using password managers to securely store and generate passwords; using business-grade cybersecurity software; backing up systems and files to portable hard drives; and seeking help from IT professionals if systems are infected by malware such as ransomware.

The website also recommends businesses talk to insurance brokers or insurers about options for reducing cybercrime risk.

However, businesses should not only tighten their defenses against external attacks. Internal attacks and employee errors can also threaten the integrity of customer and corporate data, as well as key systems. Boards and senior management teams should also become aware of these risks and ensure comprehensive plans are in place to mitigate them. Some issues that may be covered by these plans include implementing role-based access to systems and data – meaning workers only access what they need to do their jobs – and educating workers about how to manage data securely. Adopting these plans can further help businesses reduce risk and consequently lower their cybersecurity insurance premiums. In addition, these plans can also minimise the likelihood of incurring the regulatory and reputational penalties of a cybersecurity breach

Uncategorized

OpEvents – A Technical Service Desk

opEvents is used in many organizations as a valuable tool to monitor their networks, it gives valuable insights into the network, it gains information from everywhere and also delivers a consolidated view to ensure you aren’t flooded with notifications.

The update to opEvents v2.4.3 adds the ability to add editable ‘Event Status’ tags to events. These are status updates that will turn opEvents into an effective technical service desk for you. To help demonstrate how this process works we simulated a network outage in our office, don’t worry nobody was hurt during this event.

In the above image, you will be able to see that this is the synthetic event that was created for a customer outage for Opmantek. Currently, there is no status that has been assigned to the event, this is because the default status is Undef.

If you would like to set a new default status, it can be set by opevents_event_status_values in the opCommon.nmis file. There are three options that are shipped with opEvents are Ingested, Investigating and Resolved.

Every team operates differently so if you would like to customize these values they are defined in opevents_event_status_values in the opCommon.nmis file.

Back to the task at hand, I have been assigned to begin investigating this outage, to ensure that no other members of my team start to troubleshoot this issue, I update the event status to read ‘Investigating’ and begin the troubleshooting steps; this will save your team from having two people trying to solve one issue. This is made even easier because the first troubleshooting steps have already been executed, it is now a matter of interpreting the results and deciding the next steps.

We can see from the above output, the traceroute command was executed when the event started, it has returned with no information, there seems to be a local issue.

After testing the local network and seeing that our local network was working as expected, I was able to isolate the issue to be a problem with an ethernet cable unplugged;

The cable was plugged back in and the issue was resolved, the event status was changed to Resolved and we can close this issue.

Although this was a contrived issue the troubleshooting process exemplifies the power of using opEvents as a technical service desk, all the troubleshooting information that is required to resolve this issue is at your fingertips.

If you would like to see how easily this can be integrated into your environment or to test out the functionality yourself, we offer a free POC for your company and also a free 20 node license of opEvents, start today.

Uncategorized

Network Process Automation, Focus on Tasks

The most successful IT projects that I have been involved in are those where change management and user acceptance  have been recognised as a top priority.  With research organisations citing that up to 47% of jobs will be lost to robotisation it has never been more important to have user support for an automation project.

At the recent Automate 2019 conference, many of our customers and partners shared their experiences in introducing automation to their organisations and it seems that no matter the size or complexity of the organisation, one key response always emerged when this type of project was announced to staff and management.  FEAR.  People were terrified that in supporting an automation project, they were potentially putting their job at risk of being taken over by robots.

Whilst businesses are going to see automation create new ways of working and creating value, what needs to be clearly communicated to staff and stakeholders prior to undertaking an automation project is that it is more than likely that process automation will enhance their performance in their current position, rather than take the position away.

When introducing process automation to a business, it is the ongoing, repeated, time-consuming TASKS that will yield the best outcomes for automation and these are the processes that will be replaced.

To get your staff thinking more positively about the benefits of automation, ask them to think about the time that is spent each day on running and analyzing reports, making configuration changes across multiple machines, recording change and maintenance requests – they will soon realise that these tasks that will likely be automated are actually the preliminary steps that must be taken in order for them to complete a job.  So by automating these tasks, your staff can actually complete their job more efficiently and effectively!

It is not the tasks that are completed but the outcomes that are produced that indicate a job well done.  If your staff see their value as their ability to input or output data, they will lose to robots.  If they understand that their value comes from the other creative and intangible skills that they bring to a role, they will realise that there is little to fear.

To take a look at some of the ways that Opmantek customers have used automation to enhance their network operations, improve efficiencies and to move from a responsive rather than a reactive operation model, contact us and we can show you how we have successfully implemented this for other organizations.

Uncategorized

Cyber Security Technology Investment Trends 2019

According to a new research report by Global Market Insights, the global cybersecurity market is set to grow from its current market value of more than $120 billion to over $300 billion by 2024.

This growth is driven by businesses increasing need to minimize security risks and to build and protect trust, both from within organisations and without. As enterprises globally evolve and shift their business capabilities to cloud computing platforms and other networking technologies, they are becoming more vulnerable to various cyber-attacks. To prevent such attacks organisations are seeking cybersecurity technology that addresses ransomware, phishing, malware and other kinds of cyber assaults as the global average cost of a data breach is up 6.4 percent over the previous year to $3.86 million.

At the recent RSA Conference in San Francisco in March, which brought some of the biggest names in cybersecurity together under one roof, discussed the leading trends impacting the industry. One trend dominated the discussions over the four-day event, and that was that “We are not just protecting data and applications and infrastructures,” RSA President Rohit Ghai said. “We are in the business of protecting trust.”

Covering the RSA conference, Tony Kontzer said, “And in an age of advanced technology, that trust has to extend endlessly, because tech has enabled us to reach across the globe in an instant to connect with total strangers. Or, conversely, to pluck funds from their bank accounts or chip away at their reputations.”

One of the key drivers contributing to the exponential growth of the cyber security market globally is the increase in access and affordability of numerous mobile devices as well as the advancements in the connectivity infrastructure. This expansion is driving the adoption of smart devices across enterprises and consumers, simultaneously, increasing the number of cyber-attacks on mobile devices, which increased by over 40% with an average of over 1.2 million attacks per month.

Over the next five years, the identity and access management (IAM) market is expected to grow at a compound annual growth rate (CAGR) of over 17%. The public sector is key to fuelling this growth with increased cyber-attacks on state and federal bodies and the increased need to assure trust between civilians and their governments when it comes to outside influences on election results.

As the adoption of IoT devices and the use of email and web-based applications continues to grow, the infrastructure protection market is also expected to grow considerably leading up to 2020. The need for individuals to be able to trust that organisations are storing their information securely has never been more important, especially since the Marriott incident that saw half a billion customers data stolen including their names, addresses and passport numbers.

While the West Coast of the US continues to attract VC investments of up to $2.5 billion in cybersecurity companies globally, the East Coast of the US and the rest of the world are steadily increasing their investments in the industry. Israel, the UK, and China are driving most of the foreign investment outside of the US, with Israel leading the way.

Uncategorized

Starting and building a career in cyber-security

Accorind to ​With so much personal data captured and stored by businesses and government agencies, cyber-security – the protection of systems and data from malicious individuals – is becoming more and more important. Breaches can have damaging reputational, legal and financial consequences for businesses and agencies, while individuals may lose money and have their digital identities stolen.

Despite these impacts, not a month seemingly goes by without news of a massive breach. For example, an attack on Facebook last year saw hackers steal the names, contact details and other information about 29 million users worldwide – including 111,813 Australian users. More broadly, the Office of the Australian Information Commissioner’s October-December Notifiable Data Breaches Quarterly Statistics Report revealed the Office received 262 notifications of breaches – 64% of which could be attributed to malicious or criminal attacks.

According to Cybersecurity Venturescybercrime is expected to cost the world USD$6 trillion annually by 2021.

In this environment, a career spent safeguarding data and systems is an exciting option. According to Deakin University, cyber-security roles include information security officer, cyber security consultant, penetration tester and others. Demand for cyber-security specialists is skyrocketing – a recent AustCyber report revealed that a shortfall in the cyber-security workforce was costing more than AUD$400 million in lost wages and that Australia would need an additional 17,600 cyber-security workers by 2026.

Moving into cyber-security is comparatively easy for workers in related fields, such as software engineers and programmers. However, people in unrelated fields or who are completing an education may look instead to University degrees, information security certifications or other courses to gain a foothold in the industry. A range of certifications are also available to help people already in the industry build and demonstrate their skills.

Top cyber-security certifications available include Certified Ethical Hacker; Certified Information Security Manager; CompTIA Security+; Certified Information Systems Security Professional (CISSP); and GSEC: SANS GIAC Security Essentials.

Cloud security specialist FirstWave is always looking for smart, motivated people to join its engineering team. Based in North Sydney, FirstWave protects businesses and government organisations from risk by providing access to advanced, cloud-based cyber-security solutions.

Uncategorized