Integrating Amazon’s WAF with opEvents Monitoring Automation

At Opmantek, we use our own software heavily for monitoring our production and development systems, solving our own IT Operations challenges that we know our customers share, it also helps us to develop the products faster in real-world environments through early testing.

We have been using Amazon’s Web Application Firewall (WAF) to help protect our web-facing infrastructure. One of the issues with the out-of-the-box solution is how does one monitor the firewall’s logs as part of your overall IT operations and how do you perform analysis of those logs, with context, to the workloads they relate to.

Firstly to help check newly implemented rules are working as intended and secondly, to provide quick diagnoses in the event of attack.

We first tested a 3rd party product to help visualise the logs and hopefully provide out-of-the-box insights into the data, but we found the TCO of this solution was much higher than using the extensibility of the Opmantek products.  The results from the out-of-the-box solution would also have been isolated from the overall network health visibility.

Our WAF is set up with the rules sets provided by the AWS marketplace as well as internally developed custom rules sets with reputation / ip blacklists that are constantly evolving.

Our Architecture follows:

• AWS Web Application Firewall (Layer 7 Firewall)
• AWS Kinesis Delivery Stream (durable real-time data streaming service)
• Opmantek Kinesis Log Service (Kinesis stream receiving transformation service)
• opEvents ( Centralised logging and Event Management)

Our WAF is set up to send all logs through to our Kinesis Delivery Stream.

AWS kinesis delivery stream is set up to deliver batched requests over HTTPS to a specified endpoint within your own environment. We developed a small HTTP service in GoLang to securely ingest the batched logs from AWS, and we provide this AWS ingestion service to customers on request.

Our GoLang service also remaps JSON keys before writing the file out to disk.

To see what AWS publishes in its logs you can find them – here.

opEvents jsons_log service listens for filesystem changes, reads them, and runs the event through the opEvents Engine.

We have added a new property country which is the ISO country code of the request.

Element: is mapped to the Requestor’s  IpAddress.

Node: is the name of our WAF in AWS.

Description: is the WAF action, WAF rule which was triggered, ipAddress and country code. This gives opEvents unique enough data to create rolled up event counts for WAF actions. Through opEvents’ dashboard you can see a quick count of clients who have made the most POST requests, or a bot trying SQL injection against your site.

We are using opEvents to store metadata about the WAF log, headers, requesting IP, country and which WAF rules were terminated. Using the IP address we can quickly make an assumption about the requesting origin and know if we have bots scraping us from data centres or users acting unlawfully. With this quick drill down into the event data we can make quick operational changes to implement rules to stop certain traffic or add entire subnets to our IP blacklist.

How we are using this information…

Debugging WAF rules

Implementing WAF rules can be challenging especially when you have to go back and look at access history. opEvents is storing 30 days of WAF logs which we can quickly filter to find the blocked request and debug the rule and make an exception or change how our application works for better security.

Detecting bots

Some crawlers generate quite a large amount of web traffic as they quickly scan our domains, being able to have an aggregate view of requests for an ip Address and rules being triggered it’s easy to find the block of address causing issues. We then drill down into the request metadata checking the headers, location, who owns the IP and past requests patterns. From this we can then quickly ban malicious bot IP ranges.

Website usage statistics 

With MongoDB backing opEvents it’s easy to write queries and run them through the mongo shell and aggregate usage data for more in depth reporting. Which Country Code uses this endpoint the most, Which user-agent makes the most requests.

Automated Response

Using Opmantek’s Operational Process Automation methodology when we correlate sets of WAF events we trigger automated actions into our AWS Environment to respond to the incidents and avoid issues.

opEvents engine makes it flexible to ingest any type of structured data and we could quickly integrate into our production monitoring to give us greater insight into our public facing web systems.

If you would like to know more about about using opEvents and processing web firewall logs we offer live demos with our technical team here.