How 2wav Monitors Web Applications in the Cloud

AWS |

cloud |

devops |

monitoring |

We’ve built a lot of web applications over the years. Regardless of the technologies we use, or their purpose, all the applications have one thing in common: they need monitoring. Our devOps specialist André Brown was kind enough to write a rather comprehensive guide to how we do it.

Monitoring is essential to ensuring that your web application is serving users as expected. If your application is down, you want to notify your users in a timely manner, preferably before they start to email you or tweet about it. Deploying applications in the cloud requires multiple levels of monitoring to ensure that you’re keeping an eye on all the pieces that can go wrong.

In this blog posts we share with you how 2wav manages the following aspects of monitoring:

Monitoring resource utilization
Monitoring the application stack
Ensuring that our sites are actually up
Monitoring our infrastructure provider
Managing notifications

Monitoring Tools

A plethora of monitoring tools is available, but most offer the same set of features. Your choice will likely come down to personal preference and ease of use.

If you have little interest in setting up your own tool, and don’t mind paying a subscription fee for one, you can get basic monitoring up and running with little effort using a cloud based monitoring tool. Here are a few that we have experimented with at 2wav:

AWS CloudWatch

https://aws.amazon.com/cloudwatch/
2wav relies on AWS for hosting our applications, as well as for hosting most of our client’s applications. So the first thing we looked at was AWS’s monitoring solution.

AWS offers basic monitoring through CloudWatch. CloudWatch isn’t very useful for server monitoring, lacking even the ability to monitor memory usage. However, where it is useful is when you’re building more complex applications using multiple AWS services. For example, it’s indispensable for monitoring your resource utilization to trigger scaling events.

DataDog

http://datadoghq.com
DataDog is a comprehensive monitoring service that offers server, infrastructure, application, and endpoint monitoring. It can be used to monitor on server resource utilization, such as how much memory you’re consuming, or hard drive utilization. It can also be used to monitor your cloud provider’s infrastructure through integrations. For example, they integrate with Amazon Web Services to offer centralized monitoring of the AWS stack.

Monitis.com

http://monitis.com
Monitis.com offers website uptime monitoring, server health monitoring, network performance monitoring, and even monitoring for your mobile apps. Unlike other providers we tried, instead of just offering a free trial period, they offer a free tier which you can continue to use after your trial is over.

Sensu

http://sensu.i o
If you are averse to paying a monthly subscription, or need more control over what you monitor, then you can roll your own monitoring system using open source tools and frameworks. After reviewing a few, we settled on Sensu, a relative newcomer to the monitoring game.

Sensu offer a comprehensive monitoring framework that can be customized to monitor pretty much anything you need to keep an eye on, whether that’s software or hardware. It has a collection of open source plugins than can be used to extend core features. And if you’re not interested in putting together a collection of open source plugins, you can purchase the Enterprise version which has built-in plugins and integrations with enterprise tools.

Resource Utilization

The basics of monitoring involve keeping an eye on the basic building blocks of any web application—the resources available on your server. This is true whether you’re running servers on dedicated hardware in your basement or running a Tier 3 datacenter, and it remains true for monitoring in the cloud.

CPU activity, memory usage, and storage consumption are some key resources that you’ll want to oversee. Below, we talk about how to monitor each of these.

CPU Activity

Spikes in CPU utilization may indicate that your server is busy serving clients, or that some errant process has claimed the CPU as its servant. In either case, you will want to know about it.

At 2wav, we use the Sensu CPU checks plugins (https://github.com/sensu-plugins/sensu-plugins-cpu-check s) to monitor CPU activity. With the default settings, it warns us when CPU activity spikes to 80% and throws a critical warning if it increases to 90%. Later, we will talk about how we avoid getting overwhelmed by notifications for normal activities that may cause CPU usage spikes.

Memory Usage

The Sensu community also offers a plugin to monitor memory u sage. With this plugin, you can check how much RAM or swap is available on your server, which is great if you need to ensure that all servers are deployed with a minimum amount of memory. You can also check memory usage in megabytes or percentages, and warn when usage is high.

Most of our servers at 2wav are cloud servers that don’t use swap, so we primarily monitor memory usage as a percentage of available memory and trigger notifications when memory consumption is high.

Storage Space

Storage consumption can be a pretty nasty gotcha with web application servers. Things like logs that aren’t being rotated or application bugs can result in your storage space being exhausted. To make things worse, you can end up unable to run commands to diagnose the issue because your operating system has no space to write temporary files!

On more than one occasion, memory monitoring has helped 2wav to identify application bugs that were consuming dangerous amounts of storage space on development and production servers. We warn on on 80% disk usage and investigate these warnings before they have a chance to create system lockups.

Baselines and Thresholds

Once you’ve decided what to monitor, you’ll want to establish what normal application usage looks like. You can do this by profiling your application under load, or you can make an educated guess. Whichever way you go, you’ll want to setup alerts to notify you when you’re nearing your threshold, and when you’ve actually hit. Sensu allows us to do both, as all checks have a warning threshold and a critical threshold.

For example, when monitoring storage space, you don’t want to be notified when your hard drive is full, you’ll want to know ahead of time that you’re getting there. A warning when you hit 70% will give you enough time to clean up, or to plan an upgrade. A second warning at 90% should remind you if you forgot. Chances are, if you reach 100% you’ll get warning from other things—like customers tweeting to say that your site is down.

Process Monitoring

A second level of monitoring should take place at the application stack level. At this level, we start paying attention to the applications that keep your web app running.

At 2wav, we run a number of Mete or application behind an NGINX web server with a MongoDB database. We make sure that our Meteor, NGINX and Mongo applications are all doing their job. Let’s take a look at how this can be accomplished.

When setting up monitoring of your application stack, it’s important to know the pieces that keep your application running. If you’ve built the application, then you’ll know exactly what those pieces are. If you’re using an application build by a third party, then you may want to take a look at what components keep it running. You can do this by consulting the documentation, or using the quick and dirty method of checking what’s running on your server using the process listing.

Meteor, Mongo, and NGINX are key application components for many of our deployments here at 2wav. Our Meteor apps have startup scripts, which are named according to the application they run. We use the Sensu Process Checks plugin to ensure that key processes are always running. The plugin may be configured to look for one or more instances of a process, and notify if the number of instances drops below or climbs above an acceptable threshold.

Is It Actually Up?

Monitoring your server and application stack will ensure that all the things necessary for your application to run are in order. However, it does not follow that your application is necessarily running. A bug at your application stack level may cause your app to crash and respawn rapidly, too quickly for your monitoring tool to realize that it’s down. Or your web server may be running, but unable to communicate with the application that it proxies. Or maybe it’s nothing within your control, such as a DNS server failure somewhere between your users and your web application.

This is where a third level of monitoring comes in. At this level you ask “is my application actually running and accessible to my users?” Monitoring your application’s URL endpoints will help to answer this question. This is the equivalent of opening your browser and typing in the URL for your application to see if it loads, except that you have your monitoring tool do this for you at regular intervals.

In addition to checking if your application loads on a given URL, many monitoring tools also allow you to check how quickly your application loads, and will notify you of slow page loads or timeouts. Some will also check the expiration date on your SSL certificates, and warn you when it’s close to expiration.

To verify that our apps are up and running, we use HTTP endpoint monitoring provided by Sensu HTTP plugins. Using this plugin we make calls at regular intervals to our web application URLs, and notify if the server can’t be reached, or if the call times out. Such monitoring is a great catch all for anything that may be missed along the way, such as a metric that may result in your application remaining online but being slow. If the failure to connect to the application isn’t for an obvious reason, investigations of the core reason may provide insight into other metrics that you should be monitoring.

Monitoring Your Infrastructure Provider

In addition to the resources that you provision, cloud-based applications rely on a number of other moving parts to keep running. That server that you’re running your application on is actually running within a set of other highly specialized applications that are prone to faults of their own. While the applications that host your server aren’t under your control, issues within those environments can have direct impact on your server and ultimately your application.

In order to save yourself some headache chasing down application outages that aren’t due to issues with your application or your server, it’s important to monitor the environment that you’re hosting on. Most modern hosting providers offer status pages that report on the health of the various services that power end user applications. Keeping an eye on these may mean the difference between wasting two hours chasing down an imaginary server issue, and two minutes composing an email letting your users know about the outage and assuring them that it’s being worked on (albeit by your hosting provider).

At 2wav, we are heavily reliant on AWS for a number of our clients. We also use Slack a lot. AWS provides a status page with RSS feeds. Slack provides an app that can consume RSS feeds. Could there be a more perfect match? With Slack watching the AWS RSS feeds, we get notified in our #alerts Slack channel whenever anything is amiss at AWS.

Watching The Watchman

When deciding what to monitor there is a blindspot than can easily be missed: the monitoring system itself. If your monitoring system goes down, then you may not notice, especially if things are working well most of the time and you’re not used to seeing notifications in your inbox or Slack channel. Failure of your monitoring system can be mitigated if you are using a distributed monitoring system, or if you use a cloud service provider who will no doubt have a distributed and redundant deployment. However, what if you’re not a big shop but you roll your own monitoring system, and can’t justify the overhead of a distributed system?

To fix that dilemma, 2wav uses a cloud monitoring provider to monitor our monitoring system. Monitor.us, which we mentioned above, offers a free tier which is excellent at monitoring our Sensu server for failures. Using this service, we monitor the processes on the Sensu server that support monitoring (sensu-server, sensu-client, sensu-api and redis-server). We also monitor the Sensu dashboard using an HTTP endpoint monitor. While we have had no failure of Sensu to date, this extra level of monitoring gives us some added peace of mind allowing us to focus on serving our clients.

Notifications

When things go wrong, you’ll want to ensure that the right people are notified. In larger teams, you’ll want to route specific kinds of alarms to specific teams, or team leaders. In smaller teams, you may not have clearly defined roles and simply notify everyone.

At 2wav, we use a Slack channel to notify the team of failures. Many monitoring tools and frameworks offer integration with collaboration tools such as Slack, and this can be used to notify the entire team (a general channel), or notify specific team (the server channel, or the DB channel). If you use Slack and decide to give Sensu a try, then you should check out the Sensu Slack chat handler.

Another thing to bear in mind with notifications is filtering out false positives. If users are inundated with false warning of site outages, then chances are they will develop a habit of ignoring them, including that one time when the wolf is actually at the village gate.

The Sensu check attributes “interval,” “occurrences,” and “refresh” help to reduce the alert chatter in the 2wav #alerts Slack channel. “Interval” specifies how frequently checks should be run. When combined with “occurrences” you are specifying a threshold for failure. For example, an occurrence of 3, means that if your check fails 3 times you will be notified. If your “interval” for checking is 30 seconds, then in the event of an actual issue you will be notified within a minute and a half.

If you find that your configuration generates too many false positives, consider increasing one or both values taking into account how sensitive your application is to downtime. While you want to know as soon as an issue develops, a 1 minute difference in notification times may mean the difference between your inbox or Slack channel being filled with false positive that you have to clear, or being notified only on true failures.

André does software—whether that software is the hypervisor that sits on your bare metal server, shell scripts to manage that server, an open source application that helps to manage a team, or custom code developed from scratch. André currently specializes in DevOps, which harnesses his development skills to solve infrastructure operations problems. When he’s not code bending, he can be found under some weights in the gym, dancing salsa, or reading a science fiction novel.