Pushover Alerts

Alerts & Webhooks with AWS Lambda

/1 Comment/in , /by

Here at Dogsbody Technology we monitor servers and services for hundreds of clients, you may have read our previous blog post talking about our Warboard and how we make use of it. This blog post covers the other tools we use for responding to incidents and issues real time, our Dogsbody Technology Webhooks.

The main thing we use the webhooks for are Pingdom, Newrelic & Sirportly alerts. When an incident is triggered in Pingdom or Newrelic they will make an API call to our webhook with the relevant information we require to investigate an incident, the webhook will then determine the priority of the incident and send an alert to our Pushover user accounts so we are alerted and can respond to the incident.

High priority alerts, such as site outages also trigger a rotating blue police style light which is accompanied by a siren sound from the office speaker.

 

Office Siren

The Dogsbody Technology office siren

 

We also use the webhooks to notify a user when certain interactions happen in our ticketing system Sirportly, such as being assigned a new ticket or when one of their existing tickets is replied to.

To ensure our webhooks would have near to 100% uptime and we wouldn’t miss an alert, we decided the best place to host them would be using AWS Lambda & AWS API Gateway. These two services combined allow us to run the webhooks with Amazon’s high availability infrastructure while only paying small amounts on a per request/alert basis, which is the perfect type of model for this service.

To put into perspective how cost effective AWS’ pricing model for our alerts is, last month (June 2016) we received 25,282 alerts for all of our endpoints combined. This worked out at a total monthly cost of … $0.10! AWS actually provide you with a free amount of lambda execution time per month which we haven’t even reached yet, we’re only getting charged that 10 cents for the API Gateway.

Let us know if you find any of the services and technologies mentioned above interesting and we can write some more in-depth blog posts on those subjects, and even some guides on using them. The alerts talked about in this blog post come with the majority of our server monitoring packages, so be sure to get in contact if you need any of our services.

The Warboard

/3 Comments/in /by

The Dogsbody Technology Warboard sits on the wall in our office and allows us to see a detailed overview of the infrastructure we monitor real time, this has proved it’s self to be invaluable for spotting potential issues and remedying them before they ever become an issue.

We’re responsible for monitoring and maintaining hundreds of servers on a daily basis.  Checking the status of this infrastructure manually would be virtually impossible. To make this job easier we use tools such as Pingdom and NewRelic however we still felt the need for a high level overview of all servers.

When there is an issue with either a service failing on a server, or the health of a server deteriorates Pingdom and NewRelic will alert us real time via custom webhooks we have written. These are great for reacting to an issue when it happens, however it doesn’t give us a clear overview of the infrastructure we monitor before an issue occurs, this is why we created the Warboard.

The Warboard is displayed in such as way that we only see the metrics we need to. Services at the top of the Pingdom column are ordered by highest response rate, servers in NewRelic are ordered by the highest metric for each server (if CPU utilisation was a higher percentage than memory, disk usage and disk IO it would be used). We display the Warboard on a wall mounted TV for the whole team to see.

Warboard

Example screenshot of the warboard

In the Pingdom column red checks are checks that are currently down, blue checks are paused and green checks are up. In the NewRelic column red checks are servers that have hit their high threshold on their policy, amber checks have hit their warning threshold, blue checks are servers that are no longer reporting and green checks are servers that have not reached a threshold.

We also have a column for Sirportly, our ticketing system. This shows how many tickets each team member has. Below this is an overview of events in our Google Calendar where we can see upcoming events and scheduled maintenance.

The Warboard backend is all written in Python and the frontend is Python (Flask) using the Jinja2 templating engine. We’ve made the Warboard public on Github, so feel free to contribute, modify it and use it in your own environment if you please.

If you’d like us to monitor your infrastructure be sure to take a look at our maintenance packages and get in contact.

Welcome Claire Christmas!

/0 Comments/in /by

Claire ChristmasWe are pleased to announce the expansion of our team in the form of Claire Christmas (yes that really is her real name!)

Claire will be joining us as an Office Manager and brings with her years of administration and customer services skill as well as a real passion for cost savings and efficiency.

Into anything Marvel and with a background in working in technology companies Claire spent most of her career at Sun Microsystems where she started as an apprentice and then Oracle after the acquisition.

Claire will be dealing with the day to day admin of running Dogsbody Technology to allow the Dogsbody team to do what they do best – Linux Sysadmin!

When asked what she thinks she’s bring to the company she answered, “other than single handedly bringing the average age of the company up a few years I look forward to adding a female touch”.  We think she’ll fit in just fine! 🙂

We’re always looking for new members to join our team, so go and check out what jobs we have available if you think you’d be a good fit at Dogsbody Technology.

Congratulations Rob Hooper

/0 Comments/in /by

We are very pleased to congratulate Rob Hooper on completing his Linux Systems Apprenticeship.

Rob now holds…

  • City & Guilds Certificate in IT Systems and Principles
  • City & Guilds Level 3 Diploma in IT Professional Competence

We are even more pleased that Rob has chosen to continue his career with Dogsbody Technology as a permanent member of staff.  Rob is now looking forward to continuing his education with professional qualifications from Amazon Web Services (AWS) and the Linux Foundation.

If you are interested in joining the Dogsbody Technology team as a Linux Systems Apprentice please apply at https://www.dogsbody.com/jobs/

2015 Christmas Shutdown

/0 Comments/in /by

Christmas may have screamed up on us all faster than Rudolph seeing red but we wanted to remind you that we will be here to support you through out the Christmas holidays.

We will be taking some time off though and the office will be closed from 25th December 2015 to the 3rd January 2016 inclusive.

We will also be out of the office from 13:00 PM on Friday 18th December for our Christmas party. We’re off to see Star Wars: The Force Awakens!

During this time any issues will be dealt with on an emergency out of hours basis and we will only be able to support customers who are experiencing a situation where business cannot function without a resolution such as:

  • Website / Server down
  • Inability to trade online

We will continue monitoring and patching servers as usual over the Christmas period.

For our hosting customers, our status site will continue to be updated and we recommend you check here first as we may already be aware of an issue and posting updates for you. If you need to raise an issue please use the standard office telephone number and support email and your query will be routed to the engineer on call.

Thank you for your continued support throughout 2015. We hope you have a very Merry Christmas and a Happy New Year.

Congratulations Gary Rixon

/0 Comments/in /by

We are very pleased to congratulate Gary Rixon on completing his Linux Systems Apprenticeship.

Gary now holds…

  • City & Guilds Certificate in IT Systems and Principles
  • City & Guilds Level 3 Diploma in IT Professional Competence

We are even more pleased that Gary has chosen to continue his career with Dogsbody Technology as a permanent member of staff.  Gary is now looking forward to continuing his education with professional qualifications from Amazon Web Services (AWS) and the Linux Foundation.

If you are interested in joining the Dogsbody Technology team as a Linux Systems Apprentice please apply at https://www.dogsbody.com/jobs/

See you at AWS Summit London 2015

/0 Comments/in /by

We can’t wait to attend the AWS Summit in London next week on the 15th April 2015.

We will be there all day so shout if you want to meet up.

 

2014 Christmas shutdown

/0 Comments/in /by

Christmas may have screamed up on us all faster than Rudolph seeing red but we wanted to remind you that we will be here to support you through out the Christmas holidays.

We will be taking some time off though and the office will be closed from 25th December 2014 to the 4th January 2015 inclusive.

During this time any issues will be considered emergency out of hours and will only be able to support customers who are experiencing a situation where business cannot function without a resolution such as:

  • Website / Server down
  • Inability to trade online

We will continue monitoring and patching servers as usual over the Christmas period.

For our hosting customers, our status site will continue to be updated and we recommend you check here first as we may already be aware of an issue and posting updates for you. If you need to raise an issue please use the standard office telephone number and support email and your query will be routed to the engineer on call.

Thank you for your continued support throughout 2014. We hope you have a very Merry Christmas and a Happy New Year.

Welcome Gary Rixon!

/0 Comments/in /by

Gary Rixon

We are pleased to announce the expansion of our team in the form of Gary Rixon. Gary will be joining us as a Linux Systems Apprentice and brings with him a real passion for servers and hardware.

A bit of a nerd with a love of the outdoors, Gary can typically be found tinkering with computers and servers, or helping out at his local scout group. He recently returned from an awesome trip to Belgium with the scouts.

In the previous two years, Gary has been studying Computing, Physics and Maths at A-level, but his main focus has always been to get into the world of Linux and servers, so he brings with him a solid base of knowledge and the drive to learn more.

Gary will be taking over first line support from Alex, and taking part in projects of increasing levels of complexity (and fun!) as he progresses through his apprenticeship.

We’re always looking for new members to join our team, so go and check out what jobs we have available or tell us about your skills if you think you’d be a good fit at Dogsbody Technology.

Welcome Rob Hooper!

/0 Comments/in /by

RobHooper

Hello! I’m Rob; Gamer, Tea drinker, and lover of all things hackable. I have customised every computer I have ever owned within a inch of its life.

I am joining the Dogsbody team as a Linux Systems Apprentice. Excited to learn more about Linux, the cool new technologies out there as well as giving back to the community which I believe is what Linux is all about.

Until now I have been focused on my studies but they haven’t challenged me. They have been very hands off; discuss and write about things. I can’t wait to actually do, and learn from real life experiences.

While I get up to speed there are many projects to play with involving Raspberry Pi’s and AWS. Coding scripts and making tools.

I will also be taking the task of handling maintenance packages off of Dan and Alex and heading it up as a project of my own.

I am looking forward to working with all of our existing, new customers and hopefully you as we continue to grow.

 

Feature image by Cydcor is licensed under CC BY 2.0