We use WP-CLI extensively here at Dogsbody Technology. It allows us to streamline and automate our WordPress set up and maintenance routine, so we thought we’d spread the word and get everybody else in on the action.

Installation

There are a few installation methods for WP-CLI, all which are documented here. We typically use the Phar installation method, which is as simple as:

curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
chmod +x wp-cli.phar
sudo mv wp-cli.phar /usr/local/bin/wp

Basic Usage

Unless otherwise instructed, WP-CLI will operate on the site contained in your current working directory. So if you want to work on a particular site you’ll need to “cd” to the installation directory before running your command, or alternatively you can pass the --path argument to WP-CLI. e.g.

wp --path=/var/www/dogsbody.com plugin install yoast

Creating a new site

As well as managing existing sites, WP-CLI can also set up new ones. You’ll need to create a MySQL database and user, but beyond that WP-CLI can handle the rest. A basic site download/install procedure may look something like this:

wp core download --locale=en_GB
wp core config --dbname=database_name --dbuser=database_user --dbpass=database_password
wp core install --url=www.dogsbody.com --title="Dogsbody's Website" --admin_user=dogsbody --admin_password=admin_password --admin_email= --skip-email

Re-Writing a site

We often have customers wanting to take an existing site and re-configuring it to work on a new domain, or wanting to add HTTPS to an existing site and update everything to be served securely. WP-CLI makes this otherwise quite complex process much easier with with it’s search/replace feature:

wp search-replace 'https://www.dogsbodytechnology.com' 'https://www.dogsbody.com' --skip-columns=guid

(It’s advisable to skip the guid column as the guid of posts/pages within WordPress should never change).

In summary, WP-CLI is a very powerful tool and one that anybody working with WordPress sites often should at least be aware of. It can save you heaps of time and help you avoid mistakes.

If you want any help with WP-CLI, then please contact us. Or if you want some seriously fast and secure WordPress hosting, be sure to check out our WordPress hosting.

The post WP-CLI – The Swiss Army Knife For WordPress appeared first on Dogsbody Technology.

an open source software foundation dedicated to making cloud native computing universal and sustainable

They do this by hosting and “incubating” projects they see as valuable, helping them to develop and reach maturity, where they can be used widely in cloud environments.

CNCF has over 350 members including the world’s largest public cloud and enterprise software companies as well as dozens of innovative startups

The CNCF is also backed by the Linux Foundation, who are fast becoming one of the most recognised organisations in the industry. They support the open source community as a whole, aiming to protect and accelerate development of the Linux kernel, along with many other things.

Why should I care?

The CNCF is exciting as, for me at least, it provides a bit of a portal into the way that the industry is moving at the moment.  It showcases both the current behemoths of cloud computing software stacks, along with projects that are likely to replace or supplement them in the future. The CNCF split their projects into 3 main categories:

• Graduated
• Incubating
• Sandbox

Graduated projects are ones that have reached maturity and see wide adoption. The current list of these projects at the time of writing are Kubernetes, Prometheus, Envoy, CoreDNS and containerd. If you’ve been even dabbling in the cloud/linux community, then you’ve probably heard of at least a few of these projects.

Incubating projects are ones that haven’t quite hit the prime time yet, but are well on their way. These currently include projects such as rkt, a container engine that’s a potential competitor for Docker, CNI (Container Network Interface), which focuses on configuring networking within containers, and etcd, a key-value store designed for storing critical system data.

I find the CNCF useful for guiding me on what pieces of software I should be learning to enhance my skill set as they’re likely to be desirable in the short to medium term. It’s also one of the first places I’m likely to check for a piece of software that fits a particular need, as I know that CNCF projects are going to be active, well supported, and have lots of related stack overflow questions / Github issues for when I’m getting started.

Training and Certification

The CNCF also offer some training and certification options. This is useful to prove that you’re familiar and capable with some of the technologies they support. At the time of writing, the training courses and certifications they offer are all kubernetes based (which is by no means a bad thing), but I’m sure they will offer more in the future.

In summary, the CNCF acts a sort of central hub for a lot of the hottest and biggest projects right now, and even if you’re don’t have a particular need for them at this time, it’s good to know what’s out there right now, as well as coming over the hill, and it’s therefore useful for this reason alone.

Featured image by chuttersnap on Unsplash

The post The Cloud Native Computing Foundation appeared first on Dogsbody Technology.

One of the most effective solutions to this problem is Multi-Factor Authentication, or MFA (sometimes written 2FA for 2 Factor Authentication).

What is MFA?

MFA is a process by which more than one piece of information is required in order to authenticate against a service or provider.

What’s the problem?

In days of old, and still on less tech-savvy sites on the internet, a single username and password combination would be sufficient to grant you access to an account. Now in an ideal world, everybody would use lengthy and difficult to guess passwords, using different passwords for every service. But humans will be human, and take the easier route of using shorter, easy to remember, and worst of all common passwords. This inevitably leads to accounts being compromised when those common passwords are tried, or when the attacker reads the post-it that’s stuck to the bottom of your monitor…

How does MFA help?

MFA helps to resolve this problem by requiring a second piece of information; a second factor. This second factor can be many different things, with different sites offering the choices they think best. The most common are:

• email
• SMS
• Automated phone call
• Mobile device

How does it work?

Upon entering your valid username and password combination, the site or application will ask you for your second factor. If you provide this second factor correctly, then you will be authenticated. If you provide the wrong information, or no information at all, then you are denied. Simple right?

Isn’t this essentially just a second password?

Great question! Some sites may just require a second piece of text for your second factor, and in this case, it is essentially just a second password yes. However, good MFA is usually configured so that is requires something you know, and something you have. For example, a password, and an SMS. Using SMS as the second factor requires the user to have the mobile phone with the number configured on the account. Same for a phone call. This means that if somebody learns your password, it is useless unless they also have your unlocked mobile phone.

The next thing to consider is that the second factor is changing regularly and often. When a provider sends you an SMS, this is usually valid for a short period of time, say 10 minutes. If you wish to login after this time, you must receive a new SMS with a new passcode. This of course prevents people from writing the second factor down, as it would be useless a short while later, and also means that if an attacker were to find out what the second factor was, they would have a very short window in which to login.

Side note: though we’ve used SMS as an example here, there’s a growing movement of people that consider it insecure due to demonstrated attacks which are able to bypass this second factor. As with any security procedure, you should always consider it’s merits and potential weaknesses before putting it in place yourselves.

In summary MFA is both a simple and effective way of keeping your online accounts secure. We strongly recommend everyone enables it where possible. You should still continue to use strong passwords and follow best practices in terms of security too. 

Feature image background by ChrisDag licensed CC BY 2.0.

The post Multi -Factor Authentication And Why You Should Use it appeared first on Dogsbody Technology.

How was it?

The whole weekend was brilliant, and the energy when you’re there is amazing. There is a real sense of community and friendship amongst everybody, many of whom have never even met before. There’s no such thing as a silly question at EMF, if you want to know more about something, just ask; there will be lots of people willing to help you out. There are some very clever people present, so there are many opportunities for learning, and teaching too.

Power and Internet

One of the more unique aspects of EMF is the fact that you get power and super fast internet to your tent/village. This year, a ludicrously quick 1Gbps symmetrical connection was provided. This is a fast circuit for a permanent installation, let alone one for a ~3 day event (not forgetting the amazing volunteers who’d been on site for nearly 10 days when all was said and done!)

Villages

Campers are allowed, and encouraged, to set up villages on the site. These are basically collections of tents/structures where people camp together so they can work on similar projects, or just enjoy the company of other like minded people. Whilst Dogsbody didn’t have a “village” as such, we all camped together and had a communal space for hacking / eating / chatting etc. Here’s most of us hacking on our badges (more on these in a while)

The Talks / Workshops

EMF 2018 was the biggest yet, and it was packed full of content. There were roughly 120 talks, 74 workshops, 20 performances and 43 other events. And they were all amazing! There’s obviously too much stuff to discuss here, so I’ve asked the rest of the company to tell me a little about their favourite parts:

Friday night at EMF Camp was amazing – 100 watts of lasers, Fire…Professor Elemental live and seeing the Null Sector for the first time was incredible. I’m a sucker for multi coloured lights

– Claire

As a regular user of the trains, I was drawn to the railway signalling talk. I often jokingly write the trains off as hopeless and incompetent but as we all know everything is always more complex than it first appears so this was a great place to get a better idea as to what keeps going wrong. The talk was interesting, informative and well delivered.

Here are some of the things that fall under “a signalling problem”:

• Keeping track of where the trains are (this is often done by putting a battery on the track or a small box to count the axles as they pass it).
• Making sure you aren’t too close to the train in front (is usually done by either splitting the track into zones and only allowing one train per zone or by keeping track of the zone in front of each train that it will need to safely stop).
• Signalling the driver and making sure the fail safes are working in case they miss / ignore the signal.

Just from these things you end up with a huge number of ways for the trains to be delayed to ensure passenger safety.

– Jim

My favourite talk was Boiling nemo – make your own internet of things. Which detailed out the presenters descent into IoT madness, after his fish tank malfunctioned he set up a Raspberry Pi to monitor it.

He slowly takes his home IoT further and further until at the end he is comparing “Big data” with a graph showing his son leaving for school (later and later!).

My other highlight was watching Hackers, it is one of those movies that is so much better with a crowd. HACK THE PLANET!

– Rob

The Cracking HiTag2 Crypto talk was very interesting as I’ve always wondered how they work. Learning that as well as how to crack them was indeed fascinating. The way you can crack and copy them is seriously impressive and now I’ve seen how all of that is done I want to go myself.

– Chris

The talk from from two hackers that at the age of 16 were banned from using any encryption technology for 5 years was fascinating.  It’s very clear that the UK police service has a long way to go before they even know what to do with criminals in an online world.  With the UN declaring the internet now being a basic human right and the push for encryption of all online content over the last few years there can easily be a disconnect between punishment and rehabilitation.

– Dan

At the time of writing, all of the happenings of EMF 2018 are detailed here and videos of all of the talks can be found here

Badges

The badges at EMF are truly unique. Most conferences give you a name badge which has your name on it, along with maybe your occupation / job title etc. EMF goes above and beyond this quite stratospherically, giving you badges which are small, hackable microcontrollers. This year, along with the now “standard” screen and LEDs, this year’s badges had fully functional mobile phones built in. And these connected to… the fully functional GSM mobile network that was set up on site. We could write an entire post on the badges and what they’re capable of (which we may well do at some point), so I’ll just have to leave you with some pictures for now (note the SIM card above the battery wires in the second picture)

Summary

We all had a blast at EMF 2018, and we’re already looking forward to EMF 2020, which we hope to attend. We really don’t know what the organisers are going to do next, but we’re sure it’s gonna be mega.

If you attended EMF 2018 and have anything to share, please leave a comment below.

Here are some of our favourite photos from this years event.

The post EMF Camp 2018 – A Field Full of Nerds appeared first on Dogsbody Technology.

Installation

Duplicacy is written in Go, meaning it can be easily downloaded and compiled on the CLI. However, this involves installing Go on the system you wish to backup, which may not always be an option. Fortunately, duplicacy also provides binary releases, which can be downloaded and executed with ease.

To install duplicacy on a Linux system, the steps are as follows:

wget https://github.com/gilbertchen/duplicacy/releases/download/v2.1.0/duplicacy_linux_x64_2.1.0
sudo mv duplicacy_linux_x64_2.1.0 /usr/local/bin/duplicacy
sudo chmod +x /usr/local/bin/duplicacy

You can then run duplicacy by simple running the “duplicacy” command in your terminal.

Setting up your storage

As mentioned above, duplicacy supports an impressive number of storage back-ends. As of the time of writing, they are:

• Local disk
• SFTP
• Dropbox
• Amazon S3
• Wasabi
• DigitalOcean Spaces
• Google Cloud Storage
• Microsoft Azure
• Backblaze B2
• Google Drive
• Microsoft OneDrive
• Hubic
• OpenStack Swift
• WebDAV (under beta testing)
• pcloud (via WebDAV)
• Box.com (via WebDAV)

The two options that we’ve used are SFTP and AWS (Amazon Web Services) S3. To backup a system over SFTP, all you need is a working SFTP user on the remote system. No additional set up is required.

The set up for Amazon S3 is a little more involved, in summary, the steps are:

• Create an Amazon S3 bucket in your preferred region
• Create an IAM policy granting permissions on this bucket
• Create an IAM user and assign them this policy
• Configure duplicacy to use this user and bucket

Creating an S3 bucket

Creating a bucket is pretty straightforward. Login to your Amazon S3 account, go to the S3 service, click “Create bucket”, give your bucket a name, select a region, done. There are some other options when creating a bucket but these are not relevant to this post so I’ll not cover them here.

Creating an IAM policy

IAM stands for Identity and Access Management, and is central to many operations in AWS. To create your policy, navigate to the IAM service in AWS, select “policies” on the left, and click the big blue “Create policy” button at the top.

On this screen, choose the “JSON” tab. This is where we’ll specify the guts of our policy. It should look something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::dbt-gary-duplicacy-backup-example",
                "arn:aws:s3:::dbt-gary-duplicacy-backup-example/*"
            ]
        }
    ]
}

You’ll need to replace “dbt-gary-duplicacy-backup-example” with the name of the S3 bucket you created in the last step

When you’re happy with your policy, click “Review policy”, followed by “Save changes”

Creating an IAM user and assigning the policy

From the home of the IAM service, now click “Users” on the left, followed by the big blue “Add user” button at the top. Provide a name for your user, and check the “Programmatic access” box below. Click next.

On the next screen, click “Attach existing policies directly”. At the top of the list of policies now listed below, click the “Filter: Policy type” drop-down, and select “Customer managed”. Check the box for your IAM policy, and click “review” to continue, followed by “Create user” on the next page.

Your IAM user and policy have now been created.

Ensure that you save the details now presented to you. You will need these to configure duplicacy

Configuring duplicacy

On the system you wish to backup, navigate to the directory you wish to backup. For example, on the system we configured, this was the “/home” directory. You can now configure duplicacy. The steps are as follows:

sudo duplicacy init your_repo_name s3://your-region@amazon.com/your_bucket_name
sudo duplicacy set -key s3_id -value your_access_key
sudo duplicacy set -key s3_secret -value your_secret_key

There are a number of strings you’ll need to replace in the above snippet:

your_repo_name – The name you’d like to give to this set of backups. For example, “johns-desktop”

your_bucket_name – The name you gave your S3 bucket in the steps above.

your_region – This is the AWS region you select for your buck above. Please see this table, using the “region” column that corresponds to your region name. For example, “eu-west-2” for the London region

your_access_key – This is the access key for the IAM user you created above. It will be a long string of random looking characters.

your_secret_key – This is the secret key for the IAM user you created above. It will again be a long string of random looking characters. Make sure you keep this safe, as anybody who has it can access your backups!

Running a backup

If all went well with the above, then you’re ready to run your first backup. This is as easy as running:

sudo duplicacy backup

This will backup all files under the current directory. Depending on the number of and size of files, this may take some time.

Including/excluding certain files/directories from your backups

Duplicacy offers powerful filtering functionality allowing for fine grained control over what files and directories you want to backup. These can be somewhat confusing to configure, but are very useful once you’ve got the hang of them. We may do a follow up post covering these, so be sure to check back in the future.

Restoring backups from duplicacy

In order to restore from duplicacy, you need to configure your system to interact with your backups. If you’re restoring on the same system the backups were taken on, you need not take any additional steps. If you’re restoring to a different system, you need to follow the installation and duplicacy configuration steps show above.

Once things are configured, you can view the available backups like so:

sudo duplicacy list

Note that you must be in the correct directory on your system (the one where you initialised your repo), in order to view the backups

This will give you a list of your backups:

Snapshot johns-desktop revision 1 created at 2018-04-12 07:29 -hash
Snapshot johns-desktop revision 2 created at 2018-04-12 12:03 
Snapshot johns-desktop revision 3 created at 2018-04-17 17:37 
Snapshot johns-desktop revision 4 created at 2018-04-18 11:10 
Snapshot johns-desktop revision 5 created at 2018-04-18 14:38 
Snapshot johns-desktop revision 6 created at 2018-04-20 03:02 
Snapshot johns-desktop revision 7 created at 2018-04-21 03:02 
Snapshot johns-desktop revision 8 created at 2018-04-22 03:02 
Snapshot johns-desktop revision 9 created at 2018-04-23 03:02

As you can see, there are revision numbers and the corresponding times and dates for these revisions. Revisions are just another name for a backup.

You can then restore a particular backup. For example, to restore revision 7:

sudo duplicacy restore -r 7

Again, depending on the number and size of files in this backup, this may take some time.

Duplicacy offer some really cool features when using the restore command. For example, you can see the contents of a file in a backup with the “cat” option, and compare differences between two backups with the “diff” option. You can see all of the options here.

Selective restores

One of the more useful restore options is to only restore a certain file or directory from your backup. This can be accomplished with the following command:

sudo duplicacy restore -r 7 path/to/your/file.txt

This can also be extended to restore everything under a directory, like so:

sudo duplicacy restore -r 7 path/to/your/directory\*

Summary

Duplicacy is an extremely powerful and portable backup tool, allowing for reliable and fine grained backups of your data. If you have any questions on duplicacy or would like any help setting it up, please leave a comment below or contact us and we’ll be happy to help. Thanks for reading.

Feature image background by 111692634@N04/ licensed CC BY-SA 2.0.

The post Duplicacy: Backup to the cloud appeared first on Dogsbody Technology.

Kapacitor Alerts

As mentioned in previous posts, the portion of the TICK stack responsible for the actual alerting is Kapacitor. Put simply, Kapacitor takes metrics stored in the InfluxDB database, processes and transforms them, and then sends alerts based on configured thresholds. It can deal with both batches and streams of data, the difference is fairly clear from the names; batch data takes multiple data points as an input and looks at them as a whole. Streams accept a single point at a time, folding each new point into the mix and re-evaluating thresholds each time.

As we wanted to monitor servers constantly over large time periods, stream data was the obvious choice for our alerts.

We went through many iterations of out alerting scripts, known as TICK scripts, before mostly settling on what we have now. I’ll explain one of our “Critical” CPU alert scripts to show how things work (comments inline):

var critLevel = 80 // The CPU percentage we want to alert on
var critTime = 15 // How long the CPU percentage must be at the critLevel (in this case, 80) percentage before we alert
var critResetTime = 15 // How long the CPU percentage must be back below the critLevel (again, 80) before we reset the alert

stream // Tell Kapacitor that this alert is using stream data
    |from()
        .measurement('cpu') // Tell Kapacitor to look at the CPU data
    |where(lambda: ("host" == '$reported_hostname') AND ("cpu" == 'cpu-total')) // Only look at the data for a particular server (more on this below)
    |groupBy('host')
    |eval(lambda: 100.0 - "usage_idle") // Calculate percentage of CPU used...
      .as('cpu_used') // ... and save this value in it's own variable
    |stateDuration(lambda: "cpu_used" >= critLevel) // Keep track of how long CPU percentage has been above the alerting threshold
        .unit(1m) // Minutely resolution is enough for us, so we use minutes for our units
        .as('crit_duration') // Store the number calculated above for later user
    |stateDuration(lambda: "cpu_used" < critLevel) // The same as the above 3 lines, but for resetting the alert status 
        .unit(1m) .as('crit_reset_duration') 
    |alert() // Create an alert... 
        .id('CPU - {{ index .Tags "host" }}') // The alert title 
        .message('{{.Level}} - CPU Usage > ' + string(critLevel) + ' on {{ index .Tags "host" }}') // The information contained in the alert
        .details('''
        {{ .ID }}
        {{ .Message }}
        ''')
        .crit(lambda: "crit_duration" >= critTime) // Generate a critical alert when CPU percentage has been above the threshold for the specified amount of time
        .critReset(lambda: "crit_reset_duration" >= critResetTime) // Reset the alert when CPU percentage has come back below the threshold for the right time
        .stateChangesOnly() // Only send out information when an alert changes from normal to critical, or back again
        .log('/var/log/kapacitor/kapacitor_alerts.log') // Record in a log file that this alert was generated / reset
        .email() // Send the alert via email 
        |influxDBOut() // Write the alert data back into InfluxDB for later reference...
             .measurement('kapacitor_alerts') // The name to store the data under
             .tag('kapacitor_alert_level', 'critical') // Information on the alert
.tag('metric', 'cpu') // The type of alert that was generated

The above TICK script generates a “Critcal” level alert when the CPU usage on a given server has been above 80% for 15 minutes or more. Once it has alerted, the alert will not reset until the CPU usage has come back down below 80% for a further 15 minutes. Both the initial notification and the “close” notification are sent via email.

The vast majority of our TICK scripts are very similar to the above, with changes to monitor different metrics (memory, disk space, disk IO etc) with different threshold levels and times etc.

To load this TICK script into Kapacitor, we use the kapacitor command line interface. Here’s what we’d run:

kapacitor define example_server_cpu -type stream -tick cpu.tick -dbrp example_server.autogen
kapacitor enable example_server_cpu

This creates a Kapacitor alert with the name “example_server_cpu”, with the “stream” alert type, against a database and retention policy we specify.

In reality, we automate this process with another script. This also replaces the $reported_hostname slug with the actual hostname of the server we’re setting the alert up for.

Getting customer servers reporting

Now that we could actually alert on information coming into InfluxDB, it was time to get each of our customers’ servers reporting in. Since we have a large number of customer systems to monitor, installing and configuring Telegraf by hand was simply not an option. We used ansible to roll the configuration out to the servers that needed it which involved 12 different operating systems and 4 different configurations.

Here’s a list of the tasks that Ansible carries out for us:

• On our servers:
  • Create a specific InfluxDB database for the customers server
  • Create a locked down InfluxDB write only user for the server to send it’s data in with
  • Add Grafana data source to link the database to the customer
• On the customers server:
  • Setup the Telegraf repo to ensure it is updated
  • Install Telegraf
  • Configure Telegraf outputs to point to our endpoint with the correct server specific credentials
  • Configure Telegraf inputs with all the metrics we want to capture
  • Restart Telegraf to load the new configuration

The above should be pretty self-explanatory. Whilst every one of the above steps would be carried out for a new server, we wrote the Ansible files to allow for most of them to be run independently of one another. This means that in future we’d be able to, for example, include another input to report metrics on, with relative ease.

For those of you not familiar with Ansible, here’s an excerpt from one of the files. It places a Telegraf config file into the relevant directory on the server, and sets the file permissions to the values we want:

---
- name: Copy inputs config onto client
  copy:
    src: ../files/telegraf/telegraf_inputs.conf
    dest: /etc/telegraf/telegraf.d/telegraf_inputs.conf
    owner: root
    group: root
    mode: 0644
become: yes

With the use of more ansible we incorporated various different tasks into a single repository structure, did lots of testing, and then ran things against our customers’ servers. Shortly after, we had all of our customers’ servers reporting in. After making sure everything looked right, we created and enabled various alerts for each server. The process for this was to write a BASH script which looped over a list of our customers’ servers and the available alert scripts, and combined them so that we had alerts for the key metrics across all servers. The floodgates had been opened!

Summary

So, at the end of everything covered in the posts in this series, we had ourselves a very respectable New Relic replacement. We ran the two systems side by side for a few weeks and are very happy with the outcome.  While what we have described here is a basic guide to setting the system up we have already started to make improvements way beyond the power we used to have.  If any of them are exciting enough, there will be more blog posts coming your way, so make sure you come back soon.

We’re also hoping to open source all of our TICK scripts, ansible configs, and various other snippets used to tie everything together at some point, once they’ve been tidied up and improved a bit more. If you cannot wait that long and need them now, drop us a line and we’ll do our best to help you out.

I hope you’ve enjoyed this series. It was great of a project that the whole company took part in and that enabled us to provide an even better experience for our customers. Thanks for reading!

Replacement Server Monitoring

• Part 1: Picking a Replacement
• Part 2: Building the replacement
• Part 3: Kapacitor alerts and going live! (you are here)

Feature image background by swadley licensed CC BY 2.0.

The post Replacement Server Monitoring – Part 3: Kapacitor alerts and going live! appeared first on Dogsbody Technology.

In our last post we discussed our need for a replacement monitoring system and our pick for the software stack we were going to build it on. If you haven’t already, you should go and read that before continuing with this blog post.

This post aims to detail the set up and configuration of the different components to work together, along with some additional customisations we made to get the functionality we wanted.

Component Installation

As mentioned in the previous entry in this series, InfluxData, the TICK stack creators, provide package repositories where pre-built and ready to use packages are available. This eliminates the need for configuration and compilation of source code before we can use it. This allows us to install and run software with the use of a few commands with very predictable results, as opposed to often many commands needed for compilation, with sometimes wildly varying results. Great stuff.

All components are available from the same repository. Here’s how you install them (example shown is for an Ubuntu 16.04 “Xenial” system

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/lsb-release
echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get update && sudo apt-get install influxdb
sudo systemctl start influxdb

The above steps are also identical for the other components, Telegraf, Chronograf and Kapacitor. You’ll just need to replace “influxdb” with the correct name in lines 4 and 5.

Configuring and linking the components

As each of the components are created by the same people, InfluxData, linking them together is fortunately very easy (another reason we went with the TICK stack). I’ll show you what additional configuration was put in place for the components and how we then linked together. Note that the components are out of order here, as the configuration of some components is a prerequisite to linking them to another.

InfluxDB

The main change that we make to InfluxDB is to have it listen for connections over HTTPS, meaning any data flowing to/from it will be encrypted. (To do this, you will need to have an SSL certificate and key pair to use. Obtaining that cert/key pair is outside the scope of the blog post). We also require authentication for logins, and disable the query log. We then restart InfluxDB for these changes to take effect.

sudo vim /etc/influx/influx.conf

[http]
    enabled = true
    bind-address = "0.0.0.0:8086"
    auth-enabled = true
    log-enabled = false
    https-enabled = true
    https-certificate = "/etc/influxdb/ssl/reporting-endpoint.dogsbodytecnhology.com.pem"

sudo systemctl restart influxd

Note that the path used for the “https-certificate” parameter will need to exist on your system of course.

We then need to create an administrative user like so:

influx -ssl -host ivory.dogsbodyhosting.net
> CREATE USER admin WITH PASSWORD 'superstrongpassword' WITH ALL PRIVILEGES

Telegraf

The customisations for Telegraf involve telling it where to reports its metrics to, and what metrics to record. We have an automated process, using ansible for rolling these customisations out to customer servers, which we’ll cover in the next part of this series. Make sure you check back for that. These are essentially what changes are made:

sudo vim /etc/telegraf.d/outputs.conf

[[outputs.influxdb]]
  urls = ["https://reporting-endpoint.dogsbodytechnology.com:8086"]
  database = "3340ad1c-31ac-11e8-bfaf-5ba54621292f"
  username = "3340ad1c-31ac-11e8-bfaf-5ba54621292f"
  password = "supersecurepassword"
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"

The above dictates that Telegraf should connect securely over HTTPS and tells it the username, database and password to use for it’s connection.

We also need to tell Telegraf what metrics it should record. This is configured like so:

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = true
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs"]
[[inputs.diskio]]
[[inputs.net]]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.procstat]]
  pattern = "."

The above tells Telegraf what metrics to report, and customises how they are reported a little. For example, we tell it to ignore some pseudo-filesystems in the disk section, as these aren’t important to us.

Kapacitor

The customisations for Kapacitor primarily tell it which InfluxDB instance it should use, and the channels it should use for sending out alerts:

sudo vim /etc/kapacitor/kapacitor.conf
    [http]
    log-enabled = false
    
    [logging]
    level = “WARN”

    [[influxdb]]
    name = "ivory.dogsbodyhosting.net"
    urls = ["https://reporting-endpoint.dogsbodytechnology.com:8086"]
    username = admin
    password = “supersecurepassword”

    [pushover]
    enabled = true
    token = “yourpushovertoken”
    user-key = “yourpushoveruserkey”

    [smtp]
    enabled = true
    host = "localhost"
    port = 25
    username = ""
    password = ""
    from = "alerts@example.com"
    to = ["sysadmin@example.com"]

As you can probably work out, we use Pushover and email to send/receive our alert messages. This is subject to change over time. During the development phase, I used the Slack output.

Chronograf Grafana

Although the TICK stack offers it’s own visualisation (and control) tool, Chronograf, we ended up using the very popular Grafana instead. At the time when we were building the replacement solution, Chronograf, although very pretty, was somewhat lacking in features, and the features that did exist were sometimes buggy. Please do note that Chronograf was the only component that was still in beta at this period in time. It’s now had a full release and another ~5 months of development. You should definitely try it out for yourself before jumping straight to Grafana. We intend to re-evaluate Chronograf ourselves soon, especially as it is able to control the other components in the TICK stack, something which Grafana does not offer at all.

The Grafana install is pretty straightforward, as it also has a package repository:

sudo vim /etc/apt/sources.list.d/grafana.list
    deb https://packagecloud.io/grafana/stable/debian/ jessie main
sudo apt update
sudo apt install grafana

We then of course make some customisations. The important part here is setting the base URL which is required due to the fact we’ve got Grafana running behind an nginx reverse proxy. (We love nginx and use it wherever we get the chance. We won’t detail the customisations here though as they’re not strictly related to the monitoring solution, and Grafana works just fine on it’s own.)

sudo vim /etc/grafana/grafana.ini
    [server]
    domain = display-endpoint.dogsbodytechnology
    root_url = %(protocol)s://%(domain)s:/grafana
sudo systemctl restart grafana

Summary

The steps above left us with a very powerful and customisable monitoring solution, which worked fantastically for us. Be sure to check back for future instalments in this series. In part 3 we cover setting up alerts with Kapacitor, creating awesome visualisations with Grafana, and getting all of our hundreds of customers’ servers reporting in and alerting.

Part three is here.

Replacement Server Monitoring

• Part 1: Picking a Replacement
• Part 2: Building the replacement (you are here)
• Part 3: Kapacitor alerts and going live!

Feature image background by tomandellystravels licensed CC BY 2.0.

The post Replacement Server Monitoring – Part 2: Building the replacement appeared first on Dogsbody Technology.

For many years, and as long as I have worked at the company, we’d used an offering from New Relic, called simply “Servers”. In 2017, New Relic announced that they would be discontinuing their “Servers” offering, with their “Infrastructure” product taking it’s place. The pricing for New Relic infrastructure was exorbitant for our use case, and there were a few things we wanted from our monitoring solution that New Relic didn’t offer, so being the tinkerers that we are, we decided to implement our own.

This is a 3 part series of blog posts about picking a replacement monitoring solution, getting it running and ready, and finally moving our customers over to it.

What we needed from our new solution

The phase one objective for this project was rather simple: to replicate the core functionality offered by New Relic. This meant that the following items were considered crucial:

• Configurable alert policies – All servers are different. Being able to tweak the thresholds for alerts depending on the server was very important to us. Nobody likes false alarms, especially not in the middle of the night!
• Historical data – Being able to view system metrics at a given timestamp is of huge help when investigating problems that have occurred in the past
• Easy to install and lightweight server-side software – As we’d be needing to install the monitoring tool on hundreds of servers, some with very low resources, we needed to ensure that this was a breeze to configure and as slim as possible
• Webhook support for alerts – Our alerting process is built around having alerts from various different monitoring tools report to a single endpoint where we handle the alerting with custom logic. Flexibility in ours alerts was a must-have

Solutions we considered

A quick Google for “linux server monitoring” returns a lot of results. The first round of investigations essentially consisted of checking out the ones we’d heard about and reading up on what they had to offer. Anything of note got recorded for later reference, including any solutions that we knew would not be suitable for whatever reason. It didn’t take very long for a short list of “big players” to present themselves. Now, this is not to say that we discounted any solutions on the account of them being small, but we did want a solution that was gonna be stable and widely supported from the get-go. We wanted to get on with using the software, instead of spending time getting it to install/run.

The big names were Nagios, Zabbix, Prometheus, and Influx (TICK).

After much reading of the available documentation, performing some test installations (some successful, some very much not), and having a general play with each of them, I decided to look further at the TICK stack from InfluxData. I wont go too much into the negatives of the failed candidates, but the main points across them were:

• Complex installation and/or management of central server
• Poor / convoluted documentation
• Lack of repositories for agent installation

Influx (TICK)

The monitoring solution offered by Influx consists of 4 parts, each of which can be installed independently of one another

TTelegraf – Agent for collecting and reporting system metrics

IInfluxDB – Database to store metrics

CChronograf – Management and graphing interface for the rest of the stack

KKapacitor – Data processing and alerting engine

Package repositories existed for all parts of the stack, most importantly for Telegraf which would be going on customer systems. This allowed for easy installation, updating, and removal of any of the components.

One of the biggest advantages for InfluxDB was the very simple installation: add the repo, install the package, start the software. At this point Influx was ready to accept metrics reported from a server running Telegraf (or anything else for that matter. There were many clients that support reporting to InfluxDB, which was another positive)

In the same vein, the Telegraf installation was also very easy, using the same steps as above, with the additional step of updating the config to tell the software where to report it’s metrics too. This is a one-line change in the config, followed by a quick restart of the software.

At this point we had basically all of the system information we could ever need, in an easy to access format, only a few seconds after things happen. Awesome.

Although the most important functionality to replicate was the alerting, the next thing we installed and focused on was the visualisation of the data Telegraf was reporting into InfluxDB. We needed to ensure the data we were receiving mirrored what we were seeing in New Relic, and it can also be tricky to create test alerts when you have no visibility of the data you’re alerting against too, so we needed some graphs (everyone loves pretty graphs as well of course!)

As mentioned above, Chronograf is the component of the TICK stack responsible for data visualisation, and also allows you to interface with InfluxDB and Kapacitor, to run queries and create alerts, respectively.

In summary, the TICK stack offered us an open source, modular and easy to use system. It felt pleasant to use, the documentation was reasonable, and the system seemed very stable. We had a great base, one which we could design and build our new server monitoring system. Exciting!

Part two is here.

Replacement Server Monitoring

• Part 1: Picking a Replacement (you are here)
• Part 2: Building the replacement
• Part 3: Kapacitor alerts and going live!

Feature image background by xmodulo licensed CC BY 2.0.

The post Replacement Server Monitoring – Part 1: Picking a Replacement appeared first on Dogsbody Technology.

That being said, backups of reproducible data can still be very useful as well, if the time it would take you to recreate said data is more valuable to you than the data itself. For example, it’s very easy to get an operating system set up the way you like it, but if you want to get on with creating things, instead of setting things up, then it’s worth having backups that allow you to get going again quickly should the worst happen.

What should you backup? How often? Why?

As we touched on above, you should back up anything that is irreplaceable (photos, letters, nan’s recipes), and anything that would take a non-trivial amount of time to recreate. How often to backup your data depends on a few things:

• How often is it changing? Taking daily backups is pointless if your data is only changing once a week. On the flip side, backing up once a week if you’re data is changing daily also leaves a lot of room for lost work, which brings us to our next point
• Granularity – how much detail would you like your backups to cover? Lets use a novel you’re writing for an example. How often would you want to save copies of your work? Every page? Every paragraph? Every line? Every word? Whilst this isn’t the best example, seeing as storage is so cheap nowadays you could store every different letter and get away with it, it illustrates the concept nicely. Even if a paragraph in your novel only takes a few minutes to write, what about the ideas in that paragraph, can you guarantee that you’ll think of the same great words next time if you were forced to rewrite it? Make sure you can track changes in the right detail can make all the difference between having backups, and having useful backups.
• Storage costs – Let’s scrap the novel idea for now and think big, really big. Take a media production house, they’re gonna be storing Gigabytes, maybe Terabytes of data per project. This can result in some serious costs for your storage hardware. Unlike the novel, the cost of saving a copy after every change would be prohibitive, so you need to draw the line somewhere else. Where this line falls again comes back to the time-cost comparison: how much will it cost you to store the backups, and how much would it cost you to carry out the work again? Missing an important deadline to lost data is a real pain and can make you look unprofessional.

The 3-2-1 Rule

This is a common rule when talking about backups, at least at the simpler levels. The rule dictates that you should always aim for:

• At least three copies of your data
• On at least two different storage mediums
• With least one of these copies in an off-site location

For example, one copy of the data on on your hard disk, one copy on an external drive, and a final copy in the cloud. This gives you a great chance of recovering your data in the event of problems. With the ubiquity of moderately priced external storage, and the plethora of free cloud storage solutions out there, it makes it really, really easy to have multiple copies of your most valued bytes.

RAID, why it’s great, and why it’s not a backup

RAID (Redundant Array of Independent Disks) is a technology that used to be found solely in the enterprise. However, as with most things in the tech world, it has found it’s way down into the levels of your everyday users over the years. RAID allows you to keep multiple copies of your data automatically and transparently with relative easy. At the user level, you save you data just as you normally would. But behind the scenes, clever bits of hardware and/or software makes multiple copies of this data and store it on multiple physical disks. In it’s most basic form, RAID-1, also know as a mirror, does just that; mirrors your data. One file (or millions of them) stored identically on two (or more) disks. If one of the disks stops working, you can grab your data back from the other disks. Great right? Yes.

However, RAID is not the solution to all of your backup woes. RAID’s strength can also be seen as it’s downfall, and that is that it does things automatically. If you delete a file, it’s deleted from all of the disks. It’s not clever enough to realise that you didn’t actually want to remove that file forever. Remember, computers are dumb, they just do what you tell them. RAID can be seen as increasing the availability of your data. It saves you having to pull copies from your other storage methods from the 3-2-1 rule. What it doesn’t protect again is somebody clicking the wrong button and washing away all of your favourite pictures.

This is another advantage to the 3-2-1 rule. Even if you delete something on your primary storage, chances are that you realise before you sync this storage to your secondary storage. And if you don’t catch it then, then chances are you will catch it before you sync things to your off-site storage. These layers offer time delays allowing you to realise your mistakes and correct them.

Testing your backups

Testing your backups is of critical but often overlooked importance. Having all the backups in the world is still no good if they don’t actually work. For this reason, you should try to verify your backups are in good condition as often as possible. Make sure that novel opens fine in your text editor, make sure some of those family photos aren’t missing etc etc. It’ll be devastating to find out your golden backup solution is anything but in the times when you need it most.

Let us help

If in reading this blog post you’ve had a panic and realised your server is lacking any meaningful backup solutions, then please get in touch. We’d love to get your data stored away safely for you.

Feature image background by gothopotam licensed CC BY 2.0.

The post The Importance of Backups appeared first on Dogsbody Technology.

To start with, take the following bullet points and think for a moment how your architecture may differ vs another if you made each one the top priority when designing it:

• ease of management
• cost
• reliability
• performance
• scalability
• availability

Everything on one server

All components sits on a single system. For a typical modern web application, this would include all of the parts needed to run the app, such as a web server, database server, and the application code itself. A very common setup of these components would be the LAMP stack, which is an acronym for Linux, Apache, MySQL, PHP (Python, Perl), and is used at all levels, entry to enterprise, just with tweaks for the latter.

When to use it?

If you’re just after a quick and simple set up to host your basic app/site, then you’re gonna struggle to find anything easier to get started with. A tried and tested workhorse.

Advantages

• Simple! With some hosting providers you can set this up in just a few clicks

Disadvantages

• All your eggs in one basket. If this server goes offline, so does your site/app
• Not very easy to scale horizontally
• All components competing for the same limited resources

Separate Database Server

Splitting out the database component of your infrastructure from the rest of it (and this will be an ongoing theme) allows you to isolate the resources available to each of these components. This makes capacity planning much easier, and can also give you some fault tolerance in more advanced configurations. This is almost always the first infrastructure upgrade we see people spring for and we often recommend it ourselves. It’s a quick win, and pretty easy to do.

When to use it?

When you’re looking for your first set of upgrades for your infrastructure, your database layer is struggling, or you’d like better separation between your web and database components.

Advantages

• Simple upgrade from a single server set up
• Better separation between resources, leading to easier scaling

Disadvantages

• Additional cost over single server
• A little more complex than a single server setup
• More areas, primarily network performance, need to be considered

Caching Layer

Caching content can make an absolutely massive difference to site performance and scalability. Caching involves storing in memory a (usually) commonly requested asset, think a logo or a web page, so that it can be served to a visitor without having to be generated and/or loaded from the disk every time.

Two of the most popular pieces of software used in caching are Memcached and Redis, both of which are key-value stores. This means that each piece of cached data has a key, which is essentially a name used to identify the data, and a value, which is the data itself. As explained above, these values are returned to a visitor instead of having to generate the data fresh again.

Plugins for Redis/Memcached support exist for most popular CMSes, allowing for very easy integration with an existing set up. See this tutorial for an example of integrating WordPress with Redis

When to use it?

When you’re serving a lot of static content, such as images, CSS or Javascript.

Advantages

• Alleviates load on your application servers
• Big performance increases for little effort

Disadvantages

• Can be difficult to tune correctly

Load Balancing

The clue is in the name. In this set up, the load is balanced between multiple servers, allowing them all to serve a portion of the traffic. This also has the benefit of added redundancy, in that if one of the servers were to go offline, the other server(s) around around to handle the load.

When to use it?

When redundancy is important, and when scaling individual servers any higher becomes an unreasonable process.

Advantages

• Increased redundancy
• Scaling with zero downtime by adding in additional servers to the pool

Disadvantages

• Single point of failure. If the load balancer goes down, all traffic to your site will go down
• Additional complexity. A prime example is with sessions, and ensuring that visitors have a consistent web experience regardless of which app server handles their request

Summary

Managing infrastructure can be hard, especially when you get to the advanced levels such as load balancing and auto-scaling, but there are also very positive results to be had from not huge amounts of work. Planning for the future and considering the differing needs of the many parts of your app can save you a lot of heartache and expense down the line. Still unsure? Contact us and we’ll be happy to help.

Feature image credit CWCS Managed Hosting – CC BY 2.0

The post 4 Common Server Setups For Your Web Application appeared first on Dogsbody Technology.