The Client
Salon Guru provides marketing and booking services to hair and beauty salon chains. They have nearly 200 clients in the UK and the US and a key component of the offering is a website which is integrated into the booking features. These are run on a WordPress multisite setup hosted in AWS in a scalable, load-balanced EC2 cluster and fronted by Cloudflare with static assets (primarily images and JavaScript) hosted close to key geographies on single EC2 instances in other Availability Zones. Much of the cache control, invalidation and firewalling relied on the company's own scripts and integrations between various products in a piecemeal way.
The Challenge
Salon Guru made contact after what looked like a DDNS attack. Very high request traffic was overloading their primary web servers and the caching provided at server and Cloudflare level was insufficient to control that extra load due to the high requests for dynamic pages. They wanted to know how to mitigate against such attacks in the future whilst improving on site performance. In addition, they wanted improvements to be made to their static asset deployment process. Any changes to the infrastructure needed to minimise downtime. The new setup needed to meet the dual demands of excellent cache performance but instant updates of sites in the development phase so that salons could see the changes to their websites in near-real time.
Initial Intervention
Kohera responded on an emergency basis to ensure that the platform was secure and identified the cause of the high traffic and the intentions behind it. It was established that there was an automatic attempt to send spam through the contact pages of the sites and the original attacker had probably been unaware that the list of domains which he had harvested reflected a large number of sites on the same hosting infrastructure. This attack was not adequately mitigated by the existing combination of Fail2Ban on the origin webservers and minimal API calls to Cloudflare to extend firewall blocking to the cache level above. A more dynamic level of protection was clearly needed.
The Objectives
Salon Guru needed:
- An infrastructure design which was not subject to overloading in the event of abnormal traffic.
- A high performance, whole-page cached solution across multiple geographies, maintaining good TTFB (Time to First Byte) and FCP (First Contentful Paint) as against the existing benchmarks.
- Elimination of much of the current complexity, including the termination of SSL/TLS on the origin servers and resulting synchronisation of certificates.
- A migration process which could be tested to eliminate downtime and would allow a staged switch from the old to the new platform with the minimum of disruption.
- Clearly new, documented processes in which Salon Guru staff could be trained.
Kohera reviewed the current infrastructure and identified a multi-phase strategy which would lead to the elimination of Cloudflare and replacement with AWS CloudFront protected with the AWS Web Application Firewall with appropriate dynamic rule sets and manual rules in place, with statistics so that the mitigated periods of high traffic could be identified later to prove the effectiveness of the new infrastructure. DNS would be migrated to AWS Route53 allowing apex domain CNAME-like behaviour. Migration tools would be designed to allow ongoing management without Kohera having to be on-hand at all times during day-to-day running.
Switching the CDN to AWS CloudFront and migrating the DNS zones to AWS Route53 allowed better integration with the load balancer and removed the need for SSL/TLS to be terminated on the origin servers with LetsEncrypt Certs as this could be handled at the CloudFront level. Once the optimal setup of the CloudFront distributions (one per site) was decided on and revised in testing, Kohera provided Ansible tooling to create new distributions and DNS zones and maintain the existing ones by fetching a complete list of them in JSON format and processing them one by one to apply the new desired settings. Training on all the new code and design was provided to the client, leaving them in a good position to make further enhancements themselves, with no lock-in to Kohera for day-to-day operations.
Couldn't recommend James more highly. He has worked on a couple of projects now with our large Wordpress installation. In our latest project he helped us plan the migration of two hundred websites from Cloudflare to Route53/Cloudfront CDN. He fortified and optimised our entire AWS infrastructure which included writing custom tools using Ansible to do the heavy lifting. Since then our performance and traffic has increased and our setup has been very stable. James is world of knowledge and we have found his experience to be invaluable. I look forward to working with him again on our next project.
Shane Marsh, Technical Manager of Salon Guru