DB backups

DB backups

Since we’ve got a dedicated DB server, not running as a spot instance now (so it shouldn’t just disappear whenever the hell it wants) and we’re not running RDS, I’ve re-implemented nightly sql backups to S3.

Gosh technology, isn’t it great?

Testing DB migration

Testing DB migration

Yet another DB migration – this time to a dedicated box running mariadb. Testing to make sure that posting a new post works and it appears in the right database….

Certbot and Let’s Encrypt

Certbot and Let’s Encrypt

The only way to use Amazon’s free SSL certificates is if you pipe them through something like cloudformation, use them with cloudfront or a load balancer.

Tried that for December and it turns out its fairly expensive to run an ELB for a month – it’s all fine and dandy whilst I’ve got EDU credit to burn through but I can’t afford $20 a month to shortcut an easy way to SSL cert up the site.

I started to look at certbot and lets encrypt as an automatic way of getting a cert in place, changed my apache config and here we are – running through a straight connection on port 443 to the web server.

To be fair, it was a little silly having a load balancer in front of a single web server, but I didn’t have the time to do anything else with it.

I’ve got a little bit of time now, so setting things up properly.

Helpful tip

Helpful tip

It’s always a good idea not to forget your password. Especially to your blog, especially when you haven’t posted to it for months.

So, yeah, it’s also a good idea to not remember your password and then be fooled by it not working because you’ve locked your account out yet there’s no visual feedback of having done so.

I find more and more that I’m just letting my phone randomly create a password for me/store it in my keychain – seems better that way, plus when I do actually start forgetting important things it won’t matter, because my phone will for all intents and purposes be me anyway.

Happy New year everyone!

AWS RE:Invent

AWS RE:Invent

As a professional working with AWS, is there any event in the calendar year that is more exciting than AWS RE:Invent?

 

Gutted I didn’t get the chance to go this year.  I am SO EXCITED by the new tech coming out of there though.

 

Robotics AND SATELLITE DATA!?

 

Amazing stuff.

Jefferson Frank Salary Survey: Key Findings

Jefferson Frank Salary Survey: Key Findings

Sam Samarasekera, Business Manager at AWS recruitment firm Jefferson Frank, discusses the key findings of the company’s independent salary survey, exploring everything from diversity to salary benchmarks, certification and beyond. 

 

Amazon Web Services’ incredible growth comes with continued benefits to technology industry professionals, but until now they have been difficult to measure accurately. The Jefferson Frank Salary Survey just landed and, as the largest independent study of the industry, it offers a fascinating insight into our working environment. 

 

For the first time, you can get a detailed breakdown of how you can expect to be remunerated, as well as getting a more comprehensive idea of areas of growth. Not only is it valuable to make sure you are being rewarded in your present role, but it gives you a clear view of areas of development to improve your own self-worth.

 

It also provides a wealth of knowledge to anyone else involved in the industry, including customers and partners, to provide a detailed breakdown of the technology and what direction it is going in.

 

I’ve broken down some of the key findings from the 2018/19 survey:

 

How quickly is AWS growing?

 

Of the survey respondents, 65% of professionals had over 10 years’ experience working in the technology industry, with only 10% having less than three years in the industry. So this is a good indicator of the view of experienced IT professionals. Just 7% of those interviewed had more than seven years’ experience working with AWS, and 63% had only started working with it in the last three years, pointing to this being a huge period of expansion for AWS across the technology sector. 

 

How are companies implementing it?

 

With cost being cited as the most important thing to consider when choosing a cloud provider, it’s no surprise that companies are making the switch to AWS internally. Only 14% of companies that migrated to a cloud provider used contractors, which means increasing your knowledge base will make you a vital team member going forward. Its ease of implementation makes it a big draw, so if your company hasn’t already made the switch, you can expect it to sometime soon. The average time taken to migrate to the cloud is 10 months, and with 66% of companies using internal resources to make the change, it seems unlikely you will escape it.

 

Core skill set

 

So what do you need to brush up on? The amount of core products available to work with is vast and one of AWS’ key strengths, with users praising its scope for “tweaking” things rather than having to implement wholesale changes. At least one in five AWS professionals reported that they work or have worked with 43 different products, so there’s plenty to get your teeth into. The most popular by far is EC2, with 89% having experience with it. That’s a huge lead on S3 (70%) and CloudWatch (69%). 

 

Regarding AWS products, partners expect the biggest increase in demand in the next 12 months will be for Amazon EC2, AWS Lambda and Amazon EKS, which is a handy pointer if you wish to get a head start. Basically, if you haven’t already, you need to learn EC2!

 

What makes EC2 so popular?

 

The majority of respondents considered EC2 a part of their core skill set, with EC2 Auto Scaling a close second. EC2’s scalable capacity is undoubtedly its biggest feature and the reason behind its continued growth. Beyond the free tier, the speed and ease that can be added make it the go-to resource for capacity, with its pay as you go service also coming in for particular praise. Ease of use was another key reason behind its popularity and why it looks likely to continue to dominate. No wonder 76% of respondents listed EC2 as the most important area of product knowledge for cloud professionals.

 

Thirdparty tools

 

A lot of the development community has already had exposure to Jenkins, which explains why it’s the mostused thirdparty tool by quite a distance. Some 21% of respondents use it, with Terraform (13%) and Ansible (10%) lagging some way behind. The fact that it’s open source makes up for its tricky user interface, with users giving it many more ticks in the pros column than the cons.

 

Jenkins and Terraform are expected to be amongst the most indemand tools in DevOps and big data in the next 12 months, so familiarisation with these as well as Java will definitely give you an advantage.

 

The all-important certifications

 

Over half of the survey participants were not AWS certified before taking up their current role, so it isn’t essential if you’re considering a job change. However, twothirds of respondents now hold certs, so be prepared to knuckle down as it’s highly likely that you’ll be expected to gain them once you’re in position. If you simply wish to increase your earning potential or become more employable, they appear to be a must-have. 

 

The financial benefits of certifications

 

We’ll start with the bad newsless than half of responders’ employers paid for their certifications in full (49%), although a further 11% did contribute towards the cost. However, and this is the good news, 22% of professionals reported a direct increase in salary after passing their certs. Participants listed relevant certifications as their number one way to increase earning potential, so it’s a worthy investment even if you have to make it yourself. 

 

Which certs do I need?

 

The top certifications are AWS Certified Solutions Architect, AWS Certified Developer and AWS Certified SysOps Administrator. However, accreditations in programming languages such as Java and PHP will also be advantageous, although Python is expected to be the most in-demand language in the next 12 months.

 

If you’d like more detail, the Jefferson Frank Annual Salary Survey is an invaluable resource for any technology professionals. It’s available to download now and has a more comprehensive breakdown of salaries across the globe, as well as further information on products, certifications and tech trends in the past 12 months as well the coming year.

Happy Black Friday!

Happy Black Friday!

Today is the day we get to see which retailers have elastic scaling infrastructure and half decent architects with forward planning and those who do not.
 
CDKeys and ChaosCards have already been hit with downtime post mail shots – don’t email your customer base if you can’t cope with them all clicking the link at the same time.
 
Chaos tried to put a queing system in front but the site still ended up offline mid browse.
Promotional credit can not be used for Reserved Instances!

Promotional credit can not be used for Reserved Instances!

I’ve started to play around with Bamboo and a few other bits and pieces in an evening and I’ve managed to find a few promotional $150 vouchers for AWS that you would usually get from AWS training courses on ebay – prices range from about £11 to £35 depending on expiry.  Bamboo will not run on a T3 micro – it just doesn’t have the CPU capacity – definitely needs 2cpus, even for a tiny installation – that’s Java for you 😉

 

I had a spot instance running but it kept going away due to capacity.  It’s running again on a spot instance but I’ve whacked up the price to $1 an hour now so hopefully that’ll never go away.  But again, I’m putting database backups in place to make sure it doesn’t go anywhere, should it go down.

 

I’ve got $450 of credit on my account and it goes towards the EC2 cost along with quite a few other services.  Here’s the complete list:

 

  • Amazon Simple Storage Service
  • AWS Lambda
  • Amazon Glacier
  • Amazon Sumerian
  • Amazon Relational Database Service
  • Amazon SimpleDB
  • AWSCloudDirectoryInternal
  • Amazon CloudSearch
  • AWS IoT
  • Amazon Lightsail
  • Amazon Elastic Compute Cloud
  • AWS Elastic Beanstalk
  • Amazon Simple Notification Service
  • AWS Config
  • Amazon Redshift
  • AmazonCloudWatch
  • Amazon Elastic File System
  • AWS Data Pipeline
  • Amazon ElastiCache
  • Amazon CloudFront
  • Amazon Elastic Transcoder
  • Amazon Virtual Private Cloud
  • Amazon Lex
  • AWS Data Transfer
  • Amazon Simple EDI
  • AWS OpsWorks
  • AWS Storage Gateway
  • Amazon Simple Email Service
  • Amazon Machine Learning
  • Amazon SageMaker
  • Amazon Elastic MapReduce
  • AWS X-Ray
  • Amazon DynamoDB
  • Amazon EC2 Container Registry (ECR)
  • AWS Glue
  • AWS Budgets
  • Amazon Polly
  • Amazon Route 53
  • AWS Support (Basic)
  • Amazon Elasticsearch Service
  • AWS CloudTrail
  • Amazon Rekognition
  • AWS Key Management Service
  • Amazon Simple Queue Service
  • Amazon AppStream
  • Amazon Kinesis

 

So, seeing that EC2 was in the list, and there was no way in hell I was going to burn through that much credit by the end of next year, I decided to purchase some Reserved Instances.  A T3 micro for 3 years and a T3 small for one year, all upfront – total cost was $264 + 20% VAT.  And this charged itself directly to my credit card, with pretty much no confirmation.  To be fair to Amazon, I did click the purchase button, I just expected to see a final confirmation screen that showed me where the funds were going to be coming from.  That didn’t happen.

 

To be fair, there is documentation to support that:

 

https://aws.amazon.com/premiumsupport/knowledge-center/credits-ri/

 

I just didn’t see it before I clicked the old purchase button 😉

 

Their support team have been lovely, they’ve managed to cancel the Reserved Instances and are starting the process to refund me.  I upgraded to developer support at $29 a month to get my request actioned sooner.  I’ve not decided whether or not I’m going to keep this in place or not yet – there are benefits to it for sure, and as I start to do more and more with it over the coming months, it might be a good idea.  We’ll see.

 

Workmail isn’t covered by the credit, thats $4 per user per month, but to be honest, it is so much better than what I’m used to with my previous host, it makes me realise just how great an enterprise level solution it is.

WAF is pricey – for personal use

WAF is pricey – for personal use

Turns out that WAF costs $5 per acl and $5 per rule per month. And that $10 protection I had in place for a simple geo block against China and a few other countries I was getting probing attacks from wasn’t included in the AWS credit I’ve got so I’ve turned it off.

 

 

Done some playing around with ALBs and fail2ban including setting up a custom filter for script attacks but need to do some further refinement with x forwarded for headers and what not.

 

Hopefully i’ll get some more time to have a fiddle with it over the weekend.