Browsed by
Category: ec2

Bamboo fixed

Bamboo fixed

Turns out Bamboo wasn’t working properly – I think I’d started to create a test job, and that had broken the DB patch/reboot job. Just re-ran it through and all seems good now.

Need to stop it from emailing me every night and having me just ignore the emails. I might tie it into SNS or something. It’s good to have something to play with.

$40 a month on EC2 instances

$40 a month on EC2 instances

Looks like my main web server instance is costing less than $2 a month on a T3 Micro. That’s a spot instance with a fairly high bidding price set so that it’s very unlikely to ever be outbid and is still the cheapest way to provision EC2 capacity.

By the same token, my database server is also a T3 micro instance and that’s looking to cost just under $7 a month on demand – same uptime as the web server so you can see there’s a $5 saving.

The bamboo server is having to run on a medium server and that’s used around $27 this month. Don’t you just love java?

Still got $300 of credit in my account so there’s no massive rush to move off the on demand instances right now. Maybe I should think about it though.

Workmail is $4 per user per month plus tax. Route53 is $0.50 per domain per month, there’s no getting around that. If I can keep my monthly bills less than $30 a month that would certainly keep my wallet happy.

Need to do a fair bit more work with bamboo to see if I can get it running on something a little less powerful or spot instance it. We’ll see.

Stability is a wonderful thing

Stability is a wonderful thing

Not a single server outage in the last three weeks. Which is great, it means that I’ve gotten everything set up properly as far as server updates, security groups and maintenance are concerned.

Bamboo seems to be doing what it’s meant to do and is rebooting the servers nightly. Everything seems to be working properly with regards to database backups and everything. I need to double check what the monthly cost is looking like when I start to run out of promotional credit at the end of this year. Could be quite costly.

Jefferson Frank Salary Survey: Key Findings

Jefferson Frank Salary Survey: Key Findings

Sam Samarasekera, Business Manager at AWS recruitment firm Jefferson Frank, discusses the key findings of the company’s independent salary survey, exploring everything from diversity to salary benchmarks, certification and beyond. 


Amazon Web Services’ incredible growth comes with continued benefits to technology industry professionals, but until now they have been difficult to measure accurately. The Jefferson Frank Salary Survey just landed and, as the largest independent study of the industry, it offers a fascinating insight into our working environment. 


For the first time, you can get a detailed breakdown of how you can expect to be remunerated, as well as getting a more comprehensive idea of areas of growth. Not only is it valuable to make sure you are being rewarded in your present role, but it gives you a clear view of areas of development to improve your own self-worth.


It also provides a wealth of knowledge to anyone else involved in the industry, including customers and partners, to provide a detailed breakdown of the technology and what direction it is going in.


I’ve broken down some of the key findings from the 2018/19 survey:


How quickly is AWS growing?


Of the survey respondents, 65% of professionals had over 10 years’ experience working in the technology industry, with only 10% having less than three years in the industry. So this is a good indicator of the view of experienced IT professionals. Just 7% of those interviewed had more than seven years’ experience working with AWS, and 63% had only started working with it in the last three years, pointing to this being a huge period of expansion for AWS across the technology sector. 


How are companies implementing it?


With cost being cited as the most important thing to consider when choosing a cloud provider, it’s no surprise that companies are making the switch to AWS internally. Only 14% of companies that migrated to a cloud provider used contractors, which means increasing your knowledge base will make you a vital team member going forward. Its ease of implementation makes it a big draw, so if your company hasn’t already made the switch, you can expect it to sometime soon. The average time taken to migrate to the cloud is 10 months, and with 66% of companies using internal resources to make the change, it seems unlikely you will escape it.


Core skill set


So what do you need to brush up on? The amount of core products available to work with is vast and one of AWS’ key strengths, with users praising its scope for “tweaking” things rather than having to implement wholesale changes. At least one in five AWS professionals reported that they work or have worked with 43 different products, so there’s plenty to get your teeth into. The most popular by far is EC2, with 89% having experience with it. That’s a huge lead on S3 (70%) and CloudWatch (69%). 


Regarding AWS products, partners expect the biggest increase in demand in the next 12 months will be for Amazon EC2, AWS Lambda and Amazon EKS, which is a handy pointer if you wish to get a head start. Basically, if you haven’t already, you need to learn EC2!


What makes EC2 so popular?


The majority of respondents considered EC2 a part of their core skill set, with EC2 Auto Scaling a close second. EC2’s scalable capacity is undoubtedly its biggest feature and the reason behind its continued growth. Beyond the free tier, the speed and ease that can be added make it the go-to resource for capacity, with its pay as you go service also coming in for particular praise. Ease of use was another key reason behind its popularity and why it looks likely to continue to dominate. No wonder 76% of respondents listed EC2 as the most important area of product knowledge for cloud professionals.


Thirdparty tools


A lot of the development community has already had exposure to Jenkins, which explains why it’s the mostused thirdparty tool by quite a distance. Some 21% of respondents use it, with Terraform (13%) and Ansible (10%) lagging some way behind. The fact that it’s open source makes up for its tricky user interface, with users giving it many more ticks in the pros column than the cons.


Jenkins and Terraform are expected to be amongst the most indemand tools in DevOps and big data in the next 12 months, so familiarisation with these as well as Java will definitely give you an advantage.


The all-important certifications


Over half of the survey participants were not AWS certified before taking up their current role, so it isn’t essential if you’re considering a job change. However, twothirds of respondents now hold certs, so be prepared to knuckle down as it’s highly likely that you’ll be expected to gain them once you’re in position. If you simply wish to increase your earning potential or become more employable, they appear to be a must-have. 


The financial benefits of certifications


We’ll start with the bad newsless than half of responders’ employers paid for their certifications in full (49%), although a further 11% did contribute towards the cost. However, and this is the good news, 22% of professionals reported a direct increase in salary after passing their certs. Participants listed relevant certifications as their number one way to increase earning potential, so it’s a worthy investment even if you have to make it yourself. 


Which certs do I need?


The top certifications are AWS Certified Solutions Architect, AWS Certified Developer and AWS Certified SysOps Administrator. However, accreditations in programming languages such as Java and PHP will also be advantageous, although Python is expected to be the most in-demand language in the next 12 months.


If you’d like more detail, the Jefferson Frank Annual Salary Survey is an invaluable resource for any technology professionals. It’s available to download now and has a more comprehensive breakdown of salaries across the globe, as well as further information on products, certifications and tech trends in the past 12 months as well the coming year.

Promotional credit can not be used for Reserved Instances!

Promotional credit can not be used for Reserved Instances!

I’ve started to play around with Bamboo and a few other bits and pieces in an evening and I’ve managed to find a few promotional $150 vouchers for AWS that you would usually get from AWS training courses on ebay – prices range from about £11 to £35 depending on expiry.  Bamboo will not run on a T3 micro – it just doesn’t have the CPU capacity – definitely needs 2cpus, even for a tiny installation – that’s Java for you 😉


I had a spot instance running but it kept going away due to capacity.  It’s running again on a spot instance but I’ve whacked up the price to $1 an hour now so hopefully that’ll never go away.  But again, I’m putting database backups in place to make sure it doesn’t go anywhere, should it go down.


I’ve got $450 of credit on my account and it goes towards the EC2 cost along with quite a few other services.  Here’s the complete list:


  • Amazon Simple Storage Service
  • AWS Lambda
  • Amazon Glacier
  • Amazon Sumerian
  • Amazon Relational Database Service
  • Amazon SimpleDB
  • AWSCloudDirectoryInternal
  • Amazon CloudSearch
  • AWS IoT
  • Amazon Lightsail
  • Amazon Elastic Compute Cloud
  • AWS Elastic Beanstalk
  • Amazon Simple Notification Service
  • AWS Config
  • Amazon Redshift
  • AmazonCloudWatch
  • Amazon Elastic File System
  • AWS Data Pipeline
  • Amazon ElastiCache
  • Amazon CloudFront
  • Amazon Elastic Transcoder
  • Amazon Virtual Private Cloud
  • Amazon Lex
  • AWS Data Transfer
  • Amazon Simple EDI
  • AWS OpsWorks
  • AWS Storage Gateway
  • Amazon Simple Email Service
  • Amazon Machine Learning
  • Amazon SageMaker
  • Amazon Elastic MapReduce
  • AWS X-Ray
  • Amazon DynamoDB
  • Amazon EC2 Container Registry (ECR)
  • AWS Glue
  • AWS Budgets
  • Amazon Polly
  • Amazon Route 53
  • AWS Support (Basic)
  • Amazon Elasticsearch Service
  • AWS CloudTrail
  • Amazon Rekognition
  • AWS Key Management Service
  • Amazon Simple Queue Service
  • Amazon AppStream
  • Amazon Kinesis


So, seeing that EC2 was in the list, and there was no way in hell I was going to burn through that much credit by the end of next year, I decided to purchase some Reserved Instances.  A T3 micro for 3 years and a T3 small for one year, all upfront – total cost was $264 + 20% VAT.  And this charged itself directly to my credit card, with pretty much no confirmation.  To be fair to Amazon, I did click the purchase button, I just expected to see a final confirmation screen that showed me where the funds were going to be coming from.  That didn’t happen.


To be fair, there is documentation to support that:


I just didn’t see it before I clicked the old purchase button 😉


Their support team have been lovely, they’ve managed to cancel the Reserved Instances and are starting the process to refund me.  I upgraded to developer support at $29 a month to get my request actioned sooner.  I’ve not decided whether or not I’m going to keep this in place or not yet – there are benefits to it for sure, and as I start to do more and more with it over the coming months, it might be a good idea.  We’ll see.


Workmail isn’t covered by the credit, thats $4 per user per month, but to be honest, it is so much better than what I’m used to with my previous host, it makes me realise just how great an enterprise level solution it is.



This is absolutely crazy.


Look at the price difference between a t3 micro and a t3 small running spot.


It’s well under half the price.


Much cheaper running it this way than having apache on a t2 small and a seperate RDS instance.


S3 backups running well – or they are now that I’ve sorted out the cron job.  Had a little issue with a misplaced * instead of a 0 which meant I got 60 backups between 1am and 2am rather than just one backup at 1am 😉

Migration complete

Migration complete

So the RDS instance has been terminated now.

I’ve got S3 backed mysql backups running nightly, so I’m saving myself an absolute fortune.

Still some tuning to do and I’m not sure if I’m going to stick at spot pricing or look into reserved pricing.  I need to get some tuning in place first with regards to apache and mysql and make sure with Trusted Advisor that I’m running at the right sort of level.

If I can get it running properly on a t2.nano or a t3.nano I could probably save a small fortune paying just $70ish over the course of three years rather than the on-demand pricing.

Spot is okay, but now my database is running on the same ec2 instance, I risk data loss if Amazon were to terminate my instance.

It’s only the 12th of the month – look at the state of the costs:

We’d be looking at easy $13 a month just for RDS – probably more like $16 a month.  Just too expensive for what I need – and lets be honest, RDS backed mysql is a little overkill for a couple of wordpress blogs.

In terms of EC2 cost:

The biggest cost here is obviously the gaming instance I threw up for a work demo last week – even though I’ve only done 9 hours with it, the cost is $2.48.

In comparison, the spot instance running my web server at the moment (a t3 small) has cost me just $1.22 for 178 hours.

Whilst it was fun to play around with RDS and the security groups governing rules in and out, it’s just not required for my use case.

We’ll see how the next few weeks go 🙂

Route53 config updated, cron job added, wordpress storage offloaded to an s3 bucket

Route53 config updated, cron job added, wordpress storage offloaded to an s3 bucket

Interesting lunch hour today – had to fix a script I’d found to update route53 dns records on reboot because the API has changed since the guy wrote it.  Gave him my additions and it works a treat.

Also got bucket storage set up, so my images are all being server from S3.

More technical data on what I did to come soon.

Exam tomorrow.  Eeeeek.

A few tweaks

A few tweaks

So as expected, I’m going to need to make a few tweaks to the ami and the image running along with adding a bit of scripting magic to make things run more smoothly.

In regards to the ec2 instance, spot requests are definitely the way to go.  The cost saving is insane, and it’s not like this website is business critical.

That said, I would like to keep it up and running which in itself presents a few challenges.

Though the database backend is RDS backed, ensuring that all posts are brought back as they were if anything happens to the website, the EBS is deleted upon vm termination.

This presents some challenges for the photoblog plus any media that I upload here. I either need to find a way to back up somewhere, or I can leverage s3 buckets in order to do something clever.

That’s this weeks challenge.

I also need to implement some cleverness around automatically updating route53 entries from the web server should it move. Found a few articles I’m going to look at this week in regards to that.

So far I’m pretty impressed with the way the knowledge has stuck. Exam on Wednesday then I can concentrate fully on the solutions architect associate course. Full steam ahead on that one.

Tried to get the cloud gaming rig back up tonight but it would seem a month offline has screwed something with steam and parsec and the vpn. I need to go back to that, give it some TLC, update various bits and pieces and remove anything superfluous.

Once I’ve done all that ill snapshot it again, start from scratch and create an easy to follow guide for gaming using a high powered AWS gaming rig and steam 🙂