User's blog

Category: Clever Idea

  • Clever Idea: Serverless, Cloud Native CI/CD

    Clever Idea: Serverless, Cloud Native CI/CD

    If you’ve met me for more than a few minutes you’ve heard me talk about my passion project, Leave the House Out of It (lthoi.com). If you’ve really paid attention to my blog posts you’ve caught that a couple years ago I rearchitected the app to move to an event-based, serverless architecture on AWS. After a year of not doing very much with the project I’ve had the itch to make some upgrades (more on this next year). Before I did, I wanted to upgrade the CI/CD pipeline I use to manage the code.

    While I had moved away from containers/EKS, I did keep the containerized Jenkins that had been deployed alongside my code on the EKS cluster. I got an EC2 server, installed docker, and deployed the image there. Unfortunately, on an EC2 server Jenkins quickly became both disproportionately expensive and pretty slow. The cost was due to the inefficiency of running a Jenkins server for an app you deploy infrequently. In fact, because the app was all serverless and low volume, I was actually paying more for my Jenkins server then for all of the rest of my AWS charges combined. In spite of the cost, the performance was pretty terrible. Jenkins lost access to deploy agents across my cluster and instead churned away on an under powered EC2 server. This caused larger runs of the pipeline to take upwards of 9 minutes.

    Over the last few weeks, I’ve taken the final step to the AWS native world and adopted CodeBuild, CodeDeploy, and CodePipeline to replace my Jenkins CI/CD pipeline. My application has 5 Cloud Formation stacks (5 separate Lambda functions along with associated API gateways and DynamoDB databases) and an S3 bucket and CloudFront implementation that hosts the Angular UI. I ended up with 6 separate CodeBuild projects, one to build and unit test each of the lambdas and one to build the UI. The one that builds the UI I took a shortcut and simply used the build service to also deploy. For the 5 lambdas, I wrapped them in a CodePipeline along with AWS CF Deploy jobs for each.

    The only tricky part I found was that I did not want to refactor my lambdas in to an “Application” so I could not use AWS CodeDeploy out of the box. That made it difficult to use the artifacts from AWS CodeBuild. The artifacts are stored as zip files meaning I can’t directly reference them from the CloudFormation for Lambda which is expecting a direct address of where it can find the .jar file (I wrote the Lambdas in Java). I got around this by having two separate levels of “deploy”. In the first one, I use an S3 “action provider” to unzip the build artifact and drop it in an S3 bucket that I can reference from the CloudFormation. The resulting code pipeline looks like this:

    The results are compelling on several fronts:

    1. I was able to shutdown the EC2 instance and all the associated networking and storage services. It should save me a total of ~$50. It looks like on normal months I’ll be in the free tier for all of the Code* tools. So it will literally be $50/month right in my pocket. I expect all but the biggest software development shops are going to better with this model than with dedicated compute for CI/CD.
    2. In my case, I also sped up the process considerably. I had been running full build and deploys in around 9 minutes. This was due to the fact that I was using one underpowered server. AWS CodeBuild, is running 5 2 CPU machines for build and running my deploys concurrently. That has dropped my deploy time to about 1.5 minutes. (note: In fairness to Jenkins, I could have further optimized Jenkins to use agents to deploy the AWS stacks in parallel… I just hadn’t gotten around to it)
    3. The integration with AWS services is pretty nifty. I can add a job to deploy a particular stack with a couple of clicks instead of carefully copying and pasting long CLI commands.
    4. In addition, this native integration makes it easier to be secure. Instead of my Jenkins server needing to authenticate to my account from the CLI, I have a role for each build job and the deploy job that I can give granular permissions to.

    There are very few negatives to this solution. It does marry you to AWS, but if you have well written code and a well documented deployment process it wouldn’t take you long to re-engineer it for Azure DevOps or back to Jenkins. It’s definitely going to be my way forward for future projects. Goodbye Groovy.

  • Clever Idea: Leave The House Out of It (my app)

    Clever Idea: Leave The House Out of It (my app)

    One of my favorite experiences has always been going to Las Vegas on an NFL Sunday.  Looking at all of the potential wagers, reading up on all of the games, talking with my friends about what they see, and placing a few bets while drinking that stale coffee at an abandoned Blackjack table is a great way to start a day.  The best part of any weekend in Vegas though, is sitting at the little bar outside the sportsbook at the Venetian (the one where you can get free drinks by occasionally playing $1 Video Poker) and watching 14 games over 12 hours on a dozen big screen TVs.  I try to make it out a time or two per season to risk a couple hundred dollars.

    Unfortunately, as a software developer and lover of discrete math… I know just how stupid this is mathematically.  The longer you gamble on anything against the house (the gambler’s term for the casino), the less likely you are to be ahead.  Casinos use a “spread” to make sure they’re getting more than their fair share of every bet. The easiest way to illustrate this is to look at a bet that is perfectly even, like the flip of a coin.  The way the House prices bets, if you wanted to win $100 on “Heads” you would have to bet $110; same with tails, risk $110 to win $100. While you might win this once or twice, the House will eventually catch up with you.  That’s why I only make Football bets in Vegas, I only do it a couple times a year, and I never bet much.

    Over the last few years though, I’ve found better and better ways to enjoy many Sundays the same way I enjoy them in Vegas (although I trade the Venetian for my local sports bar with Sunday Ticket) and without paying a tax to the House.  It started 6 years ago when my brother and I hit upon the idea that each of us could pick a few NFL games for a friendly wager and the other person had to take the “House” side, but couldn’t charge the tax. This way as long as I picked better than he did, I would end up winning money.  This allowed us to watch the games together and enjoy them like we were at the casino, but without the need to fly to Vegas and pay the casinos (worst case scenario, I lost money and was paying my brother).

    Four years ago the idea grew again, we wanted to find a way to reduce the record keeping burden and include more of our friends.  That year, the first version of Leave The Houst Out of It (LTHOI.com) was born. The idea behind it is a simple extension of what we’d been doing for a couple of years.  We would build a “League” of friends and when any one of them wanted to pick a particular game, we would all take a tiny position against them.  Having a lot of people playing with us made the system work even better; now when someone takes a bet that I am forced to take the other side of, I am sharing the responsibility with all the other people in the league.  Over time, these “house positions” tend to even out such that we are truly “Leaving the House Out of It”.

    We soon discovered that this is an idea that doesn’t even require that money be involved.  With the website version, instead of actual gambling… my friends and I discovered that we can just play for pride.  Our Win-Loss record and earning against each other are tracked alongside the rest of the members of the “league” and we can battle for who has the best record.  Now I get all of the thrill of the Venetian with none of the risk!

  • Clever Idea: Serverless is Commitmentless

    Clever Idea: Serverless is Commitmentless

    I’ve decided to start a bit of a series of blog posts called “Clever Idea”. I’ll use the space to talk about something I’m working on either in my hobby project (an app for picking Football games with your friends) or at work that I think is clever. The intent will partly be to share a technique or technology that I think some people reading might be interested in. I’ll also be hoping that occasionally someone points out to me that I’m not as clever as I thought and there’s actually a better way to accomplish what I’m trying to accomplish.

    This season will be the first season that I’m planning to open up my little app to more than just my immediate friends. I’m planning to let my friends invite their friends to form leagues and hopefully end up with a couple dozen leagues. Being somebody who has floated from infrastructure to software development and back again throughout my career, I know that this doesn’t just mean developing new screens for adding leagues and registering new players… This means, planning out my infrastructure so that the application can scale appropriately.

    Like any good architect, I started with the usual questions… how many concurrent users do I think I will have? How many rows in the databases? etc… The problem here is, like most software projects, I don’t really know. It could be that no one will be interested in the app and I’ll only have a couple users. It could also be that sometime during the season it spreads like wildfire and I have a hundred leagues all of a sudden.

    I have a micro-service based architecture. Last season it consisted primarily of containerized Spring Boot apps running on a EKS (Kubernetes) cluster and communicating with a relational database deployed on AWS RDS. This architecture is certainly scalable relative to the vast majority of monolithic applications that exist in enterprise IT today. I had an auto-scaling group setup to support the EKS cluster and it would scale down to two nodes and up as far as I needed. Without re-architecting the database, it probably could have scaled to several hundred leagues. It’s pretty flexible, allowing my AWS bill to run from ~$200/mo (a small RDS server, a couple small K8s application nodes, and the EKS control plane) to a cluster/db large enough to support a few hundred leagues with the only down-time being when I switched from smaller DB instances to larger ones.

    It’s not nearly as flexible as Lambda / DynamoDB though. When I rebuilt the application this year it was with that flexibility specifically in mind. The app now runs entirely on Lambda services and stores data (including cached calculations) in DynamoDB. Both of these are considered serverless which means AWS ensures that my Lambda services always have a place to run and my DynamoDB data is always available, actually providing more reliability/availability than the K8s/Cluster architecture I had built. More importantly for this post, Lambda and DynamoDB are both “pay by the drink”. With Amazon, those are very small drinks:

    • The base unit for Lambda is 50ms. A typical call from the front-end of my app to the backend will result in a log line that reads: “REPORT RequestId: 6419f5ca-f747-4b77-a311-09392fc6bcc3 Duration: 148.03 ms Billed Duration: 150 ms Memory Size: 256 MB Max Memory Used: 151 MB”. AWS charges (past the free tier… which is a nice feature, but not the focus here) $0.0000002 per request and $0.0000166667 per GB/s. For 10,000 calls similar to the one above, I’d be charged $0.002 for the number of calls $.0064 for the memory consumed. We do need to remember that in a micro-service architecture, there are a lot of calls; some actions will result in 5 or 6 Lambda services to run. However, based on the numbers above… if I end up with only a handful of users, my Lambda charges will be negligible.
    • For DynamoDB, the lower end of the scale is similarly impressive. Charging $1.25 for a million writes, $0.25 for a million reads, and $0.02 per 100,000 DynamoDB Stream Reads (more on these in another post). I know from last seasons that if I only have a couple of leagues then, after refactoring for a big data architecture, I will end up with ~5,000 bets to keep track of that are rarely ever read (there are cached totals) but often get written and rewritten (let’s say 5 times per bet), ~300 games that are read every time a user loads their bet page (lets say 12,500 times players read all 300 games), and ~25 player/league records that are used on nearly every call from the UI (let’s say users are queried 50,000 times). If I use those conservative guesses for usage, a small season would cost me $0.036 for the writes and the resulting DynamoDB Stream Reads and $0.95 to satisfy all the reads of the games and leagues. That means my relatively small league is costing me less than $1.00 for the whole season.

    The reason I titled this post “Serverless is Commitmentless” is that I can build an app and host it on redundant compute and storage without really paying anything at all. If I get lucky though, and the application is a huge success, this architecture would scale to thousands of leagues before I need to rethink my DynamoDB indexes. As long as my revenue goes up faster than the AWS hosting fees when that happens, I have zero upside or downside risk from infrastructure cost when starting a new project.