For an organization with more than a handful of development teams, the hardest decision in technology right now is to decide where the cutoff between your platform teams and your software teams should be. As with most things in the world today there are loud people making loud claims on both side of the debate but the real answer is somewhere in the middle.
In one camp there are those that are screaming “developers just want to code”. They recognize that every member of the product team is expensive and they don’t want them spending hours selecting and troubleshooting infrastructure that they aren’t expert in. They also recognize the efficiencies possible if the infrastructure team creates standards.
In the other camp there is a really solid argument for self-sufficient product teams. We have all seen software product teams that know more about what infrastructure they need than the infra team trying to work with them. This is how shadow IT starts. Creating true DevSecOps teams that are responsible for everything their app needs also allows the organization to more easily invest in (or divest of) individual product teams.
How much will I sound like a consultant trying to make a few bucks if I say, “the answer is somewhere in the middle and really needs to be determined on a case by case basis”? Let me try to reward you for reading this far by breaking down a few of the things worth considering as you make this decision:
Are the products that your software teams are creating infrastructure dependent (e.g. low latency, require GPUs, edge, etc…). In this case, lean toward creating product teams that build their own infrastructure. Avoid the temptation to create single workload platforms.
Public cloud (and a willingness to commit to one cloud instead of attempting to maintain the ability to deploy any workload to any cloud) is a short cut in this debate. They allow you to build platforms that product teams can leverage simply through IaC and GUIs that would be cost prohibitive to create on premise.
Consider the use of “paved roads”. Make it really easy for app teams to “just code” without taking away the ability to customize the infrastructure if required.
If you’re going to try to change your organization’s focus from one side of the continuum to the other, do NOT underestimate the cultural inertia.
The absolute worst thing you can do is to not make a decision on this. You’ll end up with platforms that are over specialized and dev teams that don’t know how to use them. You must pick where you want to land on this continuum and make sure both your dev and infra teams are funded and motivated accordingly.
I recently took the job within Kyndryl of helping to establish and expand our Application and Data Consulting Practice. Kyndryl is known as an infrastructure company, there’s no way around that and I got lots of questions about why anyone would want to run the software portion of a company so focused on infrastructure. The truth is, for the first time in decades, the developers need the help!
In considering whether to take the job, I recalled a moment a few years ago while I was still working for a big bank with thousands of developers. I had made the switch from working on the software development side of the house to running the implementation of this bank’s Kubernetes clusters. I distinctly remember a morning when a young engineer and his manager came in to my office with a question about a support ticket they had received. One of the application teams had entered a ticket for our monitoring team to install a performance monitoring agent on a particular container. They gave the container’s full name and included approval from their management team for us to log in to the container for the install.
The application team had the ability to modify their own docker file to install the agent. Further, the fact that the application team wanted to software installed meant they had missed that containers were immutable by design and the value of having their container image and dockerfile stored and versioned. I realized then that the tables had turned. For the first time since punch cards gave way to Cobol and Assembler, the infrastructure teams were not holding back the development teams.
Of course it’s not universally true. The most advanced dev teams at most companies are still constantly challenging the infrastructure and security teams (even the cloud providers themselves) to provide more tools and technologies faster. However, there are a lot of software development teams that are not ready to make use of the advanced infrastructure that’s available to them. There are several reasons for this:
The teams aren’t trained on and don’t understand how to use infrastructure as code, horizontal scaling, asynchronous communication, and so many other things that are required for them to unlock the power of the infrastructure they’ve been given.
They are working with workloads that are stuck on legacy infrastructure like the mainframe.
The code they are working with is too ingrained in legacy development models.
The data is too disorganized and not secure enough to move to the cloud.
ERP or COTS workloads won’t allow them to leverage more advanced infrastructure.
My new team in Kyndryl is focused on helping your development team overcome these challenges and unlock the value of the infrastructure now available to you.
I found a lot of great nuggets in this book, but I’m not sure I’d recommend it to everyone. If you’re trying to sell your boss on a move to AWS, definitely give her this book. If you’re trying to figure out what your IT priorities are, I’d recommend Leading the Transformation or The Pheonix Project. If you insist on reading something that’s written by an AWS Employee, I’d even try War, Peace, and IT by Mark Schwartz.
While I found some nuggets that I enjoyed and found informative, I found most chapters just identifying a problem and then explaining how being in the cloud mitigates that problem. It’s not really surprising considering it’s 53 chapters in only 298 pages (at least how Kindle counts it). One big exception to this is Chapter 5 and Chapter 53 outline processes and are a little more detailed (but each of them could be an entire book). The other little nuggets that I liked were:
Chapter 23’s discussion of the future of Managed Services in the cloud was interesting. I believe the combining of migrating to cloud and managing applications (essentially turning them in to SaaS) was a great point. Also, the highlighting of DevOps as an important feature of an MSP.
I liked Chapter 33’s discussion of Hybrid Cloud. Too often AWS and their surrogates talk about Hybrid Cloud as a fantasy. Unless you have the kind of capacity that it’s worth building an entire datacenter that runs just for you (like Netflix), you’re probably better off having everything in the cloud. The problem is, you can’t just put it there by magic. Orban does a nice job recognizing Hybrid Cloud as a stage, maybe a VERY VERY long one, but not a destination.
I liked the discussion in Chapter 46 about a company’s vision for moving from Centralized IT to Exponentially Growing IT.
I actually finished the Unicorn Project a couple weeks ago and didn’t take great notes on it, so this won’t be as extensive of a review as I often provide. I did enjoy the book, and like it’s predecessor (the Pheonix Project) it does a great job of showing how some of the best processes in the industry can come to life in a specific company. It does a beautiful job covering both the DevOps revolution at the team level with containerized dev environments, versioned APIs, and everything else required to decouple deploys, automate test, and implement CI/CD. It also touches on the transformations necessary to abstract your mainframe/legacy environment and the evolution of management thinking around experimenting with parts of your org while gaining efficiencies in others.
My only real critique is that it’s not very realistic. It demonstrates how powerful these concepts CAN be by taking an ideal company. The ficticious company in the book has a CIO who understands the need to empower engineers and keep the vision high-level (rather than command and control), a great agile coach, an operations team that wants to put in the extra work required to give control to the developers, and a development team that is excited about experimentation… not something I often see in clients. It also glosses over things that I’ve seen very difficult. At one point it takes a development team over three weeks of hard work to put together an automated test suite for their application that includes (presumably) either mocks or test instances of upstream and downstream systems. They’re right to call this hard work, but I’ve rarely seen developers tackle it in a few weeks, even on a backend system without a UI.
The best use of this book is to give it to a CIO or CTO who struggles with what success looks like in their organization. If you still have someone who’s trying to count server consolidation ratios or incident tickets, give them this book.
I’ve decided to start a bit of a series of blog posts called “Clever Idea”. I’ll use the space to talk about something I’m working on either in my hobby project (an app for picking Football games with your friends) or at work that I think is clever. The intent will partly be to share a technique or technology that I think some people reading might be interested in. I’ll also be hoping that occasionally someone points out to me that I’m not as clever as I thought and there’s actually a better way to accomplish what I’m trying to accomplish.
The Problem
As some of you know (and I’ll be posting about more), I have run a little app for picking football games the last 4 or 5 years. It allows a “league” of people to keep score on who does the best picking games against the spread. This year I am making an improvement to allow users who are already in the league to form new leagues. A user is asked to specify the email addresses of everyone they’d like to participate in the league. If you specify someone who is already in the league, then they are added to your list of players. The first problem comes in when you want to invite someone who isn’t currently enrolled in the system.
I use Google OAuth for authentication. So I will need to “pair” the identity that an existing user has specified as wanting to be in their league with a logged in Google ID. Note that I feel this is necessary because not everyone uses the email address associated with their Google ID as their primary email address, so I would not want to make it impossible to join unless the person inviting you happened to guess the right email. One of my friends gave me the great suggestion of providing users a link. If I send an email to the new player and say they’ve been invited to play in a league, I can include a link with a query string parameter that includes their “user id” (a random GUID created by my system). This is where things get more difficult. Unfortunately (in this case), I’m using Angular for my frontend. Angular creates a downloadable browser app that players in the league use to interact with the system and that keeps itself refreshed through backend API calls. I’m a huge fan of Angular, BUT it makes my “pairing link” impossible.
Angular is a fully downloaded app that doesn’t actually go fetch other pages when the URL changes (these are handled client-side through the Angular “router”). Consequently, I can’t send someone directly to the “pairing” section of my app because the URL doesn’t “exist” on the web. In their deployment instructions, Angular themselves recommend forcing all incoming traffic to a specific URL via your app server. Seemingly this leaves me with no choice but to use an unfriendly system where I force users to land at my strange website for the first time, trust it enough to “login” even though they aren’t yet users, and paste their “pairing ID” (the GUID that represents them in my system) in to the system so it recognizes who they are.
The Clever Idea
Once you hear the idea, it seems obvious. However, I had fully coded the crappy user experience mentioned above before I came up with it… Just build a simple webpage outside of your Angular deployment artifacts, but located alongside your app. This means that Google will recognize the host as white listed when someone tries to login, your users will trust it as your site, AND you can send people to a specific page with query string parameters. Below is the simple HTML/javascript page that I setup for the purpose (you can find it here on git):
<html lang="en">
<head>
<meta name="google-signin-scope" content="profile email">
<meta name="google-signin-client_id" content="<id>.apps.googleusercontent.com">
<script src="https://apis.google.com/js/platform.js" async defer></script>
</head>
<body>
<h1>Welcome to the New Player Page for LTHOI</h1>
To pair, please follow these steps:
<ol>
<li>If you don't want to play Leave the House Out of It, do NOT fill in this form. Your invitation will expire and the person who invited you will be alerted. You will be removed from our database entirely.</li>
<li>Signin to your Google Account.
<div class="g-signin2" data-onsuccess="onSignIn" data-theme="dark"></div></li>
<li>Fill in your first name and last initial.<br>
<input type="text" id="fn" value="Your First Name"><br>
<input type="text" id="li" value="Your last initial"><br>
<button align="center" onclick="onSubmit()">Accept Invitation to Play Leave The House Out of It</button></li>
</ol>
<p id="error"></p>
<script>
var access_token="Unregistered";
function onSignIn(googleUser)
{
// The ID token you need to pass to your backend:
access_token = googleUser.getAuthResponse().access_token;
}
function onSubmit()
{
// Get the user_id from the QSP to pass
var urlParams = new URLSearchParams(window.location.search);
var uid = urlParams.get('uid');
var request = new XMLHttpRequest();
request.open('POST', ('<BackEnd>/bff/pair?uid=' + uid), true);
request.setRequestHeader('googleToken', access_token);
try {
request.send('{ "first_name" : "' + document.getElementById("fn").value + '", "last_initial" : "' + document.getElementById("li").value + '" }');
if (request.status != 200)
{
document.getElementById("error").innerText = 'That did not work. Are you sure you were invited? logged in?';
}
else
{
window.location.replace("<dev location of LTHOI so they can login>");
}
}
catch (err)
{
document.getElementById("error").innerText = 'That did not work. Are you sure you were invited? logged in?';
}
}
</script>
</body>
</html>
Whether you care about baseball or not. Whether you wanted to hear about it or not. If you’ve had more than three conversations with me in real life, I have told you that one of my favorite things is the fact that in baseball a player’s “Wins Above Replacement” (WAR) can be calculated. Statisticians can say, with a fairly high level of precision, how valuable (in terms of wins) a particular player is to a baseball team over a season. Imagine being able to calculate your colleagues value to the organization with math!
One of the problems with WAR though, is that it does not translate cleanly to fantasy baseball. There’s definitely a correlation between WAR and fantasy value, but it is far from perfect. So, this season, in preparation for my Fantasy Baseball draft, I set out to calculate how valuable all the potential players in Fantasy Baseball were. Settle in, this post is going to be long and worth it. I’ll start with a really quick overview of WAR in real baseball, then discuss the major sections of it that I had to change for fantasy baseball, finally I’ll talk about how the results were somewhat surprising.
WAR in “Real” Baseball.
At a high level, to calculate a player’s WAR we figure out the number of runs a player can be expected to add to a team with his bat/base running and the runs that he can prevent with his glove or pitching arm. We then divide that number by an approximation of the number of runs that it takes to win a game. Finally, we subtract the number of wins that a “replacement” player could have achieved. This is done by looking at the runs contributed/saved by the best player who would typically be available on waivers at that position.
Projection Systems
In order to calculate a players value for this upcoming season, we will need to get a reasonable guess of how they’ll do statistically in 2019. There are a lot of projection services, but I will be using Steamer because I can get it for free on Fangraphs. Steamer doesn’t post their exact algorithm, but I suspect that like most systems, they project a player’s statistics by assuming they will progress in a similar manner to historical players who had similar statistics at their age. So, for example, if you want to figure out how Andrew McCutchen will do this year. You look for players that have had similar careers to this point (if you’re curious… Andre Dawson, Dave Winfield, and Vernon Wells are the most similar) and then assume McCutchen’s age 32 season will be similar to what theirs was. Interestingly, Steamer was developed by a High School math teacher and two of his students in 2008 and has consistently been as accurate or more accurate than systems developed by high profile sports shops that charge for the data (ESPN, Baseball Prospectus).
Calculating “Wins” in Fantasy Baseball
If we’re going to calculate Fantasy WAR, the first thing we have to do is determine what a “win” is. Clearly, in fantasy baseball, runs scored and runs prevented don’t win games. In my fantasy baseball format, you play head to head each week against another player, but there isn’t just one winner. You effectively play 6 offensive games each week against one opponent, one for each statistical category (TB, HR, R, RBI, SB, OBP). For example, if my team has more home runs, total bases, and RBI but my opponents team has more stolen bases, runs, and a higher OBP than we each win three “games”. It’s those wins that matter at the end of the season for playoff seeding. For this reason, the “wins” that I’m counting is winning a single statistical category.
The next question is, how do I take a players projected home runs for the season and turn it in to fantasy wins in the home run category. What I need for that, is to be able to compare the player’s numbers to the average number of home runs a fantasy team will have per game in 2019. In my league their are a total of 120 players playing in each game (there are 12 teams and each team has 10 starting position players). It’s reasonable to assume that the 120 players playing will be the top 120 players according to fantasy drafts that have already been held in 2019 (this data is available in the Fangraphs data). I averaged the projected home runs that will be hit by those 120 players over the season. Since there are ~25 weeks in the season, I divided that number by 25 to get the average number of home runs per player per game in my league. I then assumed that a “win” in Home Runs can be achieved by beating one team (10 players) of those average players. For my league that means that 22.364 home runs is a “win” in home runs.
Calculating “Replacement” in Fantasy Baseball
In addition to knowing how many wins a player is worth, I also have to figure out how many wins a replacement player is worth. My league has 6 bench spots and 2 DL spots; I have assumed that teams will, on average, use half of those for position players. As I mentioned above, there are 12 teams in my league and 10 position player starters. That means that at any moment each team has spoken for 14 position players and the league has spoken for 168 total. Using that information, I have assumed that an available “replacement” player for fantasy purposes is the 169th best position player in any particular statistical category. So, for example, in HRs, the 169th best available player is Roland Guzman of the Rangers who is projected to hit 13 HRs (or .520 HR per fantasy baseball game).
Example of How it Comes Together to Get Fantasy WAR
Let’s take a look at Gincarlo Stanton’s WAR for just HRs. He is projected by Steamer to lead baseball with 45 HRs. As you saw above in the section on calculating replacement value, our replacement home run hitter is Roland Guzman and he will hit 13 HRs. Gincarlo can then be projected to hit 32 HRs above replacement. In the section on “Calculating Wins” we learned that 22.364 HRs should be enough to win the average game in my league. That means that Gincarlo is worth 32 HRs are worth an extra 1.43 wins for whatever team has him.
Comparing The Results to Average Draft Position
The top 20 players in my Fantasy Baseball League
If you’re the curious type and have made it this far, you can find my calculations here. I have pasted the top 20 players by Fantasy WAR above. The column “P ADP” indicates what order they are selected in during Fantasy Drafts that have already happened this year. There are a few things that jump off the page:
Stolen Bases are important. You’ll notice that no one whose projected to get less than 10 stolen bases even made the top 20 and that virtually all of the speedsters are towards the top. The reason this is true is because of how top-heavy the stolen base category is. It only takes 7 stolen bases to win a typical week, so Trea Turner’s 41 steals project to be worth almost 6 wins over the course of a season. Remember Giancarlo Stanton’s 45 HRs were only worth 1.43 wins because it would take 22 HRs to win a game. Be careful with this information though (see deficiency #3 below).
Steamer likes a few of these guys better than people drafting leagues seem to. Roughned Odor and Yasiel Puig in particular show up as much more valuable than you’d expect (most of the rest of the big differences are explained by the stolen bases).
Christian Yelich looks to be the most over-valued player in the top 20, with people selecting him 5th even though his value looks to be 13th. The most over valued player overall is J.D. Martinez who ranks 33rd in Fantasy WAR, but is being drafted 4th among position players.
Deficiencies In Fantasy WAR
Please do not use this list as your draft cheat sheet and then yell at me when your team is terrible. There are several key things that Fantasy WAR and/or my calculation of it are bad predictors of:
Billy Beane is famous for his quote, “My shit doesn’t work in the playoffs”. Neither would Fantasy WAR. A team selected based only on Fantasy WAR would be designed to beat the average team. It would be likely to rack up wins in the regular season, but may not stack up well against a team that has several categories very well covered. I would expect a team built on Fantasy WAR to do really well at racking up wins against bad teams but may lose 7-5 in the playoffs without making any of the 7 particularly close.
I ignored positions for this calculation. There were a number of reasons I chose to do this; the two utility players, the daily switching of lineups, the fact that center fielders aren’t treated separately. That said, obviously the catcher position as well as probably middle infield spots need to be given special attention. Unless I missed something, the most valuable catcher is JT Realmuto at 113th!
The values listed here are only really useful for your first pick. Once you already have Trea Turner and his 41 steals on your team, Mondesi isn’t nearly as likely to win you games. For this purpose, I intend to make a version of this spreadsheet for my draft where I can ignore certain categories once I have “enough” of them on the team.
If you look through this and spot something you think I missed, feel free to comment.
