Onward to the Twenties
Every year I like to take this time to think back on the things I have accomplished, the things I set out to do and failed at, and look forward to the new year and set goals. (For those of you in industry familiar with agile, yes I am do a retrospective on my sprint of a year). However this is the first year in my professional career I get to do a retrospective not only on the year but on the entire decade. So I would like to pick out some highlights of the past year then reflect on the past decade.
2019 In Review
What Went Well
Setting out in 2019, I had quite a few lofty goals in the professional and technological areas of my life. Looking back of my beginning of the year goals, I had learning Scala, Rust, Spark, and Haskell on my list of TODO, as well as gaining mastery in Elm and SQL.
Scala was a goal set forth somewhat for work. I ended up reading through a few scala textbooks and digging through our massive scala codebase at work and got a pretty good understanding and feel for the language. While I really appreciate the flexibility and power the language grants you, I feel like it gives you way too much rope to hang yourself with and too little guidance to prevent yourself from doing so. For example it has many pure functional data structures that are powerful and elegant, however they include things like List.last, which is the opposite of head, which for that data structure is the least efficient thing you can do with that list. You could also accidentally import the wrong mutable List and lose all of your guarantees you assume you got from using the immutable one. One of the most powerful thing about Scala is the ability to define your own operators on your objects. This is very cool, however the easy of doing it doesnt prevent someone from doing it incorrectly or inappropriately and making completely unreadable or unmaintainable code. This paired with implicits make for a language that has the potential to build very powerful and beautiful DSLs, but gives very little guide-rails to prevent you from doing something insane and impossible to understand.
During a change in engineering management at Jornaya, we ended up scrapping our Scala project and I was put in charge of rebuilding our scala application in Python to make it more accessible to our data scientists and to make hiring for it a lot easier. For unfortunately, I didnt get to utilize Scala professional as much as I would have liked this year.
Rust was a personal curiosity for me. I have heard a lot of the hype around its Speed and its new Memory system as well as its ability to be a low level language that looks and feels like a high level one. All of these taglines and rumors were very appealing to me. I took a rust Udemy course and did a few personal projects. Wrapping my head around Rust was the first time I really had to pay a lot of attention to Memory allocation and types. I had dealt with low level programming languages like C and C++ before, but Rust makes you really cognizant about what you are doing with memory at the same time you are solving the logical problems since it reads like a high level language. This ended up fundamentally changing how I thought about performance and memory complexity. The borrow system makes you really aware of scope and I have seen a lot of that awareness seep into my other programming activities that are not necessarily low level.
Ultimately, Rust was a bit too Low level and unstable for my liking for use with personal projects. It has a lot of concepts that I really liked like borrows, pattern matching, and result types, however it also seems to be trying to be everything for everybody, and provides a lot of different ways of doing the same thing. Paired with the major things I would use it for relying on Rust Nightly builds and the few times I was working on a project, the dependencies would walk on me in the few days I was actively working on the project; I had a difficult time taking in the extra work of managing that for the benefits. I will touch more on that later.
Spark is one of the tools that is all the rage right now, and with certain side projects aspiring to handle “big data” I definitely wanted a better understanding of the tool and the ecosystem around it. Luckily for me, Jornaya paid for a Spark Training class for a week and it was extremely helpful. I got some hands on experience and a tour of some of the dos and donts of big data handling in spark. Overall, I enjoyed my time there, however I learned enough to know that I much prefer the software engineering side to the data engineering side.
Haskell was another personal project. After learning and loving Elm, I have been wanting a way to get the level of guarantees and developer joy while writing backend systems. Historically I have avoided Haskell for its complex type system and the notion that is an “Academic Language”. However after getting through type systems in pure functional languages in Scala, and continuing my love affair with Elm, I decided that even if it is just academic, it is a language worth learning if only to follow the guidance of the author of the Algol Programming language Alan Perlis, “A language that doesn’t affect the way you thing about programming is not woth knowing.” So I spend some time through through Learn You a Haskell for Great Good, then going through it again. Then took a detour into Lambda Calculus, the going through the book again. Eventually the concepts of Functors, Applicatives, Monads, Monoids ect started to click. Then I took a Udemy course which was actually just 3 Pact courses bundled together and did a bunch of practice.
Overall, I love and hate the language. It has all of the functional purity and beauty I find in Elm, along with some of the ugly underbelly of Scala and shows its age with certain concepts. Then I found out that the standard Prelude (the part of the language that helps define the language) and the standard compiler settings are generally considered to be trash, but there is no consensus on which set to use instead. Overall it has certainly changed the way I think about programming; I often find myself beta reducing programs in python now) and the fundamental understanding of Functions, Applicatives, and Monads has signiciantly changed how I write my Elm code. I will continue exporing haskell if only because of the amount of Fun I have while writing it.
Strengthening my SQL skills was a thing I set out to do this year. Historically I am strong as domain driven design, and data normalization but have left the actual queries to an ORM and have rarely had to deal with a real time system that required more tuning than that. This year working with Athena and Spark has certainly strengthened my SQL, however I didnt achieve the mastery I set out to.
However I did spend a good portion of the year doing NoSQL design, specifically around DynamoDB, and getting very familiar with how to change the schema of a system to best benefit from the database involved. Optimizing data storage, data access and cost in NoSQL turned out to be a much more interesting problem considering the daily problems I run into work and therefore a lot more effort was spent there this year than in SQL.
My love affair with Elm continued strongly this year, expanding into a number of successful projects at work as well a number of fun side projects. Learning haskell and figuring out how to beta reduce Elm functions made pipelines much more efficient. I have begun to run up against the edges of the language and might consider looking at a language like Purescript for isomorphic js in the new year, however I have had great success with elm at work. Recently we had a contractor making modifications to one of the Elm projects and a number of things happened. Firstly, unlike usual, there was very little worry with the code because I was sure if it compiled it would at least run without exceptions. Secondly the code looked pretty, elm-format has made a lot of the problems with code review go away. Lastly, and probably most importantly, during code review, since I didnt have to worry about formatting or if it worked, I was able to look at the types and find a logical business error in the code, - without even running it. Looking at the types, I was able to obviously see a place where he needlessly reduced cardinality in a data structure, and therefore made an impossible state of the application possible to represent. The code review feedback was pretty straightforward, and he was able to correct it and produce a flawless working version in very little time. That interaction with someone who had no knowledge of the programming language to start to get in and be able to have a meaningful interaction during codereview to produce production quality code is something I have never experience before.
On the less technical front, this year I was able to take 2 vacations with my family and inlaws to Hawaii and to Poland. Both were very beautiful and I got to play with my new Insta360 camera to take some beautiful footage. I had a lot of fun continuing playing in VR this year, and I will try to post about my thoughts on that soon.
Professionally, I took over as the Technical Lead for Jornaya and lead a team in the rewrite of a legacy system to Python. Doing so had me delve a bit to Graph Theory, Category Theory, python performance tuning, Data Schema design, and figuring out ways to build a framework that make relatively complex business logic easy to express, understand, and maintain. The resultant system ended up being a library, that uses python type annotations to build a directed acyclic dependency graph that can ben executed in parallel. Pairing that with some clever caching mechanisms and intelligent tree shaking to minimize the work done, we ended up with a system that answers the same questions as the legacy scala system, but in less time and can be executed in an AWS Lambda context. Ultimately this means we have a more maintainable codebase that more people have access to (we have a lot more people who can understand python than scala) that runs faster and about 5x cheaper than the legacy system.
One of the things I am proud of this year was running an Intro to Python Training course at work. It was a Computer Science 101 and Python 101 course combined that I put together myself and setup training and homework on. While I have taught C# at the highschool level before, training peers and even my bosses in the various depths of python was a great experience for me. There were people in the audience with Computer Science degrees who were just learning python as well as people from the support team who never programmed before, so splitting the difference and holding that audience for a few hours a day for a few weeks was challenging and very rewarding.
What Didn’t Go So Well
Before I get started, I wanted to say a bit on failure. At Barcamp this year, I went to a talk about what made teams good and we did an exercise where we designed a new conference in 5 minutes or less. My team ended up with FailureCon, a conference about failure and trying to destigmatize failure and instead embrace and and learn from it. In my career, I have learned that is probably the best thing you can do for yourself. Everyone fails, and as long as they get up and learn from it they get better for it. Like a baby learning to walk, dont make a big deal of the times they fall, instead learn and get up and keep moving. Failure is the stepping stones to success. This is a much larger topic that I am sure I want to cover more in depth at some point this year, but I wanted to at least touch on it before it seems like I am shamelessly flaunting my failures for no good.
At work this year, our engineering department shrunk in both size and the number of technical people comprising the team. All of them had different reasons for leaving and significantly different impacts when they left. Through no choice of my own, I ended up in a position where I, early in my career, decided I didnt want to be in any time soon; the smartest person in the room. Now I dont mean to say that in a bragging way by any means, and there are definitely people at work much smarter than me in various subjects and in their area of expertise. However, in my area of expertise, there is no longer anybody I can go to with a question, no one who can mentor me in the things I am going after, and that has forced me to turn elsewhere for professional development. Sure Jornaya paid for me to learn Spark, however that is outside of my area of expertise and I learned that I dont want to do Spark for a living. Early this year my technical growth was no longer guided by someone I could talk to at work, and that is something I have left previous position because of. Ultimately I traded my technical growth for leadership growth in being the tech lead, however it was not planned and I probably would not have chosen that path myself this early.
The rewrite of the legacy system, while technically an outstanding success, overall had non-technical issues that we failed to handle properly or anticipate. While writing the new system, we followed the principles of good software engineering and functional programming to produce a performant, reliable, deterministic system. What we failed to realize sooner than later, is some of the choice to do the right thing ended up making fundamental differences between the new system and the legacy system. One of the key things being determinism. One would assume that good engineers would produce a system that if you put the same inputs in, you get the same outputs out. That assumption about the old system cost us about 2 months after the project in testing and investigations on why the new system was not producing the same results as the old system all the time. This ended up spawning committees and spikes and sprints full of research to figure out why these minute differences occurred in the edge cases. We should have challenged the assumptions and focused on delivering business value instead of trying to reproduce a system that is broken.
On the same project, moving to a serverless infrastructure away from a beefy Scala cluster had some unexpected consequences that increased our time to delivery. A few assumptions about clients drove these issues. The first is that people update their systems. A good number of clients had ancient versions of OpenSSL installed which the Cloudfront Distribution in front of the new infrastructure no longer supported for security reasons. This prompted a lot of research and back and forth with clients to upgrade their systems to allow them to use the new system. The second is the assumption that people follow the API spec handed to them when calling an API. The assumption was compounded by the assumption that the legacy system properly followed RESTful Standards for error handling while calling itself a REST API. Some clients were intentionally sending malformed data instead of not sending data they didnt half, and instead of getting errors, the legacy system was silently failing. Now had that been a documented behavior, or if we had better monitoring on the legacy system to understand how our clients used it the system, we may have avoided a lot of this pain, but alas it cost us quite a bit of time and a large number of client interactions that we didnt plan ahead of time.
One of the failures we are working to correct this year was a almost fanatic rigidity in our “agile” process that ironically made us less flexible and less agile. Looking at the Agile Manifesto, it blatantly states that we value “Individuals and interactions over processes and tools”. Yet somehow, we spend an absurd amount of money and time maintaining a process management tool that nobody likes and tyring to make sure that every little detail in the software was correct down to the hour instead of shedding ourselves of all of the overhead and just having strong communication. This blind adherence to our tools and procesess is exactly what Agile preaches against yet we somehow managed to do it in the name of Agile. On a number of projects it has caused us to deliver the wrong thing and has put too many steps between Ideation and Delivery. Since moving into a tech lead position I have been fighting to remediate these practices, however people are much more comfortable with what they know and having a rigid process.
I set out a goal this year to speak at a conference, and while I did a rather successful talk at Barcamp this year on “Intro to Elm”, I don’t think that counts in the spirit of the goal. I did very little preparation, I didnt need to submit a proposal ahead of time, and since I wrote the talk the day of Barcamp, I didnt get to rehearse or practice at all.
What To Do or Change in the New Year
So on the list of programming languages for this year I have: Purescript, Elixir, and Go.
Purescript to see if I cant expand my frontend elm skills deeper into functional programming, and possibly expand it into Node. This would allow for Isomorphic Purescript, which would have all of the power and feel of haskell or elm but the ability to use one language in both frontend and backend.
Elixir has been on the list of languages to learn for a while and I have done a few projects in Phoenix, (the Rails of Elixir). I love the functional purity and the blazingly fast performance, and there are a few interesting things like its Actor model and the entire BEAM ecosystem that are really intriguing to me. However I have put it off in the past for the same reasons Rust’s performance didnt wow me; Cloud Infrastructure has opened an avenue for cheap scaling that makes a lot of those performance problems obsolete. Historically If I had a python applciation that could do 1000 req/s, and average 100ms per, moving to rust or elixir that can do 100,000 req/s and average 1ms per could potentially mean a 100x savings on scaling costs. However running my python application serverlessly, means that it will likely cost me less than running a single Rust or Elixir box and they will handle the same traffic. So the ability to do low level scaling is much less important when
I can pay less money to have AWS just solve it for me serverlessly. However I still see myself getting back to Elixir for the same reasons I went back to haskell, to learn new things and change how I think about programming.
I actually spend some time digging into Go this year, and I like its simplicity and its unique concurrency model, however I hated somethings about the language early enough that I abandoned it pretty early. This was hot off of the heels of learning Haskell, so functional purity and elegant error handling were fresh in mind when I found Go was very mutable and really didnt do error handling. That was enough for me to drop it for the year. However it elegantly solves a number of problems such as packaging and distribution, and its concurrency model has boasted some numbers that are hard to ignore, so I will be taking another attempt at Go this year.
On the non language front, I would like to take another crack at Thespian and Docker. Docker seems to be ubiquitous these days in the ops community and while I have used it in a number of situations, we mostly had the opinion that Serverless was the better alternative. However there are some shortcomings of Serverless that Docker can handle so I will be taking a deeper dive into that ecosystem this year. One of the use cases I am looking at is pairing a lightweight docker system with a Thepsian Actor System. I have written about Thepsian in the past as the better Distributed Actor System in python and have had surprisingly cool results from using it. I think pairing Thespian and Docker would allow me to build a resilient and scalable system that can handle longer lifetime workloads than Serverless is fit for.
At some point this year I would also like to take a deeper dive into GraphQL. I have had to interface with a few GraphQL API’s this year and for the most part I feel like there is a lot more hype than it deserves. I feel like REST has a few shortcomings that GraphQL helps solve, but I feel like the optimizations and security and clean cut permissions of REST APIs have not been properly solved with GraphQL yet. However I would like to look at GraphQL as a single stop API Gateway on top of a system of REST microservices. This should allow me to pair down the advantages of GraphQL with the shortcomings of REST and vice versa. That paired with the really cool GraphQL-Elm Library for ensuring types between the front and backend makes for a compelling second look.
The Decade in Review
Looking back a decade, in 2010 I was a hopeful Computer Engineering Student at Stevens Institute of Technology. Little did I know the latent passion for software engineer that would come to shape my career. I was a bit overconfident and consequently ignorant to not only my passions but the power of good goal-setting and retrospectives. Within the year I would get distracted by an interesting software problem and drop out of college. The ensuing identity crisis and introduction to the world of computer science altered my course quite a bit.
Then reflecting on my time at Fox Group, as the Director of Research and Development, having the, in retrospect monumental, opportunity to bridge my skills with circuitry and signal processing with those of my budding software engineering experience. Going through the engineering practices of gathering requirements, designing, developing and maintaining systems. Having the light experience of managing people and dealing with budgets. That is really when I started to really enjoy the software side of things (it was much faster and cheaper to iterate than the hardware projects or dealing with million dollar industrial machines). I certainly miss a good portion of having hardware or physical things to interact with when solving problems. It was during that time I developed a passion for 3D Printing and how can I use it as a tool to solve problems.
I look back to my original blog at the time and almost cringle at my post “Why Python is Enough for Me”. That was during the era of Ruby and Ruby on Rails so it wasnt exactly a popular opinion. Then learning more and more programming languages and seeing how they affected my engineering practices thinking how young and naive I was at that point in my career. For the majority of that time, I was programming by myself in a silo. Nobody could tell me what I was doing was wrong or stupid, and the only metric that mattered was “Does it Work?”. Looking back at one of my larger programs back then I can barely read what I was doing. I had a program that consisted of a single 4000 line python class. It used TKinter to render out a UI. It actually solved the problem of scheduling many multi faceted jobs across many machines to deliver on time. However it did some black magic fuckery in a number of places doing things like actually reading pixels in the UI to see if a block had been scheduled there or not.
The next big era for me was Nuix and Sungard/Comcast. That time was my intro to the ecosystem of software engineering. I learned version control practices, and frameworks and working in teams. I learned TDD and BDD and writing a crapton of tests because I will have more than 3 people using the software that I write. I learned about maintainability and communication in teams. I learned about pushing back on requirements and dealing with much larger projects that multiple teams are working on and managing inter-dependencies. Most importantly I learned exactly how much I didnt know and was lucky to get the resources to start to close that Gap.
Then the last third of my decade and subsequently the latest chapter of my career has been Jornaya. Living through the startup lifecycle, the growing pains, the failures and course corrections. Getting intimately familiar with business value and balancing doing it perfectly and getting it done on time. Taking ownership of the Software Development Lifecycle and owning the delivery of customer value. Beginning to get mastery in a number of different areas I have been working the better part of a decade to achieve.
It has been quite a ride, and I am looking forward to the next 10 years.