Replacing the engine while driving: planning safety measures into the project

When a project is created for changing out an existing dependency with another one, there are many factors that should be thought of. If there is a high risk involved if something goes wrong, then more caution would need to be taken. If a lot of effort is involved to add the new dependency, then it may make sense to incrementally ship the new dependency alongside the old dependency. Lastly, depending on the complexity, will a lot of work be done up front to integrate the new dependency, or will it be hacked in then cleaned up after the launch?

These are some of the questions that can be asked to help guide the planning for a project which changes out one dependency for another. The same pattern can be used for similar types of work besides dependencies such as service calls, database queries, integrations, and components, to name a few.

Generally, a project like this can have the following steps:

  • Investigate
  • Refactor
  • Add the new dependency
  • Launch
  • Remove the old dependency
  • Refactor
  • Celebrate

Not all of these steps need to be performed in this order or at all. For example, the launch, add the new dependency, and remove the old dependency can all be completed at once if the risk and simplicity allows for it.

Refactoring may be redundant to mention given developers should be refactoring as they normally make changes to the codebase. I am being explicit about mentioning it here so that the team can use deliberate refactoring to their advantage to make the new dependency easy to add, as well as removing any unnecessary abstractions or code leftover from the old dependency.

A special shoutout goes to the celebrate step, since I certainly know that teams can be eager to move onto the next project and forget to appreciate the hard work put into achieving its success.

Now lets get into the different concerns that can apply to projects like this, and the practices that can help it succeed.

Concerns and Practices

The most important step every project should have, the investigation step informs how the rest of the project should work. During this step as much information should be gathered to give a confident enough plan for the rest of the project to proceed. Some of the actions taken during this step are to understand how the current system works, how the dependency to be replaced is integrated, how critical this dependency is to the businesses operations, how the new dependency should be integrated, and any cleanup and refactorings that should be made before, during, and after the new dependency is added and launched.

A big topic to explore is what the system would look like if the system was originally built using this new dependency. This mental model forces the envisioning of a clean integration with the new dependency, ignoring any sort of legacy code, and free of any constraints within the system. This aligns the team with the ultimate design of the system that uses this new dependency. The team should strive and fight for reaching this state at the end of the project since it can result in the cleanest and most maintainable code. If this is not part of the end goal for the project then the system may carry forward unnecessary code and bad abstractions that result in tech debt piling up for future developers.

Another big consideration is what will the system look like when it is launched? Will the new and old dependency both have to coexist in the system so that there can be a gradual transition? Or maybe the change is small enough that the old dependency can be removed and the new one added in one change and no user interruption will occur? If there does exist a period where the two dependencies need to coexist then how can this behaviour be implemented to fit in with the ultimate design discussed earlier? This may involve doing a lot of refactoring up front to get the system into a state that fits the ultimate design at the end of the project. There also exists the tradeoff that the new dependency can be hastily integrated into the system alongside the other dependency, then all of the hacks can be undone with a sufficient cleanup and refactoring after the launch. Both methods have different tradeoffs: with the up front refactoring, the new dependency is integrated the right way into the system, but may require a lot of refactoring of the surrounding and old dependency’s code. On the other hand, hacking the new dependency into the same pattern that the old dependency uses gets the project faster to launch, but can lead to bugs and integration troubles if the interfaces or behaviour are different. Regardless, at the end of the project, the system should look as if the old dependency never existed.

How much risk is involved with switching over to use the new dependency? If it is very risky then more precautions should be taken to reduce the risk. If there is very little risk then little to no precautions can be taken, and the project can move much faster. Some methods I have used to help with reducing risk are collecting metrics on the different types of requests and responses between the two dependencies. These metrics can be early warning systems to the new dependency behaving incorrectly. Being able to rollback to the old dependency via a deploy or a feature flag provides the flexibility to quickly switch back to the old dependency in case things go wrong. Dark launching the new dependency to production has been a practice I often encourage my team to use, since it allows for testing out the new code in the production environment without affecting real users. Lastly, beta testing with a percentage of users can also reduce the impact since if something goes wrong with the new dependency, only a fraction of the users are affected. Many of these practices are complimentary and can be used together to achieve the best mix of handling risk.

How much effort is involved to add the new dependency? Effort could mean the amount of changes to make to the code or the time involved. If there is a significant amount of effort then it absolutely makes sense to incrementally ship small changes to production. Even if the changes are not live code paths, at least the team can review each change, provide feedback, and course correct if needed. I have seen in small effort projects all work was deployed to production in one pull request. On large effort projects the work was split across many pull requests written and deployed over time by a team. This latter case enabled the team to dark launch the new dependency to production and had the added safety to switch back to the old dependency if needed.

Given the considerations and practices discussed throughout this article, it is best to validate that they will actually work when it comes time to execute. If a team is experienced with these projects and practices then the investigation period can be quicker, otherwise if the team is less experienced with these projects the investigation should be more substantial. Building a prototype can help build confidence in the assumptions made about running the project and guide the team going forward. A good prototype proves or disproves assumptions in as minimal time as possible. Once the team is confident in their plan, start the project, and do not be afraid to reevaluate the choices already made as the project goes on.

A few Gotchas with Shopify API Development

I had a fun weekend with my roommate hacking on the Shopify API and learning the Ruby on Rails framework. Shopify makes it super easy to begin building Shopify Apps for the Shopify App Store – essentially the Apple App Store equivalent for Shopify store owners to add features to their customer facing and backend admin interfaces. Shopify provides two handy Ruby gems to speed up development: shopify_app and shopify_api. An overview of the two gems are given and then their weaknesses are explained.

Shopify provides a handy gem called shopify_app which makes it simple to start developing an app for the Shopify App Store. The gem provides Rails generators to create controllers, add webhooks, configure the basic models and add the required OAuth authentication –  just enough to get started.

The shopify_api gem is a thin wrapper of the Shopify API. shopify_app integrates it into the controllers automatically, making requests for a store’s data very simple.

Frustrations With the API

The process of getting a developer account and developer store created takes no time at all. The API documentation is clear for the most part. Though attempting to develop using the Plus APIs can be frustrating when using the APIs for the first time. For example, querying the Discount API, Gift Card API, Multipass API, or User API results in unhelpful 404 errors.  The development store’s admin interface is misleading as a discounts section can be accessed where discounts may be added and removed.

By default, anyone who signs up to become a developer only has access to the standard API endpoints, leaving no access to the Plus endpoints. These Plus endpoints are only available to stores which pay for Shopify Plus, and after digging into many Shopify discussion boards it was explained by a Shopify employee that developers need to work with a store who pays for Shopify Plus to get access to those Plus endpoints. The 404 error when accessing the API didn’t explain this and only added confusion to the situation.

One area that could be improved is that there is little mention of tiered developer accounts. The API should at least give a useful error message in the response’s body explaining what is needed to gain access to it.

Webhooks Could be Easier to Work With

The shopify_app gem provides a simple way to define any webhooks that should be registered with the Shopify API for the app to function. The defined webhooks are registered only once after the app is added to a store. During development you may add and remove many webhooks for your app. Since defined webhooks are only registered when the app is added to a store the most straightforward way to refresh the webhooks is to remove the app from the store and then add it again.

This can become pretty tedious which is why I did some digging around in the shopify_app code and created the following code sample to synchronize the required webhooks with the Shopify API. Simply hit this controller or call the containing code somewhere in the codebase.

If there’s a better solution to this problem please let me know.

Lastly, to keep track of your sanity the httplog gem is useful to track the http calls that shopify_app, shopify_api and any other gem makes.

Wrapping Up

The developer experience on the Shopify API and app store is quite pleasing. It has been around long enough to build up a flourishing community of people asking questions and sharing code. I believe the issues outlined above can be easily solved and will make Shopify a more pleasing platform.

My Top Tech, Software and Comedy Podcast List

Podcasts are an excellent source of entertainment and learning new things. I find that when I’m doing a mindless task like working out or commuting I can actively focus on something more interesting. Being a student at the moment, I have a lot of time going to and from classes, making food, and procrastinating. I fill up as much of that time listening to podcasts since I enjoy keeping up with the latest tech news, learning new skills and having a laugh.

Click here to jump to the list if you can’t wait.

My Podcast Listening History

I’ve been a huge listener to podcasts shortly before I got my first iPod (iPod nano 3rd generation, 8GB, turquoise) which was sometime during 2006, I think. Back then I started listening to a lot of the podcasts from the TWiT and Revision3 networks. Here I am now, just over 10 years later and I’m still addicted.

Having had a twice a week paper route gave me a lot of mindless time that I soon took over by listening to podcasts. I prominently remember delivering papers in my neighbourhood during a cold, Canadian winter night listening to an excellent holiday episode of Major Nelson Radio. I also remember laughing my ass off to Diggnation, hosted by Kevin Rose and Alex Albrecht, where they share the greatest posts from Digg that week.

Security Now! was a momentous podcast for me. I started listening to it around 2006 when they were at episode 60. Ever since then I’ve been a listener. I can’t thank Steve and Leo enough for their excellent discussion on current security issues and in-depth episodes on various technologies, like their how the internet works series and explanation of the Stuxnet virus. Because I’ve been listening to Security Now! for so long I’ve learned so much about security and the web that I’m practically acing the fourth year computer security class at my University. The valuable knowledge learned will stick around with me forever, already a great asset for my professional career.

The List

Here’s my list of favourite podcasts over the years, all categorized by genre.

Technology

Keeping up with the latest in tech is a given when you’re heavily immersed in the ecosystem, less being a computer scientist.

Security Now!

Security Now Logo

One of the reasons why I’m studying computer science is because of the Security Now! podcast. Every week, Steve and Leo discuss the most interesting and current topics in security. Whether that’s huge corporate hacking, the latest ransomware, IoT security or even various health topics, it’s a polymath of useful information for anyone who’s interested in security.

This Week in Tech

TWiT LogoThe best source for tech news, This Week in Tech is hosted by Leo Laporte, a hero of mine for creating the TWiT network, and continually educating me. I wouldn’t know half of what I know now if it wasn’t for Leo’s work. Each week the latest and greatest tech news is dissected with a representative panel of tech journalists. It’s very informative to hear experts in the area give their opinion.

I remember when Twitter was getting big, it was all that TWiT would talk about for weeks on end. It was even expected: “What Twitter news do we have this week?” was saild by Leo almost every episode when Twitter was growing. Those were the days, when Leo was the #1 user on Twitter. Then the masses came and it went to shit. Okay, I still love Twitter. Rant over.

Hak5

Hak5 LogoBasically a technology hacker/DIY/hardware/software show with a lot of original content. I really got interested in Linux and hacking because of it. Just recently I saw that they’re working on quadrocopters. So cool! One of my favourite segments was the usb multibooting using grub. No need to burn multiple CDs for all of your live-boot isos and images, just put them all on one USB stick and give it a shiny menu to choose which one to use. The show has a kick-ass soundtrack and it looks like they’ve expanded to a new studio and are now producing multiple shows. These guys have grown a lot!

Maximum PC No BS Podcast

Almost forgotten, I remembered this one as I was building the Runners Up section. The Maximum PC No BS Podcast could also go under the Comedy section, but Technology suits it better. On that same paper route I had when I was young I listened to this podcast religiously as soon as each episode came out. Gordon Mah Ung and Will Smith were a perfect pair when talking shop about computer hardware, tech news and building computers for the Maximum PC magazine.

Besides being overly frustrated about certain things, Gordon had a segment called Gordon’s Rant of the Week where he would vent about anything and everything from Star Wars to breaking motherboards to shitty software. Every new year there’s usually a best of Gordon’s rants episode, which is a must listen if you find Gordon’s rants funny.

Comedy

These podcasts are timeless. You can go back and listen to all of them like I’ve done.

Rooster Teeth Podcast

One of the funniest podcasts, various members of the Rooster Teeth company talk about ridiculous stories, gaming, current news and Science. They really don’t know much about Science, but the cast always tries to argue it out until someone say’s something so illogical, the cast and crew burst out into laughter. Moments like these are animated into short videos and posted to their YouTube channel as Rooster Teeth Animated Adventures.

It was the summer of 2013, between my first and second years of university, I was living back home trying to find work. I landed this landscaping job, being paid directly by the owners of a large Caledon estate, to fix up their property. That landscaping was fun but hard work. I discovered the Rooster Teeth Podcast early into the summer. Each day, I would be listening to maybe five or six episodes in an eight hour day. I blew through the backlog of episodes really fast and ended up listening to them all before the end of the summer.

Diggnation

What’s the latest crap from Digg, you might ask? Kevin Rose and Alex Albrecht answered this tough question for 340 episodes from 2005 to 2012. The two would discuss the most interesting news bits on whichever sofa they landed on. Often very entertaining, Diggnation had me dying of laughter.

Software Development

Most, if not all of these Software Development podcasts are timeless. A lot of the topics discussed are still useful today. The only real difference is the adoption of the tools and methodologies. I usually look through the list of earlier episodes and listen to the ones that catch my eye. Once you’re hooked on a podcast, it’s not hard to find yourself downloading and listening to everything they have available.

The Ship Show

Sadly, this podcast just announced they’re ending the show a few days ago, so I’m still in my mourning stage at the moment, but The Ship Show has been a fun and informative source of everything release engineering, DevOps, and build engineering in big and small companies. The panel discusses new tools, methods and philosophies for improving parts of your tech company, often from firsthand experience. What makes this podcast special is that they delve into more of the technical and implementation details, which is great if you’re into that.

Arrested DevOps

The ADO podcast is made for people who don’t exactly know what this whole DevOps thing is about but would like to know. Matt Straton, the creator of the podcast makes this point often as he has learned DevOps from scratch. Each episode goes into depth on a DevOps related subject, often having guests from the industry who are knowledgeable in the topic to add more value to the discussion. A lot of the topics discussed are higher level than what is offered in The Ship Show, but Arrested DevOps is still as valuable since its important to understand the big picture and ask the big questions. Both Arrested DevOps and The Ship Show are complimentary to each other.

Software Engineering Radio

Sponsored by the IEEE, this podcast offers excellent interviews on a variety of Software Engineering topics. The episodes mainly consist of two or three people discussing a specific topic, whether it’s a technology or methodology. The time is taken to give listeners a good idea about the purpose and it’s usefulness. The interviewer often does their homework before performing the interview and therefore asks well thought out questions. Because the episodes cover such a wide breadth of topics, surfing through the past episodes is a must!

Software Engineering Daily

If Software Engineering Radio wasn’t good enough, Software Engineering Daily applies the same format and content to a daily schedule. The amazing producer and interviewer Jeff, also a current host on Software Engineering Radio, has amassed hundreds of episodes covering everything from technology, to business, to soft skills – all pertinent to any software engineer. Dozens of hours of content can be queued up for listening just by skimming through the history of episodes.

I wrote a post on marketing yourself from episode 243 with John Sonmez.

Runners up

Here’s a few podcasts that I’ve listened to for a long time, but didn’t make the list:

Floss Weekly – Randal Schwartz and other hosts interview open source software projects to share what the project is about. Generally pretty interesting, it’s cool to hear what people are doing in subjects that you’re usually not interested in or didn’t know existed.

Tekzilla – Great segments! Veronica Belmont and Patrick Norton were a killer team and shared great tips and tricks to do with technology.

Windows Weekly – Paul Thurott had the perfect level of satire as he talked about Windows products that no one uses, like Windows Home Server (I used it, so I can bash it), and things that people use, like Xbox and new Windows operating systems.

This Week in Google – Gina Trapani and Jeff Jarvais have excellent discussions about the cloud and everything Google.

Mahalo Daily – Veronica Belmont was the best in this short daily podcast format!


Update 2017-01-19: Added Software Engineering Daily

Quick Eclipse Tip

Whenever programming Java, it’s always a good idea to log what the program is doing. Log this error, log that object’s value – it’s a constant occurrence. More often than not, I’m adding this simple line to the fields of every class I write:

private final Log log = LogFactory.getLog(ThisClass.class);

Not only is this repetitive and time-consuming, it is easily automated in the form of a Template from within Eclipse.

The Solution

Adding a text expansion in the form of an Eclipse Template, allows for typing log, press the content assist hotkey, followed by pressing Enter, to automatically insert the Log declaration statement and add the necessary imports. Wow, that was fast, was my initial response. No way am I ever going to type that out manually again.

How to do it

In Eclipse, navigate to Window -> Preferences. In the tree on the left-hand side, go under Java -> Editor -> Templates. Here is the screen for defining text expansions that will be available when using the editor. Click New, enter “log” or whatever of your choosing as the name to expand from. Select the context drop-down to “Java type members”. Finally copy and paste the following into the Pattern field:

private final Log log = LogFactory.getLog(${enclosing_type}.class);
${imp:import(org.apache.commons.logging.Log, org.apache.commons.logging.LogFactory)}

Save it, apply changes, and exit the Preferences window. You are all good to go now!

Note

This assumes that you’re using Apache Commons Logging library for all of your logging tasks. The above template can easily be converted to define your specific logger of choice.