Practicing User Safety at GitHub

GitHub explains a few of their guidelines for harassment and abuse prevention when they’re developing new features. Some of the interesting points in the article include a list of privacy-oriented questions to ask yourself when developing a new feature, providing useful audit logs for retrospectives, and minimizing abuse from newly created accounts by restricting access to the service’s capabilities. All of these points taken into consideration make it harder for abuse to occur, making the service a better environment for its users.

See the original article.

A few Gotchas with Shopify API Development

I had a fun weekend with my roommate hacking on the Shopify API and learning the Ruby on Rails framework. Shopify makes it super easy to begin building Shopify Apps for the Shopify App Store – essentially the Apple App Store equivalent for Shopify store owners to add features to their customer facing and backend admin interfaces. Shopify provides two handy Ruby gems to speed up development: shopify_app and shopify_api. An overview of the two gems are given and then their weaknesses are explained.

Shopify provides a handy gem called shopify_app which makes it simple to start developing an app for the Shopify App Store. The gem provides Rails generators to create controllers, add webhooks, configure the basic models and add the required OAuth authentication –  just enough to get started.

The shopify_api gem is a thin wrapper of the Shopify API. shopify_app integrates it into the controllers automatically, making requests for a store’s data very simple.

Frustrations With the API

The process of getting a developer account and developer store created takes no time at all. The API documentation is clear for the most part. Though attempting to develop using the Plus APIs can be frustrating when using the APIs for the first time. For example, querying the Discount API, Gift Card API, Multipass API, or User API results in unhelpful 404 errors.  The development store’s admin interface is misleading as a discounts section can be accessed where discounts may be added and removed.

By default, anyone who signs up to become a developer only has access to the standard API endpoints, leaving no access to the Plus endpoints. These Plus endpoints are only available to stores which pay for Shopify Plus, and after digging into many Shopify discussion boards it was explained by a Shopify employee that developers need to work with a store who pays for Shopify Plus to get access to those Plus endpoints. The 404 error when accessing the API didn’t explain this and only added confusion to the situation.

One area that could be improved is that there is little mention of tiered developer accounts. The API should at least give a useful error message in the response’s body explaining what is needed to gain access to it.

Webhooks Could be Easier to Work With

The shopify_app gem provides a simple way to define any webhooks that should be registered with the Shopify API for the app to function. The defined webhooks are registered only once after the app is added to a store. During development you may add and remove many webhooks for your app. Since defined webhooks are only registered when the app is added to a store the most straightforward way to refresh the webhooks is to remove the app from the store and then add it again.

This can become pretty tedious which is why I did some digging around in the shopify_app code and created the following code sample to synchronize the required webhooks with the Shopify API. Simply hit this controller or call the containing code somewhere in the codebase.

If there’s a better solution to this problem please let me know.

Lastly, to keep track of your sanity the httplog gem is useful to track the http calls that shopify_app, shopify_api and any other gem makes.

Wrapping Up

The developer experience on the Shopify API and app store is quite pleasing. It has been around long enough to build up a flourishing community of people asking questions and sharing code. I believe the issues outlined above can be easily solved and will make Shopify a more pleasing platform.

The Software Engineering Daily Podcast is Highly Addictive

Over the past several months the Software Engineering Daily podcast has entered my regular listening list. I can’t remember where I discovered it, but I was amazed at the frequency at which new episodes were released and the breadth of topics. Since episodes come out every weekday there’s always more than enough content to listen to. I’ve updated My Top Tech, Software and Comedy Podcast List to include Software Engineering Daily. Here are a few episodes that have stood out:

Scheduling with Adrian Cockroft was quite timely as part of my final paper for my undergraduate degree focused on the breadth of topics in scheduling. Adrian discussed many of the principles of scheduling and related them to how they were applied at Netflix and earlier companies. Scheduling is really a necessity for software developers to know as scheduling occurs in all layers of the software and hardware stack.

Developer Roles with Dave Curry and Fred George was very entertaining and informative as it presented the idea of “Developer Anarchy”, a different structure to running, (or not running), development teams. Instead of hiring Project Managers, Quality Assurance, or DBAs to fill a specific niche of a development team, you mainly hire programmers and leave them to perform all of those tasks according to what they deem is necessary.

Infrastructure with Datanauts’ Chris Wahl and Ethan Banks entertained as much as it informed. This episode had a more casual setting as the hosts told stories and brought years of experience to the current and future direction of infrastructure in all layers of the stack. Comparing the current success of Kubernetes to the not-so-promising OpenStack was quite informative as it showed that multiple supporting organizations drove the OpenStack project to have different priorities and visions, whereas Google, being the single organization to drive Kubernetes, is shown to have one single, unified vision.


EDIT 2017-02-26 – Add Datanauts episode

Hook, Line and Sinker: How My Early Interests Got Me to Where I Am Today

Having an exciting software job at a newly-acquired company is opening up so many possibilities, and making possible the projects that I want to accomplish to make things better. Whether its big projects like containerizing our multiple apps for scalability, implementing Continuous Delivery to ship our software faster, or smaller projects like versioning our dependencies for traceability and ease of deployment, or updating to the latest Java version for performance improvements and to use all the new libraries; it’s nonstop fun that upgrades my problem solving skills, improves the lives of our team and customers, and gives me a track record of making positive change.

After finishing school I can focus more on teaching myself new skills and technologies that I can use and apply during my professional career. Currently I listen to DevOps and software podcasts when I’m traveling to places and I read a few articles about Docker and other technologies when I have free time. My next logical step is to start applying the knowledge I’ve gained both at work and as side projects.

At least it’s not all bad in the academia life: this fourth year Computer Security class that I’m taking is immensely fun. I’m glad that I have a captivating class this semester. I can give credit to the Security Now podcast for which I’ve listened to around 500 of its 543 episodes as of this writing (read: 10 years of listening!), for giving me the practical knowledge of current security practices and news, diving deep into the details where necessary.

Dr Anil Somayaji, the professor of the Computer Security course, is an excellent lecturer and a hacker at his roots. His interactive teaching style makes the possibly dry subject of security interesting (if you think of security as dry. Who would?), and his course work is very useful in that it promotes self teaching and helping out others. Each week every student must submit a hacking journal. It consists of the student writing about their adventures, frustrations, successes and failures of hacking on security related things – whether that involves using Metasploit to break into a computer, putting a backdoor into OpenSSH, figuring out how to configure a firewall, etc. The list goes on and on. An online chatroom is used to share resources and chat with other members of the class to figure out hacking problems and interact with the professor. (Other classes should definitely start using this)

I’m glad to have had the drive to explore and learn when I was young. Throughout my childhood I would spend my time hacking gaming consoles, jailbreaking iPods, experimenting with Linux, and most of all having a website! Not this website, there was a website before jonsimpson.ca. It was jonniesweb.com. I prominently remember creating logos for my website in MS Paint, printing them out and putting it up beside the Canadian flag that was posted in my fifth grade class. I would use MS Front Page 97 to add jokes, pictures, cheat codes, YouTube and Google Video links, games, and referrals to other friend’s Piczo sites. I remember going through a few designs: spaced themed, blue themed, red themed… I even got interested in PHP and used a sweet template. Each iteration improving with content and coding skills.

Then middle school and high school caught up with me and I stopped updating the website. Sooner or later my dad stopped supporting my hobby, eventually letting the web hosting expire.

Fast forward a few years and what was once a childhood interest has turned into an education and career choice. Building a website sparked the fire, pursuing a degree gave me the drive, and doing co-op (soon to be full-time) at work has shown me the many different problems to be solved.

My plan is to work my ass off in all of my classes, finish up my degree and follow my passions, utilizing my knowledge and expressing my solutions at both my job and in my blog. Ultimately trying to build a successful and happy career.

At the moment I’m just glad that I don’t have a crappy professor.

Push-button Deployment of a Docker Compose Project

I was recently working on figuring out how to automate the deployment of a simple docker compose project. This is a non mission-critical project that consisted of a redis container and a Docker image of Hubot that we’ve built. Here’s the gist of the docker-compose.yml file:

Whenever a new version of the zdirect/zbot image is updated and published to the registry a deploy script can be run. For example, the script used to automatically deploy a new version of a Docker Compose project is shown here:

Yup, that’s all. Its that simple. Behind the curtains, this command pulls the latest version of the image. Since the docker-compose.yml file doesn’t specify a tag it defaults to latest. The old container is then removed and a new one started. Any data specified in volumes are safe since its mounted on the host and not inside the container. Obviously a more complicated project would have a more involved deployment, but simpler is better!

Integrating this deployment script into Rundeck, Jenkins or your build tool of choice is a piece of cake and isn’t covered here, but might be in a future post. This automation allows you to bridge the gap between building your code and running it on your servers, aka the last-mile problem of continuous delivery.

Acquisition, Docker and Rundeck

travelclick-logoZDirect, the wonderful company I work for has been acquired by TravelClick, a billion dollar hospitality solutions company.

First of all: Woohoo! I can’t be more excited to be around at this time to jump-start my career.

One of the changes to occur as soon as possible is the consolidation of our datacentre into TravelClick’s. One of our devs recently found out about Docker and got interested about its power (Hallelujah! Finally it’s not just me who’s interested). Later I bring up Rundeck, a solution to organizing our ad-hoc and scheduled jobs that will assist in the move to Docker.

Docker

docker-largeHis plan is to Dockerize everything we have running in the datacentre to make it easier for our applications to be run/deployed/tested/you-name-it. The bosses are fine with that and are backing him up. I’m excited since this is a fun project right up my alley.

Since I’m working my ass off trying to finish my degree, I’m only in one day of the week to wash bottles and offer some Docker expertise. Last Friday I had a good chat with the dev working on the Docker stuff. We chatted about Kubernetes, Swarm, load balancing, storage volumes, registries, cron and the state of our datacentre. It was quite productive since we bounced ideas off of each other. He’s always busy, juggling a hundred things at once so I offered to give him a hand setting up a Docker registry.

By the end of the day I had a secure Docker registry running on one of our servers with Jenkins building an example project (ZBot, our Hubot based chatroom robot), and pushing the image to the registry after it is built. An almost complete continuous delivery pipeline. What would make this better is a way to easily deploy the newly created Docker image to a host.

Rundeck

rundeck-logoRundeck is a job scheduler and runbook automation tool. Aka it makes it easy to define and run tasks on any of the servers in your datacentre from a nice web UI.

Currently, we have a lot of cron jobs across many servers scheduled to run periodically for integration, backup and maintenance. We also ssh into servers to run various commands for support and operations purposes.

Here’s the next piece to our puzzle. Rundeck can fit into many of our use-cases. A few of them are as follows:

  • Deployment automation (bridge the gap between Jenkins and the servers)
  • Run and monitor scheduled jobs
  • Logging and accountability for ad-hoc commands
  • Integrate with our chatroom, for all the ChatOps
  • Automate more of production

As we move towards Dockerizing all of our apps, we have to deal with the issue of what we’re going to do with the cron jobs and scheduled tasks that we run. Since we’re ultimately going to move datacentres it makes the most sense to take the cron jobs and turn them into scheduled jobs in Rundeck. That way we can easily manage them from a centralized view. Need to run this scheduled job on another machine? No problem, change where the job runs. Need to rerun the scheduled job again? Just click a button and the scheduled job runs again.

The developers wear many hats. Dev and ops are two of them. Because we’re jumping from mindset to mindset it makes sense to save time by automating the tedious while trying not to get in the way of others. Rundeck provides the automation and visibility to achieve this speed.

With the movement of putting all our apps and services into Docker containers, Rundeck will allow us to manage that change while being able to move fast.


If you’re interested in joining us on this action packed journey, we’re hiring.

Twenty-one!

“Why am I not in the US of A right now?” Good question. That’s what my dad asked me when he congratulated me on my 21st birthday this past October 1st.

Not being in the US this Thursday evening for academic reasons, I spent Wednesday night with my roommates grabbing a few exotic beers at our local pub. The past week has been heavy with assignments and a midterm already, which is why this post has been delayed – but enough with the excuses!

Some of the achievements or big changes that have occurred since my last birthday have been the following:

  • Visited and vacationed in the USA for the first time
  • Moved into a new (and nicer) house with my roommates
  • Became a happier person through mindfulness
  • Expanded my beer and wine taste

This past year I’ve also been working on a bunch of small projects, some of them I’ve been actively working on, others I haven’t. I’ve pressed 3,139,757 keys and 769,532 mouse clicks, made 310 contributions to repos on my GitHub, read 5 books, and am still reading 6 books.

I’m more than half-way through my Computer Science program at Carleton. 3.5 years in and I’ve got less than a year and a half to go. It gets more exciting as the topics I’m studying get more advanced and interesting. I’m looking forward to finishing and starting my full-time career.

I must have read a countless number of articles; pretty much every day covering topics from Docker, continuous delivery, microservices, distributed systems, coding, and programming languages. I’m amazed at how much I’ve read and how I apply it to my work and experiment.

This year I really want to make drastic improvements to the operations side of my job at ZDirect, learn Go, Ruby and Coffeescript (in that order), become more mindful, and get into shape (because keyboard arms aren’t sexy).

Introducing ChatOps to my Workplace: Hubot

Hubot? You may not have heard of it, but its pretty much the workhorse of ChatOps. Hubot is a scriptable chatroom robot. It can integrate with many chat services and comes with a huge community of plugins and extensions.

In the previous post I talk about ChatOps, Slack, and how I plan on introducing it to my workplace.

Hubot is an IRC/campfire bot designed to give some character to your team’s channel. It has various commands for inserting photos in your chat, fetching stuff, and, indeed, running pre-configured commands.

There exists many other chatbots, but Hubot is the most popular. You can get scripts for anything: showing images, interacting with Jenkins, ship it squirrel, pager-duty, and hubot-plusplus, where the points don’t matter.

All of the plugins are written in coffeescript and follow a simple input/output design using regular expressions. Persistence is included as well, using Redis as the datastore.

Writing Hubot scripts can automate tasks while presenting a simple interface to interact with. Writing custom scripts adds useful insights and actions into your business and software.

So far I’ve written a custom Docker image that contains everything needed to run Hubot, bundled with a bunch of scripts. Everything is kept in a Git repository on our SCM server.

In the future, I plan on making the Docker images more extensible by separating the configuration from the code, then publishing it to my GitHub.

I also want to use Docker Compose to define a Redis container as a dependency of the Hubot container. This primarily allows for the Hubot container to be destroyed and rebuilt while the data stays safe in its own container.

Introducing ChatOps to my Workplace: Intro

slack_colour_rgbWhat is a ChatOps?

Last week my roommate talked about how he was building custom integrations for his Company’s Slack service. He was adding commands that would easily allow their client service team to extend the trial periods of their customers and other useful features. He was also in the process of working on being able to deploy their app to amazon via a single command.

All of these spells one thing: Productivity.

Trying to keep up with drinking as much of the new enterprise technology kool-aid, an excellent talk (slides) by Jesse Newland that I keep going back to every once in a while shows the power and benefit behind ChatOps.

Dissecting this buzzword, the true benefit behind this technology and culture is giving:

Visibility – Seeing what other people are doing and show what you’re doing to others to provide visibility and accountability into the operations that are performed. Sure, someone can say that they’re fixing the unresponsive server, but it’s not clear how they’re doing it and if its fixed yet without asking them. With all the operations used to fix such an issue available in a common chat room, its straightforward for others to read what was done in real time.

Learnability – Sure, you can have documentation and training to bring people on board, but nothing compares to seeing things done every day, most importantly their first day. In no time, a new employee can get up to speed faster than having to read through a lot of boring documentation.

Pairing – Allowing two or more people to solve a problem together. Much like pair programming, it is the practice of having more than one brain working on a problem or even passively observing. Pairing allows for more scenarios to be explored and better reasoning.

Automation – Simplifying tasks to the extreme. No need to log into a server, find and run a script. Just have a command in the chat room that does it. The command acts as a facade, presenting a clean api into your business activities. Being able to automate to the level of not having to switch out of the chat room is an amazing improvement in productivity.

Sneaky Implementation

On my self assigned task to improve productivity and culture at my workplace, many other developers and I are pinned in a rut of completing support tasks and fixing bugs for our software. Between tasks I’ve set up Slack and have been integrating our services into it. My goal is to ultimately reveal it slowly to other developers, gaining momentum until critical mass is reached, where the majority of my colleagues praise Slack and the power of its usefulness until it becomes the de facto messaging and team communication tool.

So far I have integrated our build server, Jenkins, and the source code repository, Gitlab, into Slack. Both of those are only dumb endpoints which dump data into Slack, but its already super interesting to have the visibility of just these services together. Nagios checks, and maybe even Hubot, a chat room robot, will come next.

The stream of messages created when someone pushes code, the code being built, and then the test results reported back. Its taking Continuous Integration and reporting all of the activities to a central location!


See the next post following the theme of ChatOps, this time about Hubot, the chatroom robot.