Have you tried it from Singapore?

I’ve been a DevOps Engineer before anyone called it DevOps.  I’ve seen every variation on how to approach troubleshooting an issue.  Well, I thought I had,  until yesterday, when we were having trouble receiving data from Weibo,  The Chinese hybrid of Facebook and Twitter.

We checked the networking to the server on our end which is responsible for ingesting data from Weibo.  We had reached out to Weibo, who provided data showing they were functioning normally.  We attempted manual versions of emulating the connection process to get the data… and so on.

Finally someone asked the not so obvious question, Have you tried it form Singapore?

This highlighted the completely new world that cloud computing has opened.  And that even if you host your own infrastructure, that you need to have a toe in the pool that is the cloud.

Trouble shooting an IT problem is a process of identifying the variables that are in place, and isolating them one by one until you are able to pinpoint the one that is at or near the issue.  ”Have you tried it from Singapore?” is the new tool in the DevOps belt now that allows us to not stop at the point where we have to concede that the issue “must” be somewhere in the route between us and them.

It’s now possible to have, in an instant, a server up and available in Cloud Data Centers around the world.  This allowed us to test the issue much more local to China.  I’m sorry, but that is huge.   We were able to open our AWS console, bring up a sever in Singapore and remove all the unknown of transferring data around half the globe, and remove that variable.  This is something that not long ago was an inconceivable troubleshooting step.  You had to trust your remote partner, and that is no longer the case.

So DevOp, the next time you run into a network issue between you and a provider, remember… Have you tried it from Singapore?

Code something, anything, to jumpstart your creativity

I’ve recently been in a recurring rotating role of testing out new tools to potentially integrate into our infrastructure.

If not doing that I’ve been aiding our research team in troubleshooting a Hadoop testbed for a prototype Social Media mining add on to our product.

In short, for all intents and purposes my only recent memories are of config files.

I needed something to recharge myself, but what?

Luckily? Twitters outage this week fell into my lap.  My companies lifeblood is the Twitter Firehose feed.  It was down, and with all my monitoring in place, I didn’t realize right away.  This was a problem, and it was one I could write code to solve.

Our engineering team had just demoed some new JDK 8 niceties that looked a lot like features that I really liked in Ruby.

I probably could have found an existing plugin to monitor my data rate intake from Twitter.  I probably could have written my own easily in Bash.  The JDK 8 demo though resonated somewhere though, and I thought it was time I refresh my Ruby memory and write it with that.

With that, the magic was back, I was excited about this tiny monitor that needed to be in place and I was feeling refreshed.

The ultimate takeaway is that we all tend to get burnt out with projects sometimes and need a way to revitalize our drive.  For me it turned out to be a simple coding exercise in a language I’d neglected for a while resparked my creativity and got me back on track.

What’s your go to?

Startup, have you considered DevOps as an initial hire?

Having worked as a sysadmin, and now under the latest moniker of DevOps for quite a few startups I’ve run into the commonality that DevOps was an afterthought, albeit with a couple notable exceptions.

I get it, someone had an idea and ran with it, got it to a point that someone wanted to fund it, and it was time to start growing and bringing in customers.

To do this the company had to begin to scale up quickly, get an initial product out the door with a minimal viable build and reiterate on that process.  They had to grow the company and the product at the same time with little worry about scaling that growth long term. It had to work NOW.

Ultimately it did work and it was time to hire.  Bring in developers both UI and Backend, sales, support, etc… And to everyone’s delight success!

This is about the time people start looking for me, or someone like me.  The engineering team realizes that while what they’ve built for an infrastructure works, it might be becoming more of a time consuming task to maintain than to actually continue to enhance the company’s product.  It’s time for DevOps.

Upon joining the startup, I tend to see that Engineering has done a great job building a product, but not an infrastructure.  It runs, but no one can really tell how well.  It’s been built with a purpose, but not been designed to be managed.

This lack of management built into the design begins to be a limiting factor in the companies ability to be agile to market demands on them.  What one tends to fail to consider is that what a startup does in the beginning is rarely what it ends up doing ultimately.  What I mean here is that the initial idea that founded the company tends to change quickly over time.  The initial offering while finding customers tends to be “off” a bit and through feedback and iteration oftens veers far from the originating idea.  This means that the technologies employed to support this product often have to change quickly dramatically as well.

This is where DevOps accel and can be a vital resource in the architecting of the initial infrastructure designs.  These designs will be small in scale, but it’s DevOps that design with scale in mind.

I’ve worked in both scenarios, but all that had adopted DevOps early on have had undeniable success.