Labor Day has passed, the days are getting shorter, the mornings are getting cooler… that means it must be time for cyclocross! Yes, yes it is. As usual, I started off with the “official” season opener for the #NECX of Quad Cross. Some people may race cross earlier, but that’s pre-season racing if you ask me. Although with Labor Day being as late as it was, there were definitely plenty of options for people to start racing. But many people waited for this past Sunday to kick things off.
Although I don’t race for Quad anymore, I still volunteer and help out with the race every year. So I woke up early, got in the car and headed over to Maynard to help with registration. Things were surprisingly smooth as people started registering — it’s almost like after a number of years of doing this, it’s starting to be something that we know how to do. That said, racers — have your license handy! If you don’t have a paper copy, install the USAC app on your smartphone and then you can show that. Having to look up your license info slows things down for everyone.
Once registration was well underway, I hopped on my bike and went for a quick pre-ride of the course. Although it had rained overnight, it wasn’t particularly muddy — just not the usual dustbowl. The course was fast and fun. But I knew a lot of pedaling was going to be required for it. I got in a second lap and then went back to help with registration some more after the Women’s 3/4 fields kicked off.
As that wrapped up, I got in a third lap (wow, it’s like I almost warmed up for real!) and it started to mist. We headed to the line and it was a little bit less than organized getting people set up. And thus, I chose poorly in terms of starting position. Then we were off and I basically got boxed in more than I would have liked 🙁 But I kept moving forward and generally was riding okay. Some of my cornering was bananas but that’s to be expected in the early season and not being used to having brakes that can actually stop me. Anyway, I raced, it hurt and I finished. Ended up 26/63 which puts me right back where I was before I upgraded a few years ago. Given that I was being demotivated fighting for not-last in the 3s, I downgraded back to a 4 this year which definitely seems the right thing to have done — I felt like I was really racing again.
Picture courtesy of Patricia Tamagini-Dayhoff
But maybe the best part of the day was hanging out afterwards. We had a huge presence for the Keep It Tight team at the race and the tent set up and were just generally chatting and encouraging everyone else in other fields. I also got to see and catch up with a ton of folks who I pretty much only see at cross races… too many to name them all. A big fat reunion for the (perhaps dysfunctional) family that is the #necx.
And next weekend, I’ll be at it again. Another chance to test myself. A chance to do better. And onwards.
Back in mid-January, the weather in New England had been unseasonably nice and it was looking like we were going to have a mild winter. I had completed the Rapha Festive 500 at the end of the year and felt like it would be a good winter of riding although it was starting to get cold in January. Someone mentioned the Rasputitsa gravel race (probably Chip) and I thought it looked like it could be fun. There was one little blizzard as we neared the end of January (and the registration increase!) but things still seemed okay. So I signed up, thinking it would help keep me riding even through the cold. Little did I know that we were about to get hit with a record amount of snow basically keeping me off the bike for six weeks. So March rolls around, I’ve barely ridden and Rasputitsa is a month away. Game. On.
I stepped up my riding and by a week ago, I started to feel I’d at least be able to suffer through things. But everyone that I’d been talking with about driving up with was bailing and so I started thinking along the same lines. But on Friday afternoon, I was reminded by my friend Kate that “What would Jens do?”. And that settled it, I was going.
I drove up and spent the night in Lincoln, NH on Friday night to avoid having to do a 3 hour drive on Saturday morning before the race. I woke up Saturday morning, had some hotel breakfast and drove the last hour to Burke. As I stepped out of the car, I was hit by a blast of cold wind and snow flurries were starting to fall. And I realized that my vest and my jacket hadn’t made the trip with me, instead being cozy in my basement. Oops.
I finished getting dressed, spun down to pick up my number and then waited around for the start. It was cold but I tried to at least keep walking around, chatting with folks I knew and considering buying another layer from one of the vendors, although I decided against.
It’s overcast and chilly as we line up at the start
But then we lined up and, with what was in retrospect not my wisest choice of the day, I decided to line up with some friends of mine who were near the back. But then we started and I couldn’t just hang out at the back and enjoy a nice ride. Instead, I started picking my way forward through the crowd. My heart rate started to go up, though my Garmin wasn’t picking up the HR strap, just as the road did. The nice thing was that this also had the impact of warming me up and not feel cold. The roads started out smooth but quickly got to washed out dirt, potholes and peanut butter thick mud. But it was fun… I hadn’t spent time on roads like this before but it was good. I got into a rhythm where on the flats and climbs, I would push hard and then on some of the downhills, I would be a little sketched out and take it slower. So I’d pass people going up, they’d pass me going down. But I was making slow progress forward.
Until Cyberia. I was feeling strong. I was 29.3 miles in of 40. And I thought that I was going to end up with a pretty good time. After a section of dirt that was all up-hill, we took a turn to a snow covered hill. I was able to ride about 100 feet before hopping off and starting to walk the bike up hill. And that is when the pain began. My calves pulled and hurt. I couldn’t go that quickly. The ruts were hard to push the bike through. And it kept going. At the bottom of the hill, they had said 1.7 miles to the feed zone… I thought some of it I’d ride. But no, I walked it all. Slowly. Painfully. And bonking while I did it as I was needing to eat as I got there and I couldn’t walk, push my bike and eat at the same time. I made it to the top and thought that maybe I could ride down. But no, more painful walking. It was an hour of suffering. It wasn’t pretty. But I did it. But I was passed by oh so many people. It was three of the hardest miles I’ve ever had.
The slow and painful slog through the snow. Photo courtesy of @jarlathond
I reached the bottom where the road began again and I got back on my bike. They said we had 7.5 miles to go but I was delirious. I tried to eat and drink and get back into pedaling. I couldn’t find my rhythm. I was cold. But I kept going, because suffering is something I can do. So I managed to basically hold on to my position, although I certainly didn’t make up any ground. I took the turn for 1K to go, rode 200 meters and saw the icy, snowy chute down to the finish… I laughed and I carefully worked my way down it and then crossed the finish line. 4:12:54 on the clock… a little above the 4 hours I hoped for but the hour and 8 minutes that I spent on Cyberia didn’t help me.
Yep, ended up with some mud there.
I went back to the car, changed and took advantage of the plentiful and wonderful food on offer before getting back in the car and starting the three hour drive back home.
Mmm, all the food
So how was it? AWESOME. One of the most fun days I’ve had on the bike. Incredibly well-organized and run. Great food both on the course (Untappd maple syrup hand up, home made cookie handup, home made doughnuts at the top of Cyberia, Skratch Labs bottle feeds) and after. The people who didn’t come missed out on a great day on a great course put on by great people. I’m already thinking that I probably will have to do the Dirty 40 in September. As for next year? Well, with almost a week behind me, I’m thinking that I’ll probably tackle Rasputitsa again… although I might go for more walkable shoes than the winter boots I wore this year and try to be a bit smarter about Cyberia. But what a great start event for the season!
Fire. Chainsaws. Alf. Basically, all of Vermont’s finest on offer.
As most who know me know, I consider myself a cyclist. I ride my bike often, do distances that most consider questionable and even at times in pretty unsavory conditions
Eight years ago, this wasn’t the case. I was your typical pretty sedentary software engineer. But I got a bike and started riding a little. I thought that maybe I would get to where I would do a 50 mile ride. Or a metric century (that’s 62 miles/100 km for those not in bike circles). But I was going up and down the bike path so was at 15-20 miles. 25 was long for me.
And then I decided one Saturday morning in May to join the group ride from the bike shop down the street, Quad Cycles. I showed up and it was a little intimidating. There were probably 30-40 people and they all looked like they knew what they were doing. As we hit the time for the ride to start, Bobby yells out asking for anyone who is new. I acknowledge and he describes the ride. I figure I’ll ride to the end of the bike path and then ride home. But we got to the end of the path and Bobby encouraged me to continue and said he would ride with me. I think I rode 30 or so miles that day, all of it with Bobby right with me.
From there, I began riding more. Bobby encouraged me to do the Red Ribbon Ride. He always was encouraging people to do a charity ride to give back for all that we had. But it was a two day ride totaling 175 miles. And it was two months away. A little intimidating for someone who hadn’t been riding at all six months earlier. But he encouraged me and I did it and it was incredible.
The rest, as they say, is history. But I saw the same thing play out many many times over the following years. Someone new to riding encouraged to push themselves, to go further than they thought they could, to give back. And always to be nice to everybody while doing so.
RIP Bobby… you will be missed even more than you could know. I am glad to have called you my friend. I only hope that I can be as encouraging and helpful to others as you once were to me. And I’ll never forget to ride with love in my heart and a smile on my face.
So I got into a little bit of a twitter argument last night about distro packaging. Which actually wasn’t where I was trying to go (this time 😉 ) One of the problems with twitter is that 140 characters can be hard sometimes. So let’s see if more characters help.
There is a big push afoot from a lot of people towards omnibus packaging. It seems especially prevalent in the world of things written in Ruby I suspect because most even “current” Linux distros are or were shipping some Ruby 1.8.x build up until very recently. And people want to take advantage of the newer language features.
I have a lot I could write on this topic. But I’m not going to today.
The main point I was trying to make is that in going down this path, people are putting together increasingly large build chains that have complex dependencies and take a long time. And so they’re starting to do things like caching of sources, looking at short-circuited builds and a whole lot of other related things. Which, incidentally, are all things that all of the Linux distributions ran into, fought with and figured out (admittedly, each in their own way) years ago. So rather than just rediscover these things and do it yet another way, there’s a ton of knowledge that could be gained from people that have built distribution build systems to make things better for the omnibus world.
So reach out to people that have worked on things like the Debian build system, the OpenSuSE build system, plague, koji and a slew of others. There are some tricks and a lot of tools which are just as valuable as when they started being used. Things like caching build roots and how to fingerprint them for changes, what ccache can and can’t be good for, how to store things reasonably in a lookaside to not depend on upstream repositories being down while you build, incremental builds vs not, ccache, …
Just because you think that building system RPM or DEB packages doesn’t meet the needs of your users doesn’t mean that you have to throw away all of the work and experience that has gone into the toolchains and build systems present there.
Working out of our first “office” in a classroom at Northeastern
Going in, I thought I had some idea of what I was doing and what I was in for. I mean, I’d been at later stage startups and heard the stories. I’d read the Eric Ries Lean Startup book. I had read all of the “blogs you’re supposed to read” and had them in my RSS reader so that I could soak up collective wisdom regularly.
In some ways, I was right… in others, I was wrong. Here’s a non-exhaustive list of things that are sticking with me after a year. That said, if you’re reading this because you’re thinking about or planning on joining a startup at an early stage, there’s a good chance you’ll have different ones that are important for you 🙂
Hiring is the single most important thing you can do. Good hires amplify everyone around them and make the team smarter and more effective. It’s pithy and everyone says it. But they say it because it’s true.
Hiring takes way more time than you think it will. Your network (and your network’s network) is the best way to source candidates. You’ll post your positions on various job boards and you’ll very very very occasionally get lucky with it. Recruiters will come from everywhere promising great results but will take up even more time since their candidate flow is way higher. Another challenge with recruiters is that when you are before the launch of your product, it can be difficult to convey what you’re doing as well as exactly the type of person you are looking for to join the team.
Fancy new technology is sexy and fun to work with. It also has a lot of problems you don’t have experience with. If you can minimize the number of these you have to deal with, you’ll be able to spend more time focusing on building the right product for the right problem
It’s hard to spend too much time talking with your target customer. If you think that what you have (be it a mockup or a prototype or something else) is “ready” to show to them, you’ve probably waited too long and could have gotten feedback already and learned something
Fail fast. Some things you try won’t work. Don’t continue to fixate on them and sink more time on them if they’re not coming together.
If you’re building something that’s a B2B product, think of your beta as a chance to test selling in addition to the product. Sure, you’re not going to have them pay today but you do want to know that you’re building something that people will pay for when you’re ready to switch to that.
Think iteratively. Once you start to hone in on a degree of product market fit, you’re going to discover that some of the things you built for prototyping/testing purposes don’t work. Don’t be afraid to replace them. But do so in a way that lets you regularly checkpoint the replacement to test that you are making things better. Grand rewrites are rarely as grand as you think and always take longer
If you’re going to spend a lot of time on something product-side, focus more on getting the interfaces right than the implementation. So, for example, if you decide that something is going to have clear boundaries passing messages over a queue, then you can switch between RabbitMQ, SQS and others with minimal effort as you learn the constraints that actually matter for your implementation.
Try to find the one piece of your product that immediately pops in a demo for most of your target users. This is one of those things where you’ll know what it is when you see it. And then use it as a hook to start drawing people in.
Interactions with the customer don’t end when they sign up for your product. Continue to nurture them and do regular feedback calls with them as you iterate on the product. This will help to make them into advocates for your service and they’re already bought into your vision making it
Bugs happen. Fixing them and providing awesome customer service is a great way to foster great customer relationships
It’s been a wild and crazy year but I have had a blast. I’ve done a bit of everything and learned things I didn’t even know there were to learn. Launching the beta of our cloud monitoring product a few months ago was an awesome experience and watching as we’ve started ramping up our sales engine to engage with customers and try it out has been phenomenal. And I’m really looking forward to the next step of launching the paid product and starting to track everything that goes along with that.
As a kid growing up, one of the things I enjoyed doing was riding my bike. In the woods, on the road, anywhere. I even did some group rides at the time although I was on a mountain bike for them. And I remember hearing of some of the bigger rides in western North Carolina at that point… Bridge to Bridge and the Assault on Mt Mitchell, notably. So when I really started to get back into riding a while ago, I thought about at some point going and doing some of those rides. Since I’m not really doing any road racing this year due to being a bit too busy with work, I decided to try to tackle some of these long and hard rides that I’ve wanted to do for a few years to keep me motivated and riding hard.
First up is the Assault on Mt Mitchell. For a bit of background, Mt Mitchell is the highest point east of the Mississippi ending up over 6000 ft. And about an hour from where I grew up. So starts out sounding a little intimidating. The ride itself actually starts in Spartanburg, SC and you then spend the first 75 miles riding along rolling hills until you reach Marion, NC. From Marion, you go up 5000 ft over the remaining 25 miles. Okay, lots of climbing when you’re already tired. This sounds awesome. I’m in.
The route map
Elevation gain
Preparation and Pre-Ride
I signed up for the ride back when registration opened in March. From that point, I received a steady stream of emails detailing the training rides that they offered and suggested including things that covered a lot of the route. Living about 900 miles away, those weren’t an option. So I basically did a pretty typical set of spring riding for me; stretched out some rides a little more to get more rides in the 80+ mile range instead of 50-60s but no real hill work, etc.
Given that my parents still live in NC, we decided to make a family trip down to see them. So I shipped my bike via FedEx to my dad’s office (unnerving!) and we flew down. We arrived on Saturday, I put my bike together and did a little loop on Sunday to stretch the legs and shake down the bike after reassembling it. All good. I packed everything I needed, the bike survived being shipped, and my legs even felt decent with the lack of riding I had done the week before.
Of course, up until this point, the weather forecast for the ride on Monday was looking less than great. Showers and thunderstorms through the day. Because riding 100 miles in the rain is fun. Ugh. Luckily, after riding in some sloppy drizzle on Sunday, the forecast for Monday magically got better. I’ll take it!
The Start
Given it’s about an hour and a half from my parents house to Spartanburg and roll out is at 6:30, we stayed at the Marriott around the corner from the start on Sunday night. So Monday morning, I woke up super early and headed to the start with plenty of time. Breakfast was my first rice cake of the day (the classic egg + bacon recipe for this batch) although in hindsight I should have gone for something more. As I had picked up my packet and number the night before, I didn’t really have to do anything other than get to the start which was nice. As I did so, the size of the event really started to become clear, around 1000 cyclists all told.
This slideshow requires JavaScript.
I made my way towards the front of where people were lining up. We had the entire street (four lanes) and I wasn’t going to get caught up in the back. I had modest goals for the event — stick with the front as long as I felt comfortable but mostly in it to finish. Time wasn’t at the front of my mind as I was thinking of it as a ride, not a race really. As the countdown got to zero, we took off. And the front went fast… we were going a sprightly 26-27 mph for the first mile or two. This was made possible largely due to the awesome support the event provided — a police detail at the front, officers at every intersection to let us through. And this largely continued for the entire route.
To Marion
As we got going, the pace settled somewhat and I just sat in to draft as much as I could. <rant>There was a ton of just random braking, though. The smell of burning carbon wheels filled the air more often than not. I think a lot of the braking was due to people crossing the yellow line, seeing oncoming traffic and then trying to rejoin the peloton. It was nerve racking and quite frankly unnecessary. And I think it was also the cause of the one person that I heard go down at one point behind me. If event organizers have made it so that we have full use of a lane rather than just two abreast, people should respect that.</rant> As a result of the pace and the braking, the lead group continued to shed people. Given that I wasn’t really trying to be in the front, I ended up on the wrong end of those sheds a few times and had to jump hard to close the gap and rejoin the lead group.
Unfortunately, around mile 60, I got gapped and couldn’t close it. 22 mph for that stretch and I was ready to drop. Was bummed not to hold out until Marion at that point but I also knew I needed to save some energy for the second part of the ride. So I ended up in a little group of about 8 people and we did a solid bit of effort working together. But when we got to Marion, my bottles were empty so I stopped to refill and lost my group. And I then just missed the second big group moving through and couldn’t quite catch them meaning that for the remainder of the ride, I was going to be doing it basically solo.
Marion to the Parkway
As I headed out of Marion after the stop, I had a difficult time finding a rhythm riding alone for the first time of the day. I think I definitely would have been better in a group in this section as it wasn’t that intense but I definitely wasn’t at my best. I kept going and didn’t stop at the next rest stop. And after that is when the climbing really felt like it began. That section of Rt 80 was grueling. Luckily, I ran into others who said it was the hardest four miles of the ride. So I believed them and just tried to settle in and keep my legs moving. But looking at the data from the ride, you can see just how slow it was. I just suffered through it and accepted that the rest of the day was going to be hard. And I just kept watching the mileage creep along knowing that the next rest stop wasn’t that far ahead. Switchbacks, steady climbing… you really can’t find anything like it in Massachusetts. On the plus side, the scenery was gorgeous or at least seemed so to my oxygen starved brain.
Finally, I reached the rest stop at the 87 mile point where you turn onto the Blue Ridge Parkway. I stopped and drank some Coke, ate a cookie and refilled my bottle. Although I had done the first 75 miles in under 3.5 hours, the next 12 had taken me a little over an hour. Of course, this section was about 2000 ft of vertical gain, mostly in the second half.
The Blue Ridge Parkway
The next section of the route was on the Blue Ridge Parkway. I’d say that the BRP is one of the classic roads for biking with lots of group rides as well as training camps and the like taking part on various chunks of it. And after riding 11.5 miles of it, I see why. The road conditions are great, there isn’t a ton of traffic and it’s a steady, hard effort. Although I expected a little bit of a respite based on what I was told on my way up Rt 80, it really didn’t come. But riding along the parkway including the scenic overlooks and the tunnels made it worth it. On a few occasions, I wanted to stop and take photos just given the sheer beauty of the scenery… but I realized if I did, I would be unable to get started again and so I just kept pedaling. This became especially true around mile 90 when I cracked kind of hard. Luckily, that was also when there was the remaining downhill segment of the day. I was soft pedaling down but was getting cold given the cloud cover and the elevation and so ended up picking things back up a little. Honestly, other than that I remember little of this section. I know I was being passed by people and also that I was passing people back but it didn’t leave as big of an impression. It was just more of a steady slog and a mental struggle to reach Mt Mitchell Parkway
Final Stretch
After 11.5 miles on the BRP, you turn onto Mt Mitchell Parkway for the final five miles. I was starting to feel like it was in the bag and started to relax a little bit at this point, feeling my energy level pick up a little. The section to the last rest stop was still kind of grueling though. Not as bad as Rt 80 and you know that it’s shorter so that helps a lot. I passed a lot of people cramping on this section, though. I was pretty happy with having stuffed a bunch of single serving Skratch Labs secret drink mixes into my pocket and using them rather than Gatorade, especially as I saw that. A few people who had seemed quite strong earlier on were definitely suffering here. But I felt like I was getting stronger for the first half here.
The final rest stop was at the entrance to the State Park and I quickly stopped for a little more Coke here as I felt the sugar would help on the final little ascent. But it was a super quick little stop and then I was on my way. The grade here let up a decent amount and so I was able to stand and really kick it a bit. As I passed the parking lot with the yellow Penske trucks (used for transporting bikes back down the mountain), I knew I was almost there and so of course that was the one point where I got a twinge of crampiness. I pushed through it, though and finished strong.
I ended up with an official chip time of 6:34:33 and a moving time from my Garmin of about 6:20. Since I had hoped to end up between 6 and 6.5 hours, that was right on target. And my time put me at 131st of the 719 people who completed the race and 10th for my age group. Not shabby at all for my first time doing it.
Post-Ride
After crossing the finish line, my bike was immediately whisked away from me and I stumbled up to where our dry bags were. I changed into something that didn’t have a chamois (hooray) and grabbed some of the tomato soup that was there as well as a bag of Doritos (mmm, salt). I then made my way to the bus to start heading back to Marion. The ride back to Marion was pretty quiet and I caught up on Twitter and chatted with the guy sitting next to me. He had done the ride a few times before and finished about 10 minutes behind me.
This slideshow requires JavaScript.
When we got to Marion, I wandered over and made myself a plate of food and kind of forced myself to eat it even though I was in the “not even hungry any more” state as I waited for my bike to make it down the mountain. Kara, Madeline and my mom met me there and then I got my bike and it was on our way back to my parents’ house for the rest of my trip.
Closing Thoughts
So after doing all of it, I have a few thoughts about the ride. First of all, it is very well run. Police escort out of Spartanburg, every turn well attended (with traffic stopped!), good rest stops (at least, the ones I stopped at). The route was awesome — great roads, low traffic, lots of good hard climbing but also some stuff that in a group can just fly by. Getting people + bikes down from the top of the mountain to Marion also went more smoothly than I expected.
This slideshow requires JavaScript.
Really the only bad I can point to is the behavior of some of the other riders. I saw somewhat rampant littering (gu wrappers, bottles, everything) and even with full use of the lane, people were frequently in the left lane when we were in the large group. Kind of disappointing and reflects poorly on cyclists in general.
Will I do it again? Probably at some point. The logistics make it difficult to commit to doing regularly but I’d definitely like to make another pass at it and see if I can get my time under six hours. To do so would require at least some concerted and different training that I’m not 100% sure how I’d get but I do think it’s doable.
Final ride data is up at Strava as usual. 242 suffer score, Training Peaks gave it a TSS of 471 (both based on heart rate, not power at this point). All in all, not a bad day on the bike.
I’m currently on the train on my way back from DevOpsDays in Brooklyn. The conference was great — lots of smart people facing a lot of similar problems and trying to see what we could learn from each other. The scale was small, with only like 100-ish people present and not a ton of huge, in your face sponsorship. And the venue was a college campus. And so I kept making these comparisons in my head to LUG meetings, installfests and small scale Linux conferences.
Obviously the subject matter was a bit different — talking about and thinking about running large scale production infrastructures is a little bit different than the next cool Linux distribution. This tended, I think, to more discussion around patterns and best practices than about the specifics of “you should do X to get Y to work”. So a higher level and more abstract discussion.
The composition of the audience and attendees was a pretty similar make-up. Linux events always had a strong majority of the attendees who self-identified as sysadmins and then there tended to be a smaller number of developers. And many of the latter group had ended up in that camp due to necessity. The breakdown for DevOpsDays felt pretty similar with an interesting twist where there were speakers who said they were (paraphrasing) “developers first and fell into operations because they needed to”.
One thing that felt more evolutionary than anything else was that the side channel discussion for the event took place on Twitter rather than on IRC. I have (fond) memories of many conferences where attendees sat in an IRC channel and then basically continued to interact on IRC long after the conference had ended. In fact, I made many friends in this fashion. Similarly there was an ongoing discussion on Twitter using the #devopsdays hash tag and I have followed (and am being followed by) a number of the other attendees and hope to keep in touch and call them friends in the future.
And maybe the thing that struck me the most strongly was where people were “from”. Not in the sense of where they lived but rather where they worked. The attendees were almost all from startups. We were in Brooklyn and not the heart of downtown Manhattan, but NYC is probably home to more financial services companies than anywhere else in the world. And all of those companies have *many* people working in software dev and operations-y roles. But they weren’t there.
So it feels like “the DevOps movement” is going through a similar growth and evangelism pattern as open source and Linux did years ago. Maybe that’s why it feels so comfortable to me.
Although I haven’t really talked about it here, I joined a new startup a couple of months ago called Stackdriver where we’re working on building a hosted solution to make infrastructure monitoring and management suck less for users of the public cloud. After a having to duct tape the various pieces together a couple of times now, it’s super clear that the need is there so it’s exciting to be working on solving it. More on the side of being at a very early startup to come in the future.
Today I had planned to do some work around some of our provisioning and deployment code and Amazon had another EBS outage making the AWS API pretty unavailable for much of the afternoon. So after doing some other things, I took a look at what fails along with EBS to help us remember what fails along with EBS and thought it was interesting enough to share.
I finally got around to trying the Chef omnibus installer and it’s a step up from what I was doing previously but still not great. Grabbing a shell script with curl or wget and piping it to your shell is an anti-pattern which I wish had never taken off. Luckily, in this case, the shell scripts is just pulling down an rpm and installing it. One step nicer would be if there were just a repo that you could use via yum and have things a yum install chef-full away. And as I thought that this afternoon, I remembered the baseurl support in createrepo. Thus, without further ado, I’ve thrown together a quick set of repos that just point to the files in the opscode s3 bucket and minimizes the amount of storage I have to do 😉 If you want to use them, just drop a file into /etc/yum.repos.d named something obvious like chef.repo
[chef]
name=Chef Omnibus Packages
baseurl=http://katzj.fedorapeople.org/chef-omnibus/el$releasever/$basearch
enabled=1
gpgcheck=0
#gpgkey=
I’ve only tested the EL6 x86_64 package but I went ahead and created the repos for EL5 and EL6, both i686 and x64_64. Yes, the packages aren’t signed right now. Hopefully that’s something that can be remedied relatively easily. And even better would be if Opscode would just integrate the simple call to createrepo into their build process for the omnibus installer.
I have a decent amount of experience at this point with puppet both from experience using it to manage the infrastructure running Fedora as well as setting it up at a pretty large scale at HubSpot. But in a new gig, I decided it was worth rounding myself out a bit and giving chef a try. Not out of any deep seated dislike of puppet but there are a few pieces that I’ve continued to run up against which are a little grating and so I figured it was worth broadening my horizons. The nice thing is that both are fairly successful open source communities and realistically, as long as you’re using a system, you probably can’t go that wrong or switch in the future.
Side-note: I’ve also been playing with Michael Dehaan’s new project, ansible which is also interesting. But I don’t think it’s mature enough to use for a production environment yet and I also was mostly interested in it as a better remote execution layer as opposed to another full fledged config management tool. But yeah. It’s there. It’s interesting. I’ll probably write more about it later.
With a little bit of chef time under my belt, I have to say that I’m not struck by drastic differences. The terminologies are different, the DSL used on the config side is a bit different but they act pretty similarly and you can get either of them to do what you want. That said, there are a few things (good and bad) that I’ve noticed about chef and figured I’d share for others who are looking at deciding for themselves. Note that a few of the things in the dislikes section may well just be me missing something and being a n00b… suggestions welcome!
Things I’ve Liked
Hosted Chef is a very very nice option to have. Props to the Opscode team for building an infrastructure to run the server side for youand especially for making the barrier to entry nearly zero by letting you manage up to five hosts for free. Given some of my headaches around running a puppetmaster previously, I’m glad not to be having to pull together everything to run a chef server
Knife is actually pretty cool. I was skeptical before using it but it does a pretty nice job of encapsulating a lot of common tasks for you
Knife gets really cool with the addition of the ec2 plugin. Launch servers, register them with hosted chef and have them ready to go. I’ve built all of the surrounding bits and as the environment I’m dealing with grows, I think I’ll grow out of being able to use knife ec2 effectively, but it’s great for an easy starting point
Chef solo seems to work okay and have a few niceties over a master-less puppet setup but I didn’t spend much time with masterless puppet, so it’s probably just that I didn’t find the related nice pieces
Things I’ve Disliked / Been Annoyed By
The package support in the Fedora/CentOS/RHEL universe is pretty poor. I realize that all the cool kids use Ubuntu these days but tons of server infrastructures are not. Todd does a great job with the puppet (+ ecosystem) packages for Fedora and EPEL. Would love to see someone do similar for all of the Chef stuff
A lot of the cookbooks that are out there and published are Ubuntu specific. Even the ones which strive to work across distros often end up coercing the Fedora universe to look more like Debian. Which isn’t necessarily a path I want to go down
Probably just a side effect of this but a lot of cookbooks using things which aren’t the standard init system (eg, depending on runit)
knife-ec2 makes you think you can get away with using it but I keep tripping across things it doesn’t support and making me consider abandoning it
Trying out cookbooks from others drives me crazy. I’m pretty sure I’m missing the good workflow here but polluting my checkout by adding vendor branches and auto-committing things. There’s gotta be something I’m missing here
So am I now a rabid chef fan? Nope. But it’s a nice system with some definite advantages for certain use cases. I suspect I’ll find more of them as I use it more.
