Unfortunately, I was pretty disappointed with my options.

• Do some building by hand on an actual instance, then do the bundling and upload off of the running instance.
• Some of the ThinCrust stuff initially looked promising, but it seems like it’s largely unmaintained these days and the ec2 conversion bits didn’t really work at this point. I was able to get my initial images this way, but mostly by having a wrapper shell script of doom that made me sad.
• There’s always the rPath tools, but I wanted to stick to something more native and fully open source
• The new kid on the block is apparently BoxGrinder but I found it to be a lot over-complicated and not that robust. I’m sorry, but generating your own format that you then transform into a kickstart config and even run through appliance-creator via exec from your ruby tool just felt wrong. No offense, but just felt like a lot more than I wanted to deal with

So, I sat down and spent an evening hacking and have the beginnings of a working ami-creator.
It’s pretty straight-forward and uses all of the python-imgcreate stuff that’s used to build Fedora live images. Your input is a kickstart config and out the other side pops an image that you can bundle and upload to EC2.

Thus far, I’ve tested it to build CentOS 5 and Fedora 14 images. I’m sure there are some bugs but at this point, it’s worth getting it out for more people to play with. Hopefully it’s something that’s a lot simpler and more accessible for people to build images and I think it will also fit in a lot better with having Fedora release engineering building the EC2 images in Fedora 15 if they want.

One of the big outstanding pieces that I still want to add is the necessary bits to be able to (optionally) go ahead and upload and register as an AMI with your EC2 account.  But release early, release often.

Comments, etc appreciated in all the normal ways.

Minor update: switched the repo to live on github instead

• If you’re using cloud to refer to EC2, use EC2 instead. It’s concrete and it means very real things about your deployment and scaling models as well as how you’re managing your infrastructure.
• If you’re using cloud to refer to some service which runs over the Internet, either refer to the service or just say the Internet. You don’t store your mail “in the cloud”, you host it with Google apps. You don’t backup “to the cloud”, you have your backups stored over the Internet with Mozy or Carbonite.
• If you’re using cloud to refer to the idea of some hosted application platform, just say the platform. You don’t run your python app “in the cloud”, you run it on AppEngine (or something else).
• If you’re using cloud to mean that you are using virtualization and have some management stack on top of it, then please just say you’re running in a virtualized environment.
• If you’re using cloud to refer to having your server infrastructure hosted in a virtualized environment by someone else, again, just say you’re running in a virtualized environment.
• If you’re using cloud to refer to a “visible mass of little drops of water or frozen crystals suspended in the atmosphere”, then congratulations, you can continue to use the word cloud. And thanks to Wikipedia for the definition

Following this simple idea will let you avoid the otherwise impossible to avoid discussion of the semantics of the word “cloud” and what you happen to mean about it and how you might be wrong and … This then means you’ll be that much closer to achieving whatever goal you hoped to achieve as you’ll spend less time talking and more time doing. And as an added benefit, you’ll avoid getting grumpy emails from me about the fact that you’ve used such a terribly over-used and under-meaninged term.

Unfortunately, they’re still stuck in the dark ages with the newest AMIs available for Fedora being Fedora 8 based. With Fedora 12 around the corner, that’s two years old — something of an eternity in the pace of distribution development. I’d love to help out and build newer images, but while anyone can publish an AMI and make it public, you can’t publish newer kernel images, which really would be needed to use the newer system.

So, if you’re reading this at Amazon or know of someone I can talk with to try to move this forward, please let me know (katzj AT fedoraproject DOT org). I’d really strongly prefer to continue with Fedora and RHEL based images for our systems as opposed to starting to spin up Ubuntu images for the obvious reasons of familiarity.

Given that this seemed like a pretty typical problem, I figured I’d take a look and see what open source projects exist out there to see if any of them were suitable or could be at least close to a good fit for what we need and want. Unfortunately, I was kind of disappointed…

• Capistrano seems to be the big player in this arena. It was originally written for Rails and still very very strongly shows that heritage. This isn’t necessarily bad, but it makes it a lot harder to get to work if you’re not doing something that’s rails-like. There are some people who have gotten some things working with Java app deployments for tomcat, but they all feel a bit hacky. The other downside for me/us is that Capistrano is very much Ruby-based, both in how its own deployment language looks as well as some of the “how it depends on things working” aspects. Also, the fact that it’s written in Ruby and thus a little bit more difficult for us to hack on if/when we run into problems is a point against. So it’s probably a non-starter for now, or at least a pretty difficult sell
• Fabric is written in python and seems to be following in the footsteps of Capistrano. Right now, it’s far far simpler. This is in some ways good but some of the pieces that we’d want (eg, scm integration) aren’t there and so I’d have to write them. And I’m not sure if the Fabric devs are really interested in expanding in that way; haven’t sent email yet, but planning to tomorrow to feel it out.
• Config Management + Binary deployment is the approach taken in Fedora Infrastructure for app deployment and it seems to be working pretty well there. It might be something to get to eventually, but that’s going to be a longer term thing and I’m not actually convinced that it’s really the best approach. For Fedora it grew out of only a couple of things which could be considered “webapps” and a lot of system config that has turned much later into more webapps. It also pre-supposes a bit more homogenous of an environment than we use at HubSpot from the work I did there
• Func is something that a few people have been working on that I keep wanting to find a use for but it seems a little less well suited to doing a lot of java app building/deployment given that it’s more https/xml-rpc based than shell based.
• Roll your own is what we’re doing now and what it seems like is pretty common. I don’t necessarily like this, but it’s certainly the path of least resistance

So, what am I missing? Is there some great tool out there that I haven’t come found that you’re using for Java (and more) webapp deployments? Bonus points if its python-based and pretty extensible.

During that time, I’ve also had the opportunity to play a big role in the development and growth of Fedora. While the start was somewhat rocky, I think we’ve now built up an incredibly strong community that successfully releases a whole distribution (arguably, several!) on a regular schedule. And within that community, we’ve grown a pretty awesome set of leaders to continue to drive Fedora forward.

While I’m planning to still keep at least in touch with the goings-on of Fedora as well as running Fedora in places, I certainly won’t have the time to spend on it that I do today. I hope to keep in touch and see people at conferences and events from time to time. But right now, I’m looking forward to what’s next for me. And for those wondering, it’s something pretty different really. More on it next week..

Fast-forward to today and it’s a bit of a mess with things contributed by various people and used in one config tool (or two) and barely maintained. Also a lot of the things being wrapped have gotten a lot better in the python standard library. The gettext module is leaps and bounds better than the one from python 1.5 and also the subprocess module is awesome for spawning processes.

Therefore, I think it’s time to continue the cycle and kill off rhpl for Fedora 12. I’m starting to make patches and file them for packages using rhpl to transition them over. Help much appreciated from anyone that wants to join in.

For the rhpl.translate -> gettext case, you generally want to replace the import of _ and N_ from rhpl.translate with something like

import gettext
_ = lambda x: gettext.ldgettext(domain, x)
N_ = lambda x: x

The problem was that the first time I tested it, it looked like it overwrote the checksums we use for the mediacheck functionality in anaconda. It turns out I just wasn’t thinking — we need to implant the checksum *after* we do the isohybrid modification.

So without further ado, I’ve built a test version of the Fedora 11 boot.iso that is usable in this form. Testing of it would be much appreciated!

How to test

1. Download the test image
2. Try to burn it to a CD like you normally would. Ensure that it still boots normally. You don’t have to go through the full install, just boot it. Extra points if you can test mediacheck
3. Find a USB stick that’s at least 256 megs that doesn’t have any data you care about on it. Now try to write the test image to it using dd (dd if=test-isohybrid-boot.iso of=/path/to/device bs=1M). Again, you don’t have to install, just boot into the installer. Note that we won’t automatically find the second stage and you’ll get asked where to find the installer images.
4. Let me know the results in the comments (including type of machine).

Assuming this works, I’ll get the changes in so that we do this by default with boot.iso and then probably also try to make it so that the loader can automatically find the second stage image on either the CD or the USB stick. I’ll also consider doing similar for the livecds, although there’s more value with liveusb-creator / livecd-iso-to-disk there as you also want to set up persistence in a lot of cases.

There was a long and rambling discussion last week about the version of GRUB that’s shipped in Fedora and specifically the fact that the support for ext4 did not land in the version we shipped in Fedora 11. Now, as was said on the thread, this is because the patches weren’t reviewed and ready in time for beta (there are a couple of different ones… so which one is right?) and so we didn’t feel comfortable putting them in after beta, especially as with the way GRUB works, the same filesystem code gets used for ext2, ext3 and ext4 with the patches. A little unfortunate? Yes. Would it have been better if we had gotten them in so that you could do an install of Fedora 11 onto a single partition? Sure. But that’s one of the costs of a time-based release schedule.

In any case, one of the things that came out of the thread was that I gave a history of the version of GRUB in Fedora. For posterity, I’ll repeat that here, with some edits.

So, the gory history for those who might be interested. Eight years ago (!), we decided that the advantage of not having to rerun lilo after changing the config file as you can just read the config file off the filesystem with grub was worthwhile. We had, at that point, been patching lilo for quite a while to have a graphical menu. Therefore, keeping a graphical menu was a branding requirement. Connectiva at the time had a patch to grub that worked. We picked it up, shipped it, and it (mostly) worked. Efforts were made to integrate upstream, but they were largely uninterested. Along the way, significant changes to the graphics patch had to be made as grub evolved and a few other efforts were made to push it upstream. Eventually, the answer was “no, we’ll do something in the next big version of grub after grub 1.0″. Then the main developers went away and we were basically left maintaining a (large at this point) fork. As there is no upstream for grub 0.9x left, we’ve been left in a position of maintaining it and we’ve added some real features that have been needed along the way as grub 2′s progress has been slow at best and we were initially unhappy with some of the direction taken

So, that’s where we are today. We essentially ship a fork of GRUB 0.9x with graphics support, support for a lilo -R type functionality (so you can reboot once into a single kernel), EFI, and some more little things that I’m not thinking of right now.

With that in mind, I sat down and spent some time with a current snapshot of grub2. Overall, it’s made a lot of progress in the time since I last looked at it (a year ago? maybe a little more?). It was actually able to successfully boot for me in KVM and there’s equivalent graphics support to what we’re carrying in our grub 0.9x package. That said, there’s still quite a bit of things to verify exist before we can switch. And just in my look, there are a number of small things that would need work, especially around the way the config file gets created and updated. And with the very short runway for Fedora 12, I don’t think there’s really time to get it into shape in time. But I do think that it makes sense to look at for Fedora 13. So I’ve started a feature page to track as some of the things get tested and worked on. Then hopefully we can make the switch pretty painlessly early in the Fedora 13 cycle.

At this week’s (public IRC) Fedora Board meeting, Seth had kindly put the issue up for discussion and since it was a public meeting, I took part as well. Now the first question that might come to mind is “Why does the Board care / make the decision?”. And the reason is that it was the Board that originally voted to keep PPC as a primary arch until there was another functional secondary arch. Now that we’re two years later and there’s still not another functional secondary arch, I think that the topic was worth revisiting.

The discussion ended up being somewhat lively, but to me there were just a few key points. On the side of requiring PPC kept as primary, the argument seemed to mostly be that by going to secondary, PPC as an arch in Fedora would not be successful. One stated reason was the timeframe. So there was a proposal for keeping the requirement in place until either a secondary arch is up and running or some time limit (six months? a year?) was hit. The other side is that it’s been two years, what difference is a known six month time horizon going to make?

In the end, the Board voted and decided that from the Board level, PPC is no longer required to be a primary arch. That does not mean that PPC is now automatically a secondary arch.

So, what’s next? The next step is that I am proposing to FESCo that they consider a proposal to have PPC become a secondary arch for Fedora 12. And in doing so, put out a call for someone to lead a PPC secondary arch group. That would entail the work in keeping builders running, creating test releases, getting testing, etc. And then also being the person responsible for getting the release out and calling it done.

By making PPC a secondary arch, there are a few tangible benefits.

• Likely less build time for packages, rawhide, etc so that hopefully development can move a little faster
• Less last-minute scrambles to fix the PPC-specific distribution issue (whether it be installing on some platform or fitting on the DVD)
• More people caring about the secondary architecture process and thus hopefully helping it accelerate.

There are, though, also some risks and downsides. I’ll list them as well, but with my refutations

• It’s possible we could not have everything in place for a Fedora 12 PPC release. While I freely admit that this is a concern, I think that this is a concern no matter when we move PPC to be a secondary arch and I actually see the argument here getting stronger over time as the number of PPC users decrease as old Macs die.
• We may lose some portability testing in builds. Also, true. Hopefully the build infrastructure for secondary arches is far enough along that those can be reported still.

Thoughts, comments, etc appreciated as always

Okay, I know I’ve said this before, but I’m wondering why we continue with the illusion that PPC is a “primary” arch for Fedora rather than a secondary. At one point, the argument was that we were waiting until there was another viable secondary arch and then ppc would become one as well. The better part of four releases later (as I’m pretty sure I was having this discussion before Fedora 7′s release), we’re still in the same place. Maybe having ppc as a secondary arch would help to make that process smoother as those that care could help to improve the process.

As it stands, we instead say out of one side of our mouth that ppc is a primary arch but we have regularly dropped it from being included in a major development milestone, livecds were broken for 18 months (!), ps3 support is regularly broken in one way or another (and ps3 is one of the touted “most valuable” ppc platforms) and the majority of the bugs against ppc I see filed are either from jlaska or the IBM bugzilla proxy.

In any case, Fedora 11 will see restored the ability to create live images on ppc after someone reported it and Josh Boyer was kind enough to sit down and send me a patch on top of the obvious things based on the original report. Which was the original thing that got me thinking on this topic again.

But to go a step further — are you using Fedora on a ppc? If so, what kind of hardware and in what sort of use case? Inquiring minds, or at least I, want to know. Drop me a line in the comments.