Unfortunately, I was pretty disappointed with my options.

• Do some building by hand on an actual instance, then do the bundling and upload off of the running instance.
• Some of the ThinCrust stuff initially looked promising, but it seems like it’s largely unmaintained these days and the ec2 conversion bits didn’t really work at this point. I was able to get my initial images this way, but mostly by having a wrapper shell script of doom that made me sad.
• There’s always the rPath tools, but I wanted to stick to something more native and fully open source
• The new kid on the block is apparently BoxGrinder but I found it to be a lot over-complicated and not that robust. I’m sorry, but generating your own format that you then transform into a kickstart config and even run through appliance-creator via exec from your ruby tool just felt wrong. No offense, but just felt like a lot more than I wanted to deal with

So, I sat down and spent an evening hacking and have the beginnings of a working ami-creator.
It’s pretty straight-forward and uses all of the python-imgcreate stuff that’s used to build Fedora live images. Your input is a kickstart config and out the other side pops an image that you can bundle and upload to EC2.

Thus far, I’ve tested it to build CentOS 5 and Fedora 14 images. I’m sure there are some bugs but at this point, it’s worth getting it out for more people to play with. Hopefully it’s something that’s a lot simpler and more accessible for people to build images and I think it will also fit in a lot better with having Fedora release engineering building the EC2 images in Fedora 15 if they want.

One of the big outstanding pieces that I still want to add is the necessary bits to be able to (optionally) go ahead and upload and register as an AMI with your EC2 account.  But release early, release often.

Comments, etc appreciated in all the normal ways.

Minor update: switched the repo to live on github instead

• If you’re using cloud to refer to EC2, use EC2 instead. It’s concrete and it means very real things about your deployment and scaling models as well as how you’re managing your infrastructure.
• If you’re using cloud to refer to some service which runs over the Internet, either refer to the service or just say the Internet. You don’t store your mail “in the cloud”, you host it with Google apps. You don’t backup “to the cloud”, you have your backups stored over the Internet with Mozy or Carbonite.
• If you’re using cloud to refer to the idea of some hosted application platform, just say the platform. You don’t run your python app “in the cloud”, you run it on AppEngine (or something else).
• If you’re using cloud to mean that you are using virtualization and have some management stack on top of it, then please just say you’re running in a virtualized environment.
• If you’re using cloud to refer to having your server infrastructure hosted in a virtualized environment by someone else, again, just say you’re running in a virtualized environment.
• If you’re using cloud to refer to a “visible mass of little drops of water or frozen crystals suspended in the atmosphere”, then congratulations, you can continue to use the word cloud. And thanks to Wikipedia for the definition

Following this simple idea will let you avoid the otherwise impossible to avoid discussion of the semantics of the word “cloud” and what you happen to mean about it and how you might be wrong and … This then means you’ll be that much closer to achieving whatever goal you hoped to achieve as you’ll spend less time talking and more time doing. And as an added benefit, you’ll avoid getting grumpy emails from me about the fact that you’ve used such a terribly over-used and under-meaninged term.

Unfortunately, they’re still stuck in the dark ages with the newest AMIs available for Fedora being Fedora 8 based. With Fedora 12 around the corner, that’s two years old — something of an eternity in the pace of distribution development. I’d love to help out and build newer images, but while anyone can publish an AMI and make it public, you can’t publish newer kernel images, which really would be needed to use the newer system.

So, if you’re reading this at Amazon or know of someone I can talk with to try to move this forward, please let me know (katzj AT fedoraproject DOT org). I’d really strongly prefer to continue with Fedora and RHEL based images for our systems as opposed to starting to spin up Ubuntu images for the obvious reasons of familiarity.

Given that this seemed like a pretty typical problem, I figured I’d take a look and see what open source projects exist out there to see if any of them were suitable or could be at least close to a good fit for what we need and want. Unfortunately, I was kind of disappointed…

• Capistrano seems to be the big player in this arena. It was originally written for Rails and still very very strongly shows that heritage. This isn’t necessarily bad, but it makes it a lot harder to get to work if you’re not doing something that’s rails-like. There are some people who have gotten some things working with Java app deployments for tomcat, but they all feel a bit hacky. The other downside for me/us is that Capistrano is very much Ruby-based, both in how its own deployment language looks as well as some of the “how it depends on things working” aspects. Also, the fact that it’s written in Ruby and thus a little bit more difficult for us to hack on if/when we run into problems is a point against. So it’s probably a non-starter for now, or at least a pretty difficult sell
• Fabric is written in python and seems to be following in the footsteps of Capistrano. Right now, it’s far far simpler. This is in some ways good but some of the pieces that we’d want (eg, scm integration) aren’t there and so I’d have to write them. And I’m not sure if the Fabric devs are really interested in expanding in that way; haven’t sent email yet, but planning to tomorrow to feel it out.
• Config Management + Binary deployment is the approach taken in Fedora Infrastructure for app deployment and it seems to be working pretty well there. It might be something to get to eventually, but that’s going to be a longer term thing and I’m not actually convinced that it’s really the best approach. For Fedora it grew out of only a couple of things which could be considered “webapps” and a lot of system config that has turned much later into more webapps. It also pre-supposes a bit more homogenous of an environment than we use at HubSpot from the work I did there
• Func is something that a few people have been working on that I keep wanting to find a use for but it seems a little less well suited to doing a lot of java app building/deployment given that it’s more https/xml-rpc based than shell based.
• Roll your own is what we’re doing now and what it seems like is pretty common. I don’t necessarily like this, but it’s certainly the path of least resistance

So, what am I missing? Is there some great tool out there that I haven’t come found that you’re using for Java (and more) webapp deployments? Bonus points if its python-based and pretty extensible.

During that time, I’ve also had the opportunity to play a big role in the development and growth of Fedora. While the start was somewhat rocky, I think we’ve now built up an incredibly strong community that successfully releases a whole distribution (arguably, several!) on a regular schedule. And within that community, we’ve grown a pretty awesome set of leaders to continue to drive Fedora forward.

While I’m planning to still keep at least in touch with the goings-on of Fedora as well as running Fedora in places, I certainly won’t have the time to spend on it that I do today. I hope to keep in touch and see people at conferences and events from time to time. But right now, I’m looking forward to what’s next for me. And for those wondering, it’s something pretty different really. More on it next week..

Fast-forward to today and it’s a bit of a mess with things contributed by various people and used in one config tool (or two) and barely maintained. Also a lot of the things being wrapped have gotten a lot better in the python standard library. The gettext module is leaps and bounds better than the one from python 1.5 and also the subprocess module is awesome for spawning processes.

Therefore, I think it’s time to continue the cycle and kill off rhpl for Fedora 12. I’m starting to make patches and file them for packages using rhpl to transition them over. Help much appreciated from anyone that wants to join in.

For the rhpl.translate -> gettext case, you generally want to replace the import of _ and N_ from rhpl.translate with something like

import gettext
_ = lambda x: gettext.ldgettext(domain, x)
N_ = lambda x: x

The problem was that the first time I tested it, it looked like it overwrote the checksums we use for the mediacheck functionality in anaconda. It turns out I just wasn’t thinking — we need to implant the checksum *after* we do the isohybrid modification.

So without further ado, I’ve built a test version of the Fedora 11 boot.iso that is usable in this form. Testing of it would be much appreciated!

How to test

1. Download the test image
2. Try to burn it to a CD like you normally would. Ensure that it still boots normally. You don’t have to go through the full install, just boot it. Extra points if you can test mediacheck
3. Find a USB stick that’s at least 256 megs that doesn’t have any data you care about on it. Now try to write the test image to it using dd (dd if=test-isohybrid-boot.iso of=/path/to/device bs=1M). Again, you don’t have to install, just boot into the installer. Note that we won’t automatically find the second stage and you’ll get asked where to find the installer images.
4. Let me know the results in the comments (including type of machine).

Assuming this works, I’ll get the changes in so that we do this by default with boot.iso and then probably also try to make it so that the loader can automatically find the second stage image on either the CD or the USB stick. I’ll also consider doing similar for the livecds, although there’s more value with liveusb-creator / livecd-iso-to-disk there as you also want to set up persistence in a lot of cases.

There was a long and rambling discussion last week about the version of GRUB that’s shipped in Fedora and specifically the fact that the support for ext4 did not land in the version we shipped in Fedora 11. Now, as was said on the thread, this is because the patches weren’t reviewed and ready in time for beta (there are a couple of different ones… so which one is right?) and so we didn’t feel comfortable putting them in after beta, especially as with the way GRUB works, the same filesystem code gets used for ext2, ext3 and ext4 with the patches. A little unfortunate? Yes. Would it have been better if we had gotten them in so that you could do an install of Fedora 11 onto a single partition? Sure. But that’s one of the costs of a time-based release schedule.

In any case, one of the things that came out of the thread was that I gave a history of the version of GRUB in Fedora. For posterity, I’ll repeat that here, with some edits.

So, the gory history for those who might be interested. Eight years ago (!), we decided that the advantage of not having to rerun lilo after changing the config file as you can just read the config file off the filesystem with grub was worthwhile. We had, at that point, been patching lilo for quite a while to have a graphical menu. Therefore, keeping a graphical menu was a branding requirement. Connectiva at the time had a patch to grub that worked. We picked it up, shipped it, and it (mostly) worked. Efforts were made to integrate upstream, but they were largely uninterested. Along the way, significant changes to the graphics patch had to be made as grub evolved and a few other efforts were made to push it upstream. Eventually, the answer was “no, we’ll do something in the next big version of grub after grub 1.0″. Then the main developers went away and we were basically left maintaining a (large at this point) fork. As there is no upstream for grub 0.9x left, we’ve been left in a position of maintaining it and we’ve added some real features that have been needed along the way as grub 2′s progress has been slow at best and we were initially unhappy with some of the direction taken

So, that’s where we are today. We essentially ship a fork of GRUB 0.9x with graphics support, support for a lilo -R type functionality (so you can reboot once into a single kernel), EFI, and some more little things that I’m not thinking of right now.

With that in mind, I sat down and spent some time with a current snapshot of grub2. Overall, it’s made a lot of progress in the time since I last looked at it (a year ago? maybe a little more?). It was actually able to successfully boot for me in KVM and there’s equivalent graphics support to what we’re carrying in our grub 0.9x package. That said, there’s still quite a bit of things to verify exist before we can switch. And just in my look, there are a number of small things that would need work, especially around the way the config file gets created and updated. And with the very short runway for Fedora 12, I don’t think there’s really time to get it into shape in time. But I do think that it makes sense to look at for Fedora 13. So I’ve started a feature page to track as some of the things get tested and worked on. Then hopefully we can make the switch pretty painlessly early in the Fedora 13 cycle.

At this week’s (public IRC) Fedora Board meeting, Seth had kindly put the issue up for discussion and since it was a public meeting, I took part as well. Now the first question that might come to mind is “Why does the Board care / make the decision?”. And the reason is that it was the Board that originally voted to keep PPC as a primary arch until there was another functional secondary arch. Now that we’re two years later and there’s still not another functional secondary arch, I think that the topic was worth revisiting.

The discussion ended up being somewhat lively, but to me there were just a few key points. On the side of requiring PPC kept as primary, the argument seemed to mostly be that by going to secondary, PPC as an arch in Fedora would not be successful. One stated reason was the timeframe. So there was a proposal for keeping the requirement in place until either a secondary arch is up and running or some time limit (six months? a year?) was hit. The other side is that it’s been two years, what difference is a known six month time horizon going to make?

In the end, the Board voted and decided that from the Board level, PPC is no longer required to be a primary arch. That does not mean that PPC is now automatically a secondary arch.

So, what’s next? The next step is that I am proposing to FESCo that they consider a proposal to have PPC become a secondary arch for Fedora 12. And in doing so, put out a call for someone to lead a PPC secondary arch group. That would entail the work in keeping builders running, creating test releases, getting testing, etc. And then also being the person responsible for getting the release out and calling it done.

By making PPC a secondary arch, there are a few tangible benefits.

• Likely less build time for packages, rawhide, etc so that hopefully development can move a little faster
• Less last-minute scrambles to fix the PPC-specific distribution issue (whether it be installing on some platform or fitting on the DVD)
• More people caring about the secondary architecture process and thus hopefully helping it accelerate.

There are, though, also some risks and downsides. I’ll list them as well, but with my refutations

• It’s possible we could not have everything in place for a Fedora 12 PPC release. While I freely admit that this is a concern, I think that this is a concern no matter when we move PPC to be a secondary arch and I actually see the argument here getting stronger over time as the number of PPC users decrease as old Macs die.
• We may lose some portability testing in builds. Also, true. Hopefully the build infrastructure for secondary arches is far enough along that those can be reported still.

Thoughts, comments, etc appreciated as always

Okay, I know I’ve said this before, but I’m wondering why we continue with the illusion that PPC is a “primary” arch for Fedora rather than a secondary. At one point, the argument was that we were waiting until there was another viable secondary arch and then ppc would become one as well. The better part of four releases later (as I’m pretty sure I was having this discussion before Fedora 7′s release), we’re still in the same place. Maybe having ppc as a secondary arch would help to make that process smoother as those that care could help to improve the process.

As it stands, we instead say out of one side of our mouth that ppc is a primary arch but we have regularly dropped it from being included in a major development milestone, livecds were broken for 18 months (!), ps3 support is regularly broken in one way or another (and ps3 is one of the touted “most valuable” ppc platforms) and the majority of the bugs against ppc I see filed are either from jlaska or the IBM bugzilla proxy.

In any case, Fedora 11 will see restored the ability to create live images on ppc after someone reported it and Josh Boyer was kind enough to sit down and send me a patch on top of the obvious things based on the original report. Which was the original thing that got me thinking on this topic again.

But to go a step further — are you using Fedora on a ppc? If so, what kind of hardware and in what sort of use case? Inquiring minds, or at least I, want to know. Drop me a line in the comments.

Today, nearly 19000 commits later, the progress continues. And in one of those somewhat expected twists, we’re actually now deprecating the (interactive) text mode and stripping it to its very core. The graphical install has succeeded, I think, beyond what anyone would have expected.

That said, hopefully I won’t be writing a post like this in another ten years

First on the list was testing out the livecd. As is often the case, there were a variety of things which had either entirely or partially broken. Also, there was a (good) suggestion to go ahead and install xguest with the live image so that people can take advantage of the good work that Dan has done there. Luckily, this was all pretty straight-forward things and involved a few fixes here and there.

They did, though, highlight the fact that we lose out on some pretty valuable testing by not making live images available more regularly. The problems always come down, though, to how would we distribute such images — the mirrors probably wouldn’t like 700 megs x 2 arches x n spins (at least desktop + KDE would make sense) churning on a daily basis I don’t think. Especially since live images aren’t rsync-able. Would people be okay with torrent only distribution of more frequent snapshots? And be okay with live snapshots that were just produced in an automated fashion without any testing at all before they go out? Comments appreciated

The bigger thing that took some time was helping to get the new anaconda storage code working with the live install. This is something which isn’t big and glitzy because right now, it’s all unglamarous backend code. But Dave Lehman hammered out a nice start to overhaul the storage code in anaconda to take into account more of the things which are “modern” storage/partitioning needs. This has then been supplemented by an avalanche of patches from the rest of the anaconda team to get things into shape. My patches were some small ones to deal with some of the more interesting quirks with how we do an install from the livecd. Luckily, as of this afternoon, it looks like we have something there that will work pretty nicely. A shout out to the anaconda dudes for the hard work they’ve been putting into getting it into shape and pulling out what was one of the last pieces of anaconda that’s more than five years old. In Fedora 12, hopefully we can move on to the next step which is overhauling the user experience for partitioning in a major way.

The git way seems to largely be that the most common way for someone new to contribute code is that they write a patch (or series of patches), commit them locally, and then send them via email for review and eventual merging. These are made easier through the existence of commands like git format-patch, git send-email and git am (apply-mailbox). These really are wonderful tools if your workflow is around using email for patch review and merging as is done in the kernel and many other places. There’s certainly an argument to be made that this sort of mailing list review ends up improving code quality. But it’s not the only way. And since git is about there being multiple ways to do things, maybe some of the other ways need to be made easier too…

One of the ones that seems to pop up commonly is that people want to have their trees available for others to pull from/merge in. This is doable today, but as Scott notes, it’s not entirely straight-forward and requires some knowledge of how to set things up. What if, instead, there was a git publish command that was able to essentially push your working branch to a remote location (via ssh/scp) and do the things necessary so that it’s clonable via http out of the box and not require any initial login and creating a repo, etc. The question that comes to mind is “if I run git publish again, how does it differ from a git push” and the answer is probably not much, but I’m open to other opinions

The obvious second step after that would be to add easy-to-use support for publishing to somewhere like gitorious. That would help for people who don’t have web hosting or a number of other things. And that is a gap that bzr ends up filling by allowing people to use launchpad for arbitrary bzr hosting.

The other workflow that might be interesting to better support would be something like Review Board. I keep wanting to set up an instance to play with and possibly even have people use for things like livecd-tools or anaconda patches. It seems to combine some of the upsides of mailing list review (easy for all to see, easy to annotate changes, …) without some of the downsides (mailing lists suck, a topic for another day) There’s some movement underway to try to get an install going for Fedora Infrastructure, so if you’re interested, help is wanted!.

What do people think? Would a git publish command as described above (minus the aside of the Review Board stuff) be useful/interesting?

The book was a quick and easy read. There’s really not anything ground-breaking or revolutionary present, instead, it’s a good overview of lots of common sense things. And even though it’s a few years old for the most recent edition, it tends to still apply today. It would be nice to see an update taking into account some of the impact that AJAX has had on web design, but I’m not certain that the impact is really that large for what he’s really focused on.

The most basic point really is to make things obvious. Don’t think that you’re a whizz-bang designer and can make something incredibly new in terms of interfaces for the web unless you’ve really really spent a lot of time testing it out. Because even though the obvious is boring, the very fact that it’s boring makes people comfortable.

The other key point that he stressed that should be obvious to people developing software is to test early and test often and then iterate. It’s better to test the same web site twice with 3 people each time than to test once with 8 people. Again, kind of obvious, but too often passed up on in the effort to build something “beautiful” the first time.

If it were an incredibly long book I would have gotten annoyed by some of the simplicity of it all but at about 200 pages it was perfectly effective. So if you’re working on websites, I’d recommend reading it either by buying a copy or by visiting your local library.

And as a side note, your local library is a wonderful resource that too many people neglect. I’ve been making it a point to check more books out from the library over the past few months as opposed to purchasing them. I figure I’ll turn around some of the money I previously spent buying books to support the library and have the benefit of more to read and less clutter in my house. But the ability to go to their website, request books, and then just drop in and pick them up, read them, and return them is great.

Thanks to bpepple for unintentionally encouraging me to do this (I’ve been thinking about it for a while, but never get around to it) by asking for a review on fedora-devel-list last night!

Now, if we could get to people actually commenting on the core ideas of being entirely event driven and helping to fix various poor kernel/userspace interactions there (device-mapper’s integration with udev is poor and mdadm’s incremental mode didn’t work for me when I tried poking at it last) maybe there’d be some progress. Unfortunately, klibc and busybox and $existing_tools end up being the core of any discussion anywhere.

And it’s that propensity which sometimes make me unclear on the future of Linux. :-/

Breaking it down into components, I think there are a few things which we have to worry about

1. Where to put things on the disk. 

   The more I think about it, the more I think that having “everything” for one image (including the bootloader bits) will be easiest.  The suggestion of using the iso label for namespacing seems to make reasonable sense.

2. Boot loader config

   There are a couple of possibilities here… the simplest would just be that we edit the syslinux.cfg and append more entries.  A little bit more complicated would be to just have a simple “top-level” syslinux.cfg which uses CONFIG directives to load the additional config files.  Third would be a com32 module to iterate and set up the menu items automatically.  I lean towards the second, at least for at first unless someone is super-motivated to write a com32 module

3. How to know the new directory

   There are multiple places scattered that have hard-coded looking for LiveOS.  There’s the initrd, there’s the livesys initscript and there’s livecd-iso-to-disk/liveusb-creator.  While we could add a kernel command line option and go and change each of these to set it/respect it, it feels like maybe the less general solution.  Maybe it’s just that we should always have livelabel= available so that we can use it other places? 

Of these, I’m especially not sold with the last question and would love to see some thoughts or ideas from others. From there, it should be pretty straight-forward to implement so that we can have everything in place for Fedora 11 Beta

On a related note — I wonder if there’s value in moving the livesys initscript from being based in the live image kickstart config to being provided by a package (most obviously, initscripts)

I have, though, done various updates to twitter and identi.ca if you have some obsessive need to know what I’ve been doing.  It hasn’t been that exciting, though.  Basically boils down to the following relatively short list

• Went cross-country skiing a couple of times.  With the very wintry weather we’ve had thus far this winter, it’s been something good to be able to get outside and do as it hasn’t exactly been ideal biking weather
• FUDCon F11 was held in Cambridge at MIT.  Since it was in E51 and I knew where things were, I spent a fair bit of time running around.  I had some good conversations, but didn’t give any presentations and didn’t really get any hacking done with the hackfest
• The SDM 09s have started and I helped some with their first design challenge.  Was fun to watch and they seem a good bunch
• Have been trying to read a fair bit and so made good progress on my book backlog.  Still hoping to finish that before classes start back up
• Some poking and prodding in the hopes of getting Fedora 11 alpha out the door in a semi-decent shape
• More work on the new initramfs tooling, although it’s making slower progress than I’d really like
• Getting extra sleep

The majority of the ideas behind the code are that we want to use the same tools and utilities that exist on the booted system (so regular bash, regular udev, no nash, no statically linked nonsense, no klibc) and also that we want to not have the initramfs as a slow-down. The latter means that we run pretty much everything off of udev rules right now so that as soon as the devices are available, we mount and switch into the rootfs.

So go ahead 'git clone git://fedorapeople.org/~katzj/dracut.git' and read the README, HACKING and TODO files. Also sent to lkml to get the mailing list started. More hands more than welcome…

Overall, it's been a good semester. The workload ended up being a bit higher than I expected, but it was probably what I should have thought. I knew that System Architecture was going to be time-consuming, but it was still more so than I thought. Similarly, Project Management ended up requiring more time both for the homeworks and the project than I really expected from the outset.

Those along with working to get all of the Fedora 10 bits working on the OLPC meant that it was just a very busy semester. But, now it's actually the time when I get nearly two monhs to relax and “just” work. Well, and I also am hoping to try to get some progress underway for my thesis so that I don't have to do it all while juggling classes also. It should still be a good sort of break, though. And then, it's on to the spring semester. Which I still need to figure out what I'm going to take — suggestions welcome