alex.polvi.net

Oh, Firefox

Firefox really enjoys melting down infrastructures. Whenever possible, it draws a bunch of attention, saturates a bunch of bandwidth, and often makes a sysadmins life chaotic. While 1.5 did draw a lot of attention and use a lot of bandwidth, we still were able to get our sleep.

November 2004 I was up late with justdave and other sysadmins as they prepped for a 1.0 midnight launch. If I recall correctly, around 4am most of the mirrors had gone poof. Bouncer, the weighted random mirror redirector created by Mike Morgan, was put into production and saved the day. The mirror network came back alive, and the end users just kept downloading and downloading.

This time the mirror network was rock solid. We had more traffic then the 1.0 release, but with improved bouncer and a solid mirror infrastructure, distribution of 1.5 was a non-event.

It was particularly encouraging to see that other people noticed too! Just the fact that netcraft reported on good sysadmining (most news sites are negative) is great aswell.

Slashwhat?

The slashdot effect is dead. The new kid on the block is digg.

Figure 1: One of the mirrors in ftp.mozilla.org round robin.

Try to guess when we were on the digg homepage… Sort of makes sense, slashdot has not done anything new for the end user in many years.

Beginning of a new era

I really feel that the launch of mozilla.com is of historical significance. One of my various jobs on release day was to flip the httpd configs for the site. I had nothing to do with the design or the content, but I did have the honor of striking enter and making it open for public consumption. It was sort of a stressful launch, but the kind and patient Paul Kim was giving nothing but support.

As sysadmins we strive to keep our services under any demand. To do this, we supported mozilla.com across twelve hosts load balanced with with Linux Virtual Server (LVS). We heavily rely on LVS to do all the web-app heavy lifting. LVS combined with end GNU/Linux hosts allow us to scale to infinity. The only real challenge has been scaling things like SFX… but that solution is in the works.

The team

Along with mozilla.com brought Justin, our Director of IT. Sysadmins now have a full team, manager and all. Between the four of us we are working to take our infrastructure to the next level. Moving away from the ’shoot off the hip’ attitude and into a position where nothing catches us by surprise.

Yep — growing product and a growing infrastructure make for fun and challenging sysadmin position. :)

This entry was posted on Saturday, December 10th, 2005 at 3:47 pm and is filed under Mozilla.