Non-Profit Tech Blog

Daily Kos unveiled their new server infrastructure just recently:

Hardware is part of the solution, Bingham says. “To handle the traffic better, we moved to a cluster of six quad core Xeons with 8GB RAM for webheads that all boot off a central NFS (Network File System) root, with the capability of adding more webheads as needed,” he said. Daily Kos also added two 16GB eight-core Xeons and a 6×73GB RAID-10 array for database files running a MySQL master/slave setup [...]

Many database-driven sites facing scalability challenges have looked to the distributed caching system memcached, which helps speed dynamic web applications. Bingham says memcached has played an important role in scaling the Daily Kos site. “I greatly expanded memcached usage with 1GB instances memcached running on each webhead, which they all share,” said Bingham. “The backend also places fully rendered pages into memcached, which a hacked up lighttpd running as the front end proxy then serves these pages from memcached directly to anonymous users. This has helped the sites performance immensely, since not only does it spread the work of rendering these pages around to the all of the webheads, but it greatly reduces the amount of work the backend has to do.”

9 million pageviews were served with this setup. memcached turns out to be a piece of technology that Causes uses for its own servers. I’m quite curious as to why they don’t use an EC2-based infrastructure and why they still use lighttpd but who knows?

It turns out that Daily Kos’ widget for serving election results were designed by Free Range Studios. Ok, to be honest, I used the NY Times electoral results more since I was on my Windows Mobile phone. The NY Times election page was superior to the Daily Kos’ Flash widget in that it rendered very quickly and had better information architecture. Normally, I like the work of Free Range Studios but the Daily Kos widget had all the charm of a NORAD map charting incoming Soviet ICBMs a la the movie Wargames.

I love talking about hardware configurations as they can give you a ton of insight as to how you can architect your own solutions for whatever application you create that gets the same level of traffic. Those of you looking to build a Daily Kos-like site can check out Scoop. I don’t recommend using this software unless you’re prepared to do extensive customization. The last stable release of scoop has a 2006 timestamp. That doesn’t bode well for security updates, etc.