We all know that Project Agape (uh-GAH-pay) has left the nptech world breathless with its amazing viral growth on Facebook. As a result of that growth and the constant hits on my earlier Project Agape post, I decided to do a quick interview with Project Agape. Micah Sifry has already done an excellent interview on the political and social implications of Project Agape but I had yet to see any technical information on how the Causes app on Facebook was built and maintained so I decided to call in for an interview. In short order, Project Agape turned around my request and hence today I have the chance to present to you quite a bit of information on how Project Agape was built and how it somehow manages to service all the users in the Causes app.

At the beginning of the interview, Joe Green, co-founder of Project Agape, ran down some vital statistics. As we were speaking, he estimated that the Causes app might acquire it’s one millionth user. It was at 998,000 and counting at the beginning of the interview and the interview took an hour. You do the math. Their largest Cause was the breast cancer Cause at 345,508. He also remarked that Causes were spreading faster than the groups themselves despite the fact that the groups had more extensive tools for faster viral growth such as mass invitations.

I started asking more technical questions and Chris Chan, one of their engineers, was brought in to answer them. Here are the harder numbers and specs for you web developers out there:

• mySQL 5.0
• Ruby on Rails
• 14 application servers, 2 database servers and 5 boxes in reserve
• 3 Image Servers (2 Squid, 1 image source)
• Apache 2 on Image Source
• WebApp servers have 2GB of ram
• 1 server for serving static content
• 26 Mongrel services per box
• 4 GB of RAM per box
• 50 – 60 MB of RAM per Mongrel service for a total use of around 1.5 GB of memory set aside for Mongrel

Additional:

• A pair of F5 Loadbalancers
• All systems running FreeBSD 6.2
• memcached
• nginx (not pound)

Specific Rails technologies used: nginx (not pound), Evented Mongrel, squid

Facebook and Rails gotchas

The Project Agape team was remarkably open when it came to discussing issues they ran into while building and maintaining their app. What follows is a list of things that Chris Chan ran into.

When Facebook ran requests via the API back to Project Agape’s servers, they came from only one or two IP addresses which caused the load to only be distributed to only one or two servers in the Facebook configuration. Oops. They had to adjust their load balancing techniques accordingly.

The Active Record bugaboo rears its head again. Rails uses an ORM (object-relational mapping) to closely tie database tables with objects in code. For instance, your Employees table in your mySQL database is referred to in code by using the word Employee. Unfortunately, having this layer of abstraction sometimes means that you can end up with suboptimal SQL code. That means relying less on Active Record’s beautiful way of denoting joined tables via a dot notation (“Employee.phone_number”) and just writing the SQL code yourself.

The “has_many” association in Ruby works quite well when in test but doesn’t work well when the “has_many” association you’re using now refers to an object containing a massive number of records. If a Cause “has_many” users, imagine the size of the object you’ll be building on the server just to service that request. Chris Chan mentioned an object several hundreds of megabytes in size. Ouch.

Also, Facebook’s profiles use a push model meaning that the code showing, for example, the number of people in a Cause can’t be easily updated because the push would have to occur for every profile of every member in Causes. Instead, Causes profiles link to an external URL hosted on Project Agape’s servers that serve a cached number.

All of the above should impress upon those of you looking to use Facebook in the nonprofit sector what Marc Andreesen’s said about the Facebook platform:

The implication is, in my view, quite clear — the Facebook Platform is primarily for use by either big companies, or venture-backed startups with the funding and capability to handle the slightly insane scale requirements. Individual developers are going to have a very hard time taking advantage of it in useful ways.

Project Agape has spent several hundreds of thousands of dollars handling the viral growth of its membership (37,000 users per day since Facebook Platform launch). They’ve got five developers. Clearly, a Facebook app is not trivial and not recommended for most nonprofits to try to approach.

UPDATE (6/21/2007): More details about the fundraising efforts about Facebook. They’re now up to $100,000 raised over a population of one million users. Project Agape itself is launching its new website on July 1st, 2007. They’re changing their name as well. I have a hunch that their new name will be Philotic as that name appears in their off-Facebook website pages but who knows?

The problem is this: do I have the time to keep imaging my most important app servers over and over again? Probably not, and the cost of something like Symantec’s Live State Recovery is prohibitive. Why not swap the hard drive with a relatively inexpensive piece of hardware like an SSD and then back it up with an Ghost image on a spare hard drive every once in a while? Let’s do the analysis:

A hard drive’s real-life lifespan is around 3-4 years on average. If you have drives this old, you’re definitely skating on thin ice. You have options at this point. You can image the drive now to another new hard drive and let that sit around for the day your drive finally dies. However, you’ll have downtime and your backed-up data will be old. This may or may not be an issue.

Installing an SSD now will keep you going for up to a decade. That’s so long that your turnkey application may be made redundant by then. Also, you know that the hardware is simply more reliable than a hard drive just by virtue of the way it’s made. Sure, go ahead and keep that spare drive around just in case and do your normal backups (if possible, I’ve noticed my turnkey servers tend not to play well with others). The only thing we have to fear are the remaining moving parts (the CPU fan or power supply fan). However, those things are trivial compared to a drive going down. I’ll take the loss of a mobo over a hard drive any time.

Sandisk, Samsung, and Ritek have made some press releases regarding their SSDs and we should start seeing them this year. As soon as they drop to less than $500 for a 32 GB device, I’m in for one at least.

Here’s are three office technologies that I guarantee will have more impact on your nonprofit than say, Second Life (which I believe is a boondoggle and a half) or Myspace (marginally interesting but Myspace doesn’t support an open API).

Copiers. They’ve advanced so far they’re not even copiers anymore. They’re MFPs (multifunction printers). Basically, all MFPs can print, scan, fax, e-mail, FTP or use SMB to send digital documents into your organization’s document store. These MFPs are not your father’s MFP.

As always, every MFP manufacturer out there has a different take on how to do things. Kyocera sells a fascinating turnkey solution called iTag which combines a workflow process designer, Sharepoint, and a Dell Server with the ability to dynamically modify the control panel of a printer in order to create custom workflows for your documents. Sharp is selling their printers with something called OSA which is tada! an open API for their printers. Canon has been selling the MEAP which gives developers the ability to write Java code that actually sits on their printers and can modify the printer’s control panel. All of these changes in MFP technology can only mean one thing — an easier interface for document retrieval and storage. You can literally pull case management documents straight off the printer without requiring a PC. This means that volunteers can be easily trained to implement document-based workflows and that employees won’t need to wrestle with a document management system just to do work.

Windows as a service. mygenii.org is now selling Windows dialtone for nonprofit organizations. What does it mean? It means you can use Remote Desktop on a Windows XP machine or thin client to connect to a session hosted on mygenii’s machines from anywhere in the world. In other words, we push the hassles and expense of maintaining dozens of workstations over to mygenii and as our desktops slowly go out of service we will be replacing them with thin clients. Essentially, this is a back to the 1970s model of resource allocation. Everyone at your organization will give up their local desktops in order to have a virtual desktop hosted elsewhere. This works extremely well for case workers should they want to connect to their work files at home. This stuff is so new that not many for-profits are doing it. Once you work the numbers though, you’ll see that this solution is a no-brainer.

Interactive voice response systems (IVR). One of the few things that do scale well in a nonprofit environment is telephony. I see seasonal variances with telephone usage due to the cold weather we experience every winter in New York that in turn causes our clients to seek shelter (and call my organization). This means we have to resort to automated means of answering the telephones. Using an IVRis the only way to scale a solution that can answer up to hundreds of calls an hour with the kind of information that our clients need.

Yes, I know, copiers aren’t sexy. Nobody has web pages devoted to Copiers 2.0 and nonprofit technology like social networking does. There will be no Second Life version of a bunch of people picking up the telephone and answering phone calls from distraught and anxious clients. There’s just going to be nonprofit workers trying to be resourceful with what they’ve got. It’s in our interest to get out of their way and provide them with the tools THEY need, not what some marketing consultant wants.

Many of the things that will be presented on this blog are not technical but are still essential to being a good IT director. Some of these things will include handling consultants, handling vendors and the myriad relationships you’ll form with them. Others topics will include the latest greatest new IT items that may have some relevance to a non-profit IT director. Check this space frequently as I hope to eventually to use this blog as a working knowledgebase of things relevant to non-profit IT workers.