We all know that Project Agape (uh-GAH-pay) has left the nptech world breathless with its amazing viral growth on Facebook. As a result of that growth and the constant hits on my earlier Project Agape post, I decided to do a quick interview with Project Agape. Micah Sifry has already done an excellent interview on the political and social implications of Project Agape but I had yet to see any technical information on how the Causes app on Facebook was built and maintained so I decided to call in for an interview. In short order, Project Agape turned around my request and hence today I have the chance to present to you quite a bit of information on how Project Agape was built and how it somehow manages to service all the users in the Causes app.
At the beginning of the interview, Joe Green, co-founder of Project Agape, ran down some vital statistics. As we were speaking, he estimated that the Causes app might acquire it’s one millionth user. It was at 998,000 and counting at the beginning of the interview and the interview took an hour. You do the math. Their largest Cause was the breast cancer Cause at 345,508. He also remarked that Causes were spreading faster than the groups themselves despite the fact that the groups had more extensive tools for faster viral growth such as mass invitations.
I started asking more technical questions and Chris Chan, one of their engineers, was brought in to answer them. Here are the harder numbers and specs for you web developers out there:
- mySQL 5.0
- Ruby on Rails
- 14 application servers, 2 database servers and 5 boxes in reserve
- 3 Image Servers (2 Squid, 1 image source)
- Apache 2 on Image Source
- WebApp servers have 2GB of ram
- 1 server for serving static content
- 26 Mongrel services per box
4 GB of RAM per box- 50 – 60 MB of RAM per Mongrel service for a total use of around 1.5 GB of memory set aside for Mongrel
Additional:
Specific Rails technologies used: nginx (not pound), Evented Mongrel, squid
Facebook and Rails gotchas
The Project Agape team was remarkably open when it came to discussing issues they ran into while building and maintaining their app. What follows is a list of things that Chris Chan ran into.
When Facebook ran requests via the API back to Project Agape’s servers, they came from only one or two IP addresses which caused the load to only be distributed to only one or two servers in the Facebook configuration. Oops. They had to adjust their load balancing techniques accordingly.
The Active Record bugaboo rears its head again. Rails uses an ORM (object-relational mapping) to closely tie database tables with objects in code. For instance, your Employees table in your mySQL database is referred to in code by using the word Employee. Unfortunately, having this layer of abstraction sometimes means that you can end up with suboptimal SQL code. That means relying less on Active Record’s beautiful way of denoting joined tables via a dot notation (”Employee.phone_number”) and just writing the SQL code yourself.
The “has_many” association in Ruby works quite well when in test but doesn’t work well when the “has_many” association you’re using now refers to an object containing a massive number of records. If a Cause “has_many” users, imagine the size of the object you’ll be building on the server just to service that request. Chris Chan mentioned an object several hundreds of megabytes in size. Ouch.
Also, Facebook’s profiles use a push model meaning that the code showing, for example, the number of people in a Cause can’t be easily updated because the push would have to occur for every profile of every member in Causes. Instead, Causes profiles link to an external URL hosted on Project Agape’s servers that serve a cached number.
All of the above should impress upon those of you looking to use Facebook in the nonprofit sector what Marc Andreesen’s said about the Facebook platform:
The implication is, in my view, quite clear — the Facebook Platform is primarily for use by either big companies, or venture-backed startups with the funding and capability to handle the slightly insane scale requirements. Individual developers are going to have a very hard time taking advantage of it in useful ways.
Project Agape has spent several hundreds of thousands of dollars handling the viral growth of its membership (37,000 users per day since Facebook Platform launch). They’ve got five developers. Clearly, a Facebook app is not trivial and not recommended for most nonprofits to try to approach.
UPDATE (6/21/2007): More details about the fundraising efforts about Facebook. They’re now up to $100,000 raised over a population of one million users. Project Agape itself is launching its new website on July 1st, 2007. They’re changing their name as well. I have a hunch that their new name will be Philotic as that name appears in their off-Facebook website pages but who knows?
Loading ...
