We all know that Project Agape (uh-GAH-pay) has left the nptech world breathless with its amazing viral growth on Facebook. As a result of that growth and the constant hits on my earlier Project Agape post, I decided to do a quick interview with Project Agape. Micah Sifry has already done an excellent interview on the political and social implications of Project Agape but I had yet to see any technical information on how the Causes app on Facebook was built and maintained so I decided to call in for an interview. In short order, Project Agape turned around my request and hence today I have the chance to present to you quite a bit of information on how Project Agape was built and how it somehow manages to service all the users in the Causes app.
At the beginning of the interview, Joe Green, co-founder of Project Agape, ran down some vital statistics. As we were speaking, he estimated that the Causes app might acquire it’s one millionth user. It was at 998,000 and counting at the beginning of the interview and the interview took an hour. You do the math. Their largest Cause was the breast cancer Cause at 345,508. He also remarked that Causes were spreading faster than the groups themselves despite the fact that the groups had more extensive tools for faster viral growth such as mass invitations.
I started asking more technical questions and Chris Chan, one of their engineers, was brought in to answer them. Here are the harder numbers and specs for you web developers out there:
- mySQL 5.0
- Ruby on Rails
- 14 application servers, 2 database servers and 5 boxes in reserve
- 3 Image Servers (2 Squid, 1 image source)
- Apache 2 on Image Source
- WebApp servers have 2GB of ram
- 1 server for serving static content
- 26 Mongrel services per box
4 GB of RAM per box- 50 – 60 MB of RAM per Mongrel service for a total use of around 1.5 GB of memory set aside for Mongrel
Additional:
Specific Rails technologies used: nginx (not pound), Evented Mongrel, squid
Facebook and Rails gotchas
The Project Agape team was remarkably open when it came to discussing issues they ran into while building and maintaining their app. What follows is a list of things that Chris Chan ran into.
When Facebook ran requests via the API back to Project Agape’s servers, they came from only one or two IP addresses which caused the load to only be distributed to only one or two servers in the Facebook configuration. Oops. They had to adjust their load balancing techniques accordingly.
The Active Record bugaboo rears its head again. Rails uses an ORM (object-relational mapping) to closely tie database tables with objects in code. For instance, your Employees table in your mySQL database is referred to in code by using the word Employee. Unfortunately, having this layer of abstraction sometimes means that you can end up with suboptimal SQL code. That means relying less on Active Record’s beautiful way of denoting joined tables via a dot notation (“Employee.phone_number”) and just writing the SQL code yourself.
The “has_many” association in Ruby works quite well when in test but doesn’t work well when the “has_many” association you’re using now refers to an object containing a massive number of records. If a Cause “has_many” users, imagine the size of the object you’ll be building on the server just to service that request. Chris Chan mentioned an object several hundreds of megabytes in size. Ouch.
Also, Facebook’s profiles use a push model meaning that the code showing, for example, the number of people in a Cause can’t be easily updated because the push would have to occur for every profile of every member in Causes. Instead, Causes profiles link to an external URL hosted on Project Agape’s servers that serve a cached number.
All of the above should impress upon those of you looking to use Facebook in the nonprofit sector what Marc Andreesen’s said about the Facebook platform:
The implication is, in my view, quite clear — the Facebook Platform is primarily for use by either big companies, or venture-backed startups with the funding and capability to handle the slightly insane scale requirements. Individual developers are going to have a very hard time taking advantage of it in useful ways.
Project Agape has spent several hundreds of thousands of dollars handling the viral growth of its membership (37,000 users per day since Facebook Platform launch). They’ve got five developers. Clearly, a Facebook app is not trivial and not recommended for most nonprofits to try to approach.
UPDATE (6/21/2007): More details about the fundraising efforts about Facebook. They’re now up to $100,000 raised over a population of one million users. Project Agape itself is launching its new website on July 1st, 2007. They’re changing their name as well. I have a hunch that their new name will be Philotic as that name appears in their off-Facebook website pages but who knows?
Loading ...
Project Agape/Causes has done a great job at getting onto Facebook as soon as they opened their platform and indeed has had great growth numbers. But ultimately, given that it is a fundraising service, shouldn't the effectiveness be judged by how much funds have been raised, rather than the number of users? From what I can tell, a handful of well-known nonprofits have raised decent amounts ($5-15K) but the majority of causes have raised minimal if any.
I'm not trying to knock the service, as again I think what they are doing is great and well-executed, but I think it is a little preliminary to call this the next great thing in fundraising based on the number of users. It's called *fundraising*, after all.
Alan-
Great interview — really shows off how non-trivial building an app is.
Bill, +1 & amen, brother. Users are here today, gone tomorrow.
For some reason, nonprofit workers are always looking to compare online donations with existing fundraising methods without compensating for the long history of direct mail, cold calling, benefits, walkathons, etc., and the vast amount of analysis devoted to them. How many billions of dollars have been spent on fundraising analysis in the last few decades in our sector? It takes time to perfect things although I would argue that current non-electronic methods are incredibly inefficient.
This is a new age for nonprofits. Technology drives many more business decisions than they used to simply because we know that future monies for nonprofits aren’t going to be reached via traditional means but by various electronic methods. You know that and I know that. Is Causes the next great thing in fundraising? I would suggest yes simply because of its viral growth. Bill, I would have to ask you if there was anything you’ve ever seen in the offline world that grew a one million-strong list of users in 27 days. If you’ve got a great example, I’d love to see it.
Does having one million users mean anything? In the world of marketing analytics, yes it does. These folks are untapped — they’re a very large test bed for all sorts of marketing experiments. Take a look at the Facebook API and at Facebook itself — do you understand the ramifications of a nonprofit social network that has complete access to the various pieces of opt-in demographic information for one million users? This is equivalent to asking every person in a one million user database a complete Census-like survey and getting an incredibly high response rate for each survey. If that fact hasn’t been made clear to you, I hope it does now. Frankly, I had hoped people had picked up on the obviousness of that. It’s a marketer’s dream. I would suggest that if done correctly, data mining and predictive analytics applied to this Causes database will result in higher response rates, higher donations per user and higher year over year retention rates. This is cutting-edge fundraising staring at nonprofits in the face and instead all they want is an instant payoff. It’s a terribly short-sighted viewpoint and I urge you to instead try to grapple with the larger ramifications of viral social networks. This is NOT the time to stick your head in the sand.
From a strategy perspective, Agape is so much sexier than, say, Change.org because it integrates with an existing network instead of building its own. But both of them go beyond mere “fundraising” by directly involving people in these awareness campaigns.
Both of them are brilliant because they really reinvent the landscape for mission-driven networking. And we are only seeing the tiniest, teensiest, itty-bitty speck of what this type of networking is going to be capable of. The bottom line is a fair measure of success, but Allan is right: fuck the money metric. 1 million self-selected people. 27 days.
Alan and Chris, I am not doubting that Causes offers some significant *potential* for fundraising. But to say that Causes can be classified as a great success based on having 1M users install the app in a few weeks is rather premature.
Why is it premature? Ultimately, these users have to convert to donations for them to have value (again, it is a *fundraising* service after all). Currently, the “$donation per user” metric for Causes is around $0.05 from my calculations, meaning that when projected out to the entire Facebook user base this would result in around $1M in total donations, with the vast majority of this going to a handful of well-known causes (the top 5 or so causes currently account for ~60% of donations to date). $1M in annual fundraising for a handful of well know causes is not a great success from my perspective.
Can this $donation/user increase via the application of data mining and predictive analytics, as you say? Yes, it can. But will it? We’ll see. Maybe Causes will raise $1M a year, maybe more, maybe less, but in any case the jury is still out.
Look, there are 18 other Facebook apps with more users than Causes. Does this mean that they are “more successful” because they have more users? No! The fact is that Facebook users are adding apps like crazy, for all sorts of reasons, and some of these apps will be heavily used and will deliver the metrics that the app providers are looking for, and some will not. To say “fuck the money metric, there are 1M users” is incredibly naive at best.
To further my point, consider Lending Club, the Facebook-only P-P lending service.
Lending Club has 10K users. Causes has 1M users. Thus, by your analysis, Causes is 100X more “successful” than Lending Club. So Causes is a huge success, Lending Club a big failure.
But what are these services’ ultimate objective? For Causes, to raise funds for nonprofits. For Lending Club, to facilitate loans between users. And how are they doing by these objectives?
Causes: ~$50K funds raised
Lending Club: $300K loans facilitated
Hmmm…which one is the more successful again?
I realize that comparing a P-P lending service to a fundraising service is not necessarily meaningful, but I think the point should be clear that just because you have a shitload of users, doesn’t necessarily mean you are going to get a shitload of whatever you are ultimately trying to get. It’s called the conversion funnel, and obviously Lending Club’s funnel is a whole lot better than Causes’. And unless Causes conversion funnel improves dramatically then it will have an *immaterial* impact on the $6B online fundraising market.
Again…I like Causes. I think that it very well may offer a significant potential for fundraisers. But it is way, way, way to early to call it a smashing success.
The fact that there are 18 other apps that have more users than Causes is not much of a fact. None of them are in the donations to nonprofits space. Also, your donation per user metric is wrong. It’s about ten cents a user. Project Agape is self-reporting $100,000 in funds raised over one million users. I think the reason that people like me and Chris focus on the explosive growth of the Causes app is that as web developers we know how difficult it is to acquire users and we know the app is only 28 days old. It’s WAY WAY too early to tell.
Just for kicks, I removed the Causes app and re-added it just to see what permissions Causes asks for. The key permissions are “Know who I am and access my information”, “Publish stories in my News Feed and Mini-Feed”, “Place a link below the profile picture on any profile”. It’s really not that hard to extrapolate what the Project Agape aka Philotic people are going to do. They will have the ability to insert content about nonprofits at will into a web page that is read by millions of Facebook users sometimes once or even twice a day and tailor it based on that user’s Facebook profile. I don’t know how much more I have to spell it out to you but it’s clear that this $donation per user metric will rise. I think myself and many technologists can easily come up with two or three advanced algorithms that would knock the socks off any eCRM’s analytics engines if we had that kind of information at our disposal. And with the viral growth of the Causes app, by this time next year, even a modest metric of fifty cents per user will be worth millions of dollars.
For you to naysay this now while it’s in its infancy, just seems even more naive to me. It’s clear that the Project Agape site will increase the level of interactivity with Causes users and start to ramp up that metric. The conversation between Project Agape and its users has yet to begin, wouldn’t it just be better to listen and observe so that one can learn to enter the conversation? Or would it be better to just walk away as you propose simply because the money on the table is too small?
I am not naysaying. I am not “walking away”. I am saying that it is too early to call Causes a runaway success based on it reaching 1M users. And I’m willing to wager that Causes management agrees with me.
@Bill: I feel you. I don’t want to be a cheerleader.
I just feel like getting lots of attention is a good thing *in itself* and I am viewing that from an evaluation perspective as an output — an achieved goal — rather than a metric.
If I was an E.D. at a NPO I would see any financial return (actual real, live donations) as gravy. In my mind being in the network *is* the goal.
Reminds me of a Seth Godin riff about being on Oprah.
It seems that I have really gotten a lot of crap, especially from Allan, because I have a contrary opinion:
“It’s a terribly short-sighted viewpoint”
“This is NOT the time to stick your head in the sand.”
“For you to naysay…”
“I don’t know how much more I have to spell it out to you”
I thought the purpose of these forums is to have an exhange of ideas, to discuss, to debate. Allan, it looks like you are taking the Bill O’Reilly “agree with me or else” approach. Have fun talking to yourself.
Umm, Bill, having a blog means that I DO have fun talking to myself. However, the comparison to Bill O’Reilly is really quite unkind.
I’m running a more than 2:1 ratio between comments and posts on this blog. A ratio I’m very proud of as it means that people do feel that they can post their comments even IF it gets heated. In fact, this comments thread is nowhere near as “enthusiastic” as other comments threads we’ve had on the blog and will most probably have in the future. The whole point here is to suss out what’s important and what’s not about the quickly changing technologies that are going to forever alter the way nonprofits work. I believe that Causes is one of them. You seem to think that’s not a success because it hasn’t raised enough money in the 28 days of its existence. I think that simply raises the bar way too high and I guess that’s where we’ll just have to disagree.
“You seem to think that’s not a success because it hasn’t raised enough money in the 28 days of its existence…”
Read the thread again. To distill, it has been this:
You: “Causes is a huge success because it has reached 1M users in 27 days”
Me: “That is great but it is still too early to tell how effective a fundraising service it will be”
You: “Your head is in the sand, you are a naysayer, etc”
I have never said it’s NOT a success. What I said is that it is too early to definitively say it IS a success, as you have.
In any case, what you or I think it not important. Time will tell. And if you want to make a bet on what their monthly fundraising totals are in 6/12 months I’d love to hear your thoughts.
Hillarious! Me and a friend were already making bets on $ donation per user. I was saying at least a quarter per user by end of year 1. He wanted to make a fifty cent per user bet.
Does anyone have a good site they’d recommend so we can put a public bet down? I will wager my Google glow-in-the-dark sippy cup (still unsnapped so it can still glow for the eventual winner) or my Google Rubik’s Cube? If I were you, I’d go for the cup — you can post a pic here talking about the sweet nectar of victory et al if you win. This is some serious swag only to be found on Google Developer Days. (of which there were only two)
Is $ per user the kind of metric you’re looking for? I’ll be back in about half an hour — gotta buy dinner for the wife but I’d love to make this bet with you.
OK, Allan, I will put up my eBay “RASCI the decision making Mongoose” (seriously, a plushy toy given out to eBay employees to help decision making) against your Google sippy cup. Whomever gets closest to answering the following wins:
“How much funds will Facebook Causes raise in December 2007″
Once you have arrived at your answer let me know and I will submit mine (before seeing yours).
Ok Bill, I’ve got an e-mail into Project Agape asking whether or not we’ll be able to get a report on the amount of funds raised in December 2007. If Project Agape agrees, then we could go ahead with this.
My preliminary guess is that Causes members will number into the four to five million mark by then, so I’m guessing about $1.5 million? BTW, when you say Facebook Causes you’re not counting whatever Project Agape raises on its own web site? I’m assuming they can break out the numbers .
Project Agape has replied that they will provide December numbers to us for the Causes app. I was also thinking that that we also just donate money to the charity of each other’s choice. This way other people on either side of the divide can join in. Just a $50 donation for either one of us should suffice. I will still hand over the Google cup to you if you win but I think it would be interesting if we could open up the betting. What say you?
OK, Allan, I estimate that Causes will do $150K in volume from Facebook in December.
So, if they do more than $825K, you win; less, I win.
Would be happy to put up $50 for charity in addition to my cherished Rasci the Decision Making Mongoose.
- BT
I agree with Allan on the significance of this, but I don’t agree that Project Agape will necessarily be the clear winner because they are the first mover. Lots of other folks in this space are putting their Facebook apps together, including Change.org and at least 3 others that I know about. There is very high churn on Facebook apps. It is new, people are adding and deleting. A better, more interesting app comes along that plays the same role and it will also grow.
A question I have is about people’s interest in “causes” in the generic sense vs. specific causes. Would a Cancer or Darfur APPLICATION do better for that specific issue than the number of people choosing that cause in a generic causes application?
@Bill – you’re on for the bet. I’ll devote a special sticky post for this…
@Michael: Great questions but I don’t know whether first mover advantage means anything so far. It’s still too soon to tell.
I think the best marketed orgs will do well on Causes. If you had great offline and online marketing campaigns prior to Causes, you’re going to do well. It’s no surprise that newer orgs that have more of that focus are just kicking butt over more established orgs. Take a look at invisiblechildren.org and how it’s got FOUR times as many members as Amnesty. Now THAT’S marketing and the Internet allows people to magnify those strengths easily.
@Allan:I am sorry to add to a dead chain.
@Bill: where did you get the $6bn in online giving statistic from. I am trying to assemble a cursory 2.0 strategy for a community development organization– we work at a larger scale than the “long-tail” might optimally serve, but nonetheless a sense of the market would be really valuable, and overall metrics (how much is given online, how has it grown, what are the most successful locii of online giving) are proving hard to come by…
Thanks for any help you (or anyone)can offer.
No worries, I would suggest looking at this article. It’s clear that online giving is growing quickly but it’s still very small when compared to other revenue streams. The two largest fundraisers that were solely web-based that I’ve seen online is Network for Good raising $100 million and MissionFish at $76 million. People are forecasting another five years before the volume from online giving starts to catch up with other revenue streams, especially direct mail.
alan-
just posted a long response but somehow 6+4=10 was the wrong answer on the filter! anyway: all i said was
(a) thanks, and
(b) do you know of any good examples, besides macarthur found. in 2nd life (i know you’re not sold on 2nd life), of major funding sources entering the social web as way of distributing their money and looking to connect to lendees? i.e., examples that don’t trade on the long-tail/aggregating many small loans model? those seem harder to find.
thanks again. i enjoy the blog and am glad to have discovered it.
Well, I’m definitely the Second Life skeptic and it’s probably impaired my researching Second Life properly. If you want a rosier view of Second Life, you might try beth.typepad.com and lotusmedia.org. Beth and Ruby are more supportive of SL and may have better figures to bolster their case. You might also want to ask the Techsoup people as well.
As for examples that don’t trade on the long tail/aggregating small loans model, I don’t understand what you mean. You’re looking for online microcharities that take in much larger loans or much larger one-time donations?
thanks again; i wasn’t clear. i meant a more complete reversal: rather than models that harness the long-tail and many small donations to GET donations, i’m curious about whether there are examples of big-money entities– foundations, community banking wings of commercial banking, etc.– creating a presence on the social web as a way to meet new partners to whom they may want to GIVE money. as i said, the only example i’ve seen of this is the MacArthur Foundation putting itself out there on SL a little.
so i guess i’m looking for macrocharities who are exploring ways to use web 2.0 to give away large grants effectively, as well as the reverse.
(to give you a little more context): i work for Community Development Financial Institution whose core lines of business are a loan fund and 2 VC funds; all robust by CDFI standards. We’re basically trying to get an overview of the new markets created by the social web and web 2.0 and how organizations like us might operate both as lenders (random e.g.: creating microcharities/new marketplace for retail products that help low-income communities) and recipients (i.e., connecting to the commercial banks and major funding partners who are important investors for us). hence all my leading questions. thanks for your patience and help so far.
you can check us out at http://www.bostoncommunitycapital.org if you’re interested; the website is a little late-90s, but is decently informative.
Aha I see what you mean… The problem is two-fold. First, it is not known if potential donors will be willing to contribute to large projects over $5000. Also, nonprofits have yet to place their larger projects on the web.
You would need a very healthy marketplace in order to do this or foundations such as yours willing to seed that marketplace by funding the larger projects. If bigger players were willing to play on the web level that would be true, very few foundations have taken the necessary plunge though. Over at the startup I’m building, socialmarkets.org, we don’t take that plunge either.
Frankly, it’s a problem as some of the projects we’d like to present to the public are in the hundreds of thousands of dollars. However, the average donation for each user would have to be incredibly high to deal with listings that large. It’s a classic chicken and egg problem though. I’d love for you to advocate for this in the foundation world but I haven’t heard of any foundations wanting to use Web 2.0 sites in this manner.
It’s December 2007 – who won the bet?
December hasn’t ended — so I’m waiting to see this out as well. It’s going to be really interesting. I think what’s interesting to me is that Causes launched a way for people to give donations as gifts. I’m wondering if that’s going to help…
[...] million pageviews were served with this setup. memcached turns out to be a piece of technology that Causes uses for its own servers. I’m quite curious as to why they don’t use an EC2-based infrastructure and why they [...]