The unsolved scaling challenge

Everyone knows that scaling websites so that they can serve millions of users a day is hard. Most Twitter users have seen the famous Fail Whale often enough, which usually shows up because some component of Twitter's website isn't handling an increase in traffic so well. Many people familiar with web services can also appreciate the additional challenge that comes with providing an API, especially when, as is the case of Twitter, it has to handle 10 times the traffic of the website.

What people often miss is the huge amount of non-technical work that is required to keep the consumers of a website happy as the user base grows into the tens of millions. If you're charging users you can factor support into the cost and grow your support team to ensure that every user can talk to someone when they need to. Many services don't have that luxury, but are able to assign teams of smart people to building knowledge bases that can answer almost all user questions via their websites. So what happens when, like Twitter, more than 90% of your traffic isn't coming through the website but is coming via over 50,000 applications using your API? You end up with a relatively small group of developers that need highly specialist support to ensure that they in turn can provide their own users with a reasonable level of support. You could charge that group to ensure that you can build a skilled developer support team to answer every request in person. However, what do you do if, like Twitter, you are aware that this developer community is adding a huge amount of value to your ecosystem and you don't want to slow its growth?

The question of how to scale a developer community like Twitter's is one I've been asking myself for over a year now. It's led to me exploring various ideas and meeting a host of interesting people. I started 2009 by organising the world's first events for Twitter developers, the Twitter Developer Nest, of which there have since been six in London and one in New York, involving over 350 members of the Twitter developer community. I spent the majority of the year working with startups and agencies integrating with Twitter, experimenting with new features as they have been rolled out and building a Twitter based service of my own. Towards the end of the year I had the opportunity to spend a day talking with several members of Twitter's Platform team as well as Ev, Biz and Dick.

As a result of my experience I've built a comprehensive understanding of the inner workings of Twitter's ecosystem and developed a wealth of material on the subject that I'd like to share through a series of blog posts. Topics I'll cover include:

While I'm under no illusion that there will ever be a simple solution for solving the challenge of scaling a developer community, there are two things I'd like to share with you now which I believe are particularly important. Both platform providers and members of developer communities need to be continually asking themselves if they are being as open and kind as they can be. The success of Twitter's Platform to date can be traced back to the openness of Twitter's early API and the kindness of early 3rd developers, but there is much more that can be done as the community grows.

Be Open

If you are a platform provider like Twitter there is always an opportunity to be more open with your developer community. A group of smart people have invested a massive amount of their time and often money into your product without any kind of service level agreement. Sometimes those developers can seem like they don't appreciate your work, more often than not it's because they don't understand what it is that you do. The more information you can provide them with, the greater their ability to assess the risks they have exposed themselves to and appreciate the value that you add. Once they have a better idea of where they stand they will be far more likely to take the added risks of working with new features and innovating on top of your platform.

Every time a developer is surprised by something you announce publicly, they'll lose a little bit of confidence in their ability to work with your platform while staying on top of the risks. If your developer community is a key competitive advantage it's not worth damaging it as a whole in order to make short term gains in the press and with a handful of partners. Special relationships with certain partners will inevitably exist as is always the case in business. By making all developers aware of steps required to become a special partner you can guide them down a path that can help you meet your strategic goals.

Be Kind

If you are a developer working with a Twitter-like platform there is always an opportunity to be kinder to the providers of the platform and other members of the community. You're getting something for free and the individuals that keep it working are human beings just like you. Sometimes they can get overwhelmed to the point of not being able to provide you with human responses. Be patient and encouraging while giving constructive feedback. You'll gain far more from taking time to build relationships with platform providers and working together to tackle problems than you will from publicly attacking them. More often than not other members of the community can be of more help than providers are able to be. By reciprocating this help in whatever way you can you can help build the community into a more rewarding environment for everyone.

These two things are surprisingly easy to take for granted while being critical to the success of a developer community. As such I am making 'Open Kind' the mantra of this series of blog posts. Of every option we have along the journey of scaling a developer community we should ask "is it open kind?", "are we being open kind?" and "can others use it to be open kind?".

What are your thoughts on this unsolved challenge? My next post will focus on the the things that set developer communities like Twitter's apart from more traditional developer communities, please let me know if you feel there are any particular aspects I should focus on. I'm looking forward to discussing this whole issue further with people in person over the course of this year, please do say hello if you spot that our paths cross. I'll be at SXSWi in March. In April I'll be attending Chirp, The Official Twitter Developer Conference in San Francisco followed by speaking at #140conf in New York. In May I'm organising WarbleCamp, The Twitter Developer Unconference and throughout the year I'll continue to participate in The Twitter Developer Nest (DevNest)