Comments on: Twitter needs to get creative http://snowedin.net/blog/2008/05/22/twitter-needs-to-get-creative/ Thu, 11 Mar 2010 21:35:50 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: erik http://snowedin.net/blog/2008/05/22/twitter-needs-to-get-creative/comment-page-1/#comment-862 erik Sun, 25 May 2008 07:23:28 +0000 http://snowedin.net/blog/2008/05/22/twitter-needs-to-get-creative/#comment-862 Leif, I don't buy it. Saving items in a "sent items" store on the user's local server should be trivial. Almost all email applications do this. And the friends list can be stored locally too. There's no need to compile friends lists across different users (i.e., who are the friends of BOTH Joe and Bob?) so that's a trivial lookup. You're right that the number of recipients is sometimes large (tens of thousands) but only a few users have that many followers. What matters for scalability is the average number of recipients, which is probably pretty reasonably (~50 or so). I just don't buy the "oh, it's so complicated" argument. It's not complicated. They just chose a shitty architecture to begin with because they didn't think it through, and now they're having a hard time because they have to do the transition while under a huge load and whilst trying to maintain uptime. Not that I don't sympathize with them... architecture is hard. I'm just pointing out that the architecture they need isn't some crazy new thing. It's been around a long time, they just didn't realize they needed it. Erik Leif,

I don’t buy it. Saving items in a “sent items” store on the user’s local server should be trivial. Almost all email applications do this. And the friends list can be stored locally too. There’s no need to compile friends lists across different users (i.e., who are the friends of BOTH Joe and Bob?) so that’s a trivial lookup.

You’re right that the number of recipients is sometimes large (tens of thousands) but only a few users have that many followers. What matters for scalability is the average number of recipients, which is probably pretty reasonably (~50 or so).

I just don’t buy the “oh, it’s so complicated” argument. It’s not complicated. They just chose a shitty architecture to begin with because they didn’t think it through, and now they’re having a hard time because they have to do the transition while under a huge load and whilst trying to maintain uptime.

Not that I don’t sympathize with them… architecture is hard. I’m just pointing out that the architecture they need isn’t some crazy new thing. It’s been around a long time, they just didn’t realize they needed it.

Erik

]]> By: Leif http://snowedin.net/blog/2008/05/22/twitter-needs-to-get-creative/comment-page-1/#comment-863 Leif Sat, 24 May 2008 06:07:03 +0000 http://snowedin.net/blog/2008/05/22/twitter-needs-to-get-creative/#comment-863 I think there are some pretty big differences between Twitter and SMTP, most of which have to do with recalling or modifying system state. For instance, an SMTP message, once delivered to a peer, can be forgotten about completely by the sender. Twitter, on the other hand, wants to remember every single message that gets sent by every user for all time, I suppose so that people can check the web site at any point in the future and see what someone sent at any point in the past. SMTP would croak, too, if it had to handle this. A second difference having to do with system state is recalling the sometimes-large number of friends/recipients that messages need to be relayed to. With SMTP, the recipients of each message are literally written within the message itself : The onus is on the message writer, not the system, to remember whom to send a message to. Twitter takes this onus from the user and places it on its database. So I'm not really convinced you can replace all of the features of Twitter with something like SMTP, at least not without a lot of person-hours. But with just one or two people at Twitter working on infrastructure, such tasks are probably pie-in-the-sky compared with just keeping the database from catching on fire. I think there are some pretty big differences between Twitter and SMTP, most of which have to do with recalling or modifying system state.

For instance, an SMTP message, once delivered to a peer, can be forgotten about completely by the sender. Twitter, on the other hand, wants to remember every single message that gets sent by every user for all time, I suppose so that people can check the web site at any point in the future and see what someone sent at any point in the past. SMTP would croak, too, if it had to handle this.

A second difference having to do with system state is recalling the sometimes-large number of friends/recipients that messages need to be relayed to. With SMTP, the recipients of each message are literally written within the message itself : The onus is on the message writer, not the system, to remember whom to send a message to. Twitter takes this onus from the user and places it on its database.

So I’m not really convinced you can replace all of the features of Twitter with something like SMTP, at least not without a lot of person-hours. But with just one or two people at Twitter working on infrastructure, such tasks are probably pie-in-the-sky compared with just keeping the database from catching on fire.

]]>