Saturday, July 10, 2010

Everyone has problems : Twitter and Reddit

For those of you out there that just expect everything to work on the internet,   here is a great internal view to some of the hard challenges that face some of the largest sites on the internet. 

Twitter listed its challenges as of late:
“Twitter has experienced several incidences of poor site performance and a high number of errors due to one of our internal sub-networks being over-capacity.
We're working hard to address the core issues causing these problems --more on that below-- but in the interests of the open exchange of information, wanted to pull back the curtain and give you deeper insight into what happened and how we're working to address this week's poor site performance.
What happened?
In brief, we made three mistakes:
* We put two critical, fast-growing, high-bandwith components on the same segment of our internal network.
* Our internal network wasn't appropriately being monitored.
* Our internal network was temporarily misconfigured.
What we're doing to fix it
* We've doubled the capacity of our internal network.
* We're improving the monitoring of our internal network.
* We're rebalancing the traffic on our internal network to redistribute the load.”

Reddit has also had some huge challenges with traffic:
“We've been kinda bummed at reddit these days. It seems like every week something comes up that slows performance to a crawl or even leads to a total site outage. And we almost never get a chance to release new features anymore.
Our four engineers -- KeyserSosa, jedberg, ketralnis, and myself -- are working full time (plus many evenings and weekends and sometimes even the middle of the night) just to keep things going. Perhaps we're doing it wrong: there might be ways to optimize our code, or technologies that could allow us to work more efficiently, but we're too busy to investigate these things, or to migrate to the ones that look promising. It becomes a vicious cycle.
The bottom line is, we need more resources.”

No comments:

Post a Comment