Arun Suresh
Principal Engineer at Yahoo!
Yahoo! India
Arun is a Principal Engineer with Yahoo!. He works with the Content Platforms team based out of bangalore. Arun had been a Java/C++ developer for most of his career, but sometime in early 2009, a friend introduced him to Erlang. This resulted in a series of "Aha" moments for Arun arising from modeling programs in terms of light weight processes, using persistent data structures and reactive error handling philosophy professed by Erlang. Needless to say, hes been hooked ever since.
Arun Suresh is Giving the Following Talks
How Erlang powers Yahoo!'s Content Acquisition Platform
Harley is a Content Acquisition Engine used by various web properties at Yahoo!. It is a highly available and fault-tolerant distributed system written in Erlang. Harley was designed from the ground up to be tolerant to node crashes by dynamically reallocating load to live peers. It elegantly handles addition/removal of nodes with minimal ops intervention. Harley uses RabbitMQ both as a distributed work queue and as its inter-node messaging layer for communication with other Erlang and non-Erlang Nodes. We have also recently integrated with "riak_core" to handle our distributed feed allocation logic. Harley has been in production for almost a year now and is used by Yahoo! to acquire content from various providers such as Associated Press, Reuters, The Newspaper Consortium, CNN etc over multiple protocols such a HTTP, (S)FTP, NNTP etc.
This talk will focus on:
1) Our motivation for using Erlang/OTP to build Harley.
2) How we leveraged RabbitMQ.
3) Our distributed feed allocation layer and our experience in integrating with "riak_core".
4) The challenges we faced while integrating and deploying Harley into the standard Yahoo! stack.
This talk will focus on:
1) Our motivation for using Erlang/OTP to build Harley.
2) How we leveraged RabbitMQ.
3) Our distributed feed allocation layer and our experience in integrating with "riak_core".
4) The challenges we faced while integrating and deploying Harley into the standard Yahoo! stack.