【51CTO專訪】你聽說過Pinterest沒？沒有的話你就Out啦。Pinterest是瀑布流圖片展示形式的發(fā)源地，在2010年3月創(chuàng)建。創(chuàng)建之后的9個(gè)月內(nèi)并沒有引起太大的反響，不過從2010年年底開始，忽然發(fā)展的一發(fā)不可收拾，在今年年中的時(shí)候已經(jīng)超越LinkedIn，成為全美第三大社交網(wǎng)站（前兩位分別是Facebook和Twitter）。今年8月，Pinterest將邀請(qǐng)注冊(cè)模式開放為自由注冊(cè)模式，推出移動(dòng)App，準(zhǔn)備發(fā)起下一波用戶入駐的高峰。

在8月的ArchSummit大會(huì)上，來自Pinterest的兩位工程師到場(chǎng)分享了Pinterest的架構(gòu)擴(kuò)展歷史。他們分別是Marty Weiner和Evrhet Milam。Marty在2011年1月加入Pinterest，Evrhet在2011年11月加入。對(duì)于這樣一個(gè)初創(chuàng)企業(yè)而言，工程師加入的時(shí)間不會(huì)很長；但從他們的身上，筆者感受到了他們對(duì)Pinterest強(qiáng)烈的歸屬感與認(rèn)同感。

[[92811]][[92812]]

Marty Weiner & Evrhet Milam

Marty名片上的Title很有意思，叫做Cloud Ninja——云忍者之意。本次對(duì)他倆的采訪，就從這個(gè)Title的含義開始……

以下，S為51CTO編輯，M是Marty，E是Evrhet。

S：哈嘍，Marty，Evrhet，感謝你們接受51CTO的采訪。首先，Marty，你的名片上用了Cloud Ninja這個(gè)稱號(hào)。這個(gè)稱號(hào)很酷！是跟云計(jì)算有關(guān)系么？

M：是的。我們的工作都在云端控制。而且這個(gè)稱號(hào)聽起來很好玩：當(dāng)四處出現(xiàn)各種問題的時(shí)候，你四處跳來跳去，感覺就跟忍者差不多。

S：那么，你懂得架構(gòu)，你懂DevOps，DBA，安全，以及各種為了保持Pinterest穩(wěn)定運(yùn)行的技能。但你一開始是程序員。后面那些技能你是從哪里學(xué)的？

M：邊干活兒邊學(xué)唄。我們基本上都是。我之前做編譯器的，對(duì)Web擴(kuò)展什么的不咋了解。當(dāng)然，我也做過另一種意義上的擴(kuò)展，比如讓Java跑的更快什么的，但那個(gè)領(lǐng)域不同，跟我現(xiàn)在做的方面層級(jí)不同。所以現(xiàn)在我們就是，每個(gè)人在各個(gè)方向都做點(diǎn)，架構(gòu)，DevOps，數(shù)據(jù)庫，產(chǎn)品設(shè)計(jì)……未來我們工程師多了之后應(yīng)該會(huì)細(xì)分一些。Evr，你要不要談?wù)勀阍谧龅氖虑椋?/p>

E：我關(guān)注的領(lǐng)域是用戶參與度的成長，也就是關(guān)心用戶有沒有在用我們的產(chǎn)品，有沒有持續(xù)的使用，這樣。所以我更多關(guān)注功能，關(guān)注讓用戶喜愛產(chǎn)品的點(diǎn)，讓他們能夠持續(xù)的從我們的產(chǎn)品中得到驚喜。

S：所以，偏產(chǎn)品？

E：是的。這包含很多事情，比如改良推薦的質(zhì)量和相關(guān)性，改善界面，讓用戶的homefeed頁面看起來很酷。

M：同時(shí)，每個(gè)人都時(shí)刻關(guān)注網(wǎng)站的狀態(tài)，確保在網(wǎng)站出問題的時(shí)候可以快速的從備份恢復(fù)回來。

S：那你們沒有產(chǎn)品經(jīng)理嘍？

M：目前還沒有“官方”的產(chǎn)品經(jīng)理。不過應(yīng)該很快就有了吧。

S：所有的事情都是工程師在做？

M：哦，我們有一個(gè)創(chuàng)始人，他關(guān)注整個(gè)產(chǎn)品的愿景。我們有設(shè)計(jì)師。有關(guān)注商務(wù)方面的人。我們有關(guān)注國際化的同事。我們會(huì)一起交流，他們告訴我們?cè)撏男┓较蜻M(jìn)行工程。從某種意義上來說，我們有專注于產(chǎn)品經(jīng)理的人。

S：那么，工程團(tuán)隊(duì)是怎樣工作的？怎么分工？

E：我們目前有差不多30位工程師。我們將他們分成很多小團(tuán)隊(duì)——越來越多的小團(tuán)隊(duì)。一開始我們有一個(gè)完整性團(tuán)隊(duì)，一個(gè)增長團(tuán)隊(duì)，然后是一些小團(tuán)隊(duì)，比如Spam團(tuán)隊(duì)，最近有一個(gè)Feed質(zhì)量團(tuán)隊(duì)，還有一個(gè)平臺(tái)團(tuán)隊(duì)——專門負(fù)責(zé)API開發(fā)和合作伙伴、第三方應(yīng)用配合的團(tuán)隊(duì)?；旧暇褪呛芏嚓P(guān)注不同功能和產(chǎn)品的零碎小團(tuán)隊(duì)。哦，我們還有一個(gè)移動(dòng)團(tuán)隊(duì)。

M：然后我們招聘工程師的時(shí)候也會(huì)看，可能只有30%是看他的技術(shù)能力，看他是不是一個(gè)優(yōu)秀的工程師；還有很重要的30%是文化層面的——我們要確保他們能夠融入我們這個(gè)大團(tuán)隊(duì)的文化。他們有激情么？他們有趣么？他們是不是喜歡亂開玩笑？他們是否熱衷于構(gòu)建、創(chuàng)造東西，讓東西跑起來？如果不是的話，如果他們只是無聊的碼農(nóng)，那他們就不適合我們的團(tuán)隊(duì)。

S：你們自己面試？

M：是啊，我們要確保團(tuán)隊(duì)文化的傳承，確保新人能夠匹配并延續(xù)現(xiàn)有的文化。這也是確保生產(chǎn)力很重要的一點(diǎn)。

S：這樣的團(tuán)隊(duì)組織感覺很另類啊。

M：是的。我覺得一般企業(yè)在初創(chuàng)階段都需要設(shè)立這樣的門檻。確保你帶進(jìn)來的人都是有意思的，大家一起構(gòu)建東西。

S：那，團(tuán)隊(duì)里的每個(gè)成員都是啥都懂么？

M：大部分成員對(duì)各方面都有一定的了解。他們不一定深入的了解MySQL，但至少知道一些。團(tuán)隊(duì)里面會(huì)進(jìn)來之前做過MySQL的人，比如運(yùn)維出身的人，他們就可以互相溝通。而這些運(yùn)維出身的伙計(jì)們也需要了解編碼，因?yàn)樵诔鰡栴}的時(shí)候他們需要有能力做一些修補(bǔ)。

S：那你的專長應(yīng)該算是在哪個(gè)領(lǐng)域呢？

M：我目前在Spam團(tuán)隊(duì)，事實(shí)上我們的大部分工作像是救火隊(duì)員，哪兒出了問題就撲過去補(bǔ)救。我這個(gè)團(tuán)隊(duì)在內(nèi)部也被叫做BlackUPS，聽起來很酷吧，但實(shí)際上我們的工作就是當(dāng)出了問題，但又沒有具體的團(tuán)隊(duì)負(fù)責(zé)這個(gè)問題的時(shí)候，我們就沖上去修補(bǔ)。確保各個(gè)方面都正常運(yùn)轉(zhuǎn)，這個(gè)職責(zé)本身就要求我們對(duì)各個(gè)方面都了解。

E：當(dāng)然，我們也有專注于數(shù)據(jù)和推薦這方面的人員，他們相比我們更加專精。他們往往是機(jī)器學(xué)習(xí)或者數(shù)據(jù)分析相關(guān)的博士畢業(yè)的。除了他們之外，剩下的人都是萬精油工程師，從服務(wù)器擴(kuò)展，到開發(fā)新功能，啥都會(huì)。而一般而言我們需要的人也是啥都會(huì)才好。

S：有意思。那么，Marty，你在來Pinterest之前是做JVM的。Evh你呢？

E：我之前在Yelp做。也是一個(gè)萬精油工程師，做一些信用卡安全防護(hù)，數(shù)據(jù)倉庫，地理位置信息處理方面的工作。

S：你們覺得之前的工作和現(xiàn)在在Pinterest的工作有什么不同么？

M：在Pinterest我學(xué)習(xí)到很多在大的規(guī)模下如何做擴(kuò)展的知識(shí)，以及在這種規(guī)模下系統(tǒng)是如何出現(xiàn)問題的。我在這里能夠從比較高的角度觀察整個(gè)系統(tǒng)。我學(xué)習(xí)到如何對(duì)大型系統(tǒng)做工程。以前做JVM雖然也有相同的問題，不過JVM和MySQL集群面臨的問題非常不一樣。在編譯器這個(gè)級(jí)別，你這個(gè)系統(tǒng)只有兩種狀態(tài)，工作，或者不工作。不是1就是0。而在MySQL的情況下，你會(huì)發(fā)現(xiàn)這個(gè)系統(tǒng)里面有很多技術(shù)組件，每個(gè)組件的工作狀態(tài)是95%！所以你編寫東西的時(shí)候，就需要對(duì)這種不確定性，這種bug非常的容忍。有的時(shí)候一些組件壞掉了，不工作了，但整個(gè)系統(tǒng)還在，所以我們的工作之一就是要提升系統(tǒng)的容忍度——在任何不利的環(huán)境下仍然能跑著。當(dāng)然，之前做編譯器方面的經(jīng)驗(yàn)還是幫了我很大的忙，至少我知道怎么去debug常規(guī)的問題，把它們解決掉，讓它們回到工作狀態(tài)。

E：Yelp的情況也差不多吧。都是運(yùn)行一個(gè)用戶量很大的網(wǎng)站。兩個(gè)地方的體驗(yàn)差不多，不過我覺得在Yelp學(xué)到的最重要的事情之一就是知道了如何去擴(kuò)展一個(gè)團(tuán)隊(duì)。你要確保代碼是可維護(hù)的，有單元測(cè)試，有文檔，以確保團(tuán)隊(duì)的其他成員可以繼續(xù)我們的工作。Yelp在這方面做的非常棒，我從這個(gè)過程中學(xué)到很多，比如HR方面的流程，怎么弄來更多的工程師，以及講正確的人放在正確的事上。

M：有一點(diǎn)我覺得很有趣，那就是我們做了很多擴(kuò)展——Pinterest擴(kuò)展的非?？?，以至于很多代碼都不成熟。有些代碼簡(jiǎn)直是不堪入目。如果你不適當(dāng)?shù)姆怕聛?，這個(gè)情況就會(huì)越來越糟。所以我們招來的一些工程師，有專門負(fù)責(zé)穩(wěn)定這個(gè)狀態(tài)的：他們會(huì)重寫大量的代碼，把它們整理干凈了，以準(zhǔn)備好接下來的擴(kuò)展。

S：聽起來不錯(cuò)。那么，確保你有一個(gè)好的團(tuán)隊(duì)是很重要的事情。

M：是的。我們?cè)?月的時(shí)候有5個(gè)工程師，現(xiàn)在有30個(gè)，整個(gè)團(tuán)隊(duì)變得更大了。其實(shí)很多事情跟你的規(guī)模無關(guān)，而是跟你運(yùn)作的多快有關(guān)。當(dāng)你從6個(gè)人變成15個(gè)人，或者15個(gè)到60個(gè)，或者100到150個(gè)，或者任何一次將團(tuán)隊(duì)規(guī)模擴(kuò)充一倍的動(dòng)作，你都需要重新定義所有的事情。我們要確保文化的傳承，我們要確保團(tuán)隊(duì)成員們都緊密的團(tuán)結(jié)在一起。我們要確保他們的代碼不會(huì)發(fā)展的太快，但同時(shí)我們也不想設(shè)置太多的條條框框——那會(huì)減少樂趣，減少“我們?cè)谕瓿梢恍┡１剖虑?rdquo;的感覺。大公司往往有這樣的問題：你想要改變？先找三層領(lǐng)導(dǎo)簽字吧！那么目前我們?nèi)匀惶幱诳焖俚臓顟B(tài)，但還不至于特別松散。保持樂趣很重要。

S：真棒。那么，你們都做了很長時(shí)間的工程師，你們覺得過去這兩三年是不是發(fā)生了很多變化？

M：對(duì)我而言，絕對(duì)是發(fā)展非常快速。

S：你覺得這對(duì)于開發(fā)者們而言是不是有些艱難？

M：怎么說呢，我們招進(jìn)來的、我們認(rèn)為合適的人，他們喜歡快速變化的環(huán)境?？斓沫h(huán)境讓他們感到愉悅。我們這邊有很多人都是從大企業(yè)里出逃的，他們覺得那里太慢，所以想回到快速變化的環(huán)境中來。

S：他們想要快？

M：是的，他們想要在我們這樣快速移動(dòng)的初創(chuàng)企業(yè)中工作。我見過很多人，他們都說“我受夠了那些管理了！我想寫代碼，完成產(chǎn)品，再寫代碼。我想完成一個(gè)產(chǎn)品又一個(gè)產(chǎn)品。Pinterest目前還足夠年輕，提供了這樣的機(jī)會(huì)。所以說，快速是我們的追求。

E：是啊是啊，我加入Pinterest的原因之一就是這個(gè)。這里有非常多的開發(fā)新產(chǎn)品的機(jī)會(huì)?；叵胛覄偧尤隤interest的那兩周，可能是我有生以來最緊張的兩周。各種各樣的問題層出不窮，你跑來跑去的，一刻也不能停。我之前在Yelp的兩年從來沒有過這樣的經(jīng)歷——確實(shí)非常不一樣。你會(huì)緊張，不停地解決問題，但同時(shí)這也帶來極大地滿足感。最后你會(huì)覺得，哦，這真有趣！在Pinterest的人都很好，我們像個(gè)大家庭，一起睡覺，一起出去玩什么的。我很享受這個(gè)過程。

M：我們經(jīng)常做的一件事就是到看板前面看我們的用戶數(shù)量增長的曲線。怎么說呢，無論發(fā)生了什么事，你加班到凌晨3點(diǎn)也好，或者隔一兩周來個(gè)通宵也好，只要一看到那條曲線，你就會(huì)覺得：這是值得的，這是值得的！而且，這個(gè)產(chǎn)品本身就是個(gè)非常有趣的產(chǎn)品。Pinterest很美——我們有非常好的設(shè)計(jì)師，有好的愿景，而且它非常簡(jiǎn)單。有很多喜愛Pinterest的人，我知道的，我喜歡上Twitter看所有有關(guān)Pinterest的推。“我愛死Pinterest了。”或者是那些經(jīng)過了一天緊張工作的人們說，“好吧，緊張的一天，但至少我還有Pinterest~”看到這些的時(shí)候，我真的有很大的滿足感。他們的輕松緩解了我的壓力，因?yàn)槲抑?，我的壓力為很多人帶來了輕松！

S：太棒了。那么，最后一個(gè)問題。你們的分享提到了為什么Pinterest做擴(kuò)展的時(shí)候選擇了分片而非集群。那么，這個(gè)決策是如何定的？整個(gè)流程是怎么樣的，是你們做做調(diào)查比較一下就下決定，還是要做測(cè)試、數(shù)據(jù)分析什么的？

M：這個(gè)啊，一開始當(dāng)我們只有2、3個(gè)人的時(shí)候，很多情況下其實(shí)就是“好吧，我們希望這能行”，就去做了。我們當(dāng)時(shí)沒有足夠的條件去測(cè)試，一一篩選那些不合適的技術(shù)。而當(dāng)我們知道某項(xiàng)方案可行的時(shí)候，我們會(huì)做一個(gè)拆分，選擇我們認(rèn)為可能給我們最佳結(jié)果的方式去做。最后它成功運(yùn)行，其實(shí)是有一些運(yùn)氣的成分。當(dāng)然，現(xiàn)在我們有能力做一些測(cè)試了，不過更多的情況下我們還是直接構(gòu)建了系統(tǒng)，推送到部分的服務(wù)器上，看是不是會(huì)出現(xiàn)什么問題，再做進(jìn)一步動(dòng)作。比如我們的關(guān)注服務(wù)，一開始會(huì)推送給10%的用戶，看看運(yùn)行的情況，發(fā)現(xiàn)有些問題，解決掉它們。然后再推送到20%，好吧又出現(xiàn)了新問題，那么解決它們，再前進(jìn)。所以，就這樣直接在生產(chǎn)環(huán)境中進(jìn)行測(cè)試，直到我們推送到100%，這就完成啦?？刂破饋聿⒉焕щy。你可以推送模擬的負(fù)載來測(cè)試系統(tǒng)，觀察系統(tǒng)是如何反應(yīng)的，但是模擬負(fù)載與真實(shí)的負(fù)載從來都不同。從來都不同。

E：同時(shí)，這也取決于你的團(tuán)隊(duì)氛圍。當(dāng)有一些新的想法的時(shí)候，工程師們會(huì)討論它們。我們對(duì)這新想法感到興奮么？好的想法往往能引發(fā)更多的討論，工程師們也會(huì)開始嘗試將新想法帶進(jìn)來，測(cè)試，部署?？傊?，是工程師對(duì)新想法的興奮感，這才能為我們遇到的問題帶來好的解決方案。

S：有領(lǐng)導(dǎo)者嗎？

M：我們有一個(gè)Head of Engineers。然后，每個(gè)小團(tuán)隊(duì)都有一個(gè)小組長，不過小組長主要的工作不是管理，而是帶領(lǐng)團(tuán)隊(duì)一起編碼。當(dāng)然，還有我們的創(chuàng)始人。兩個(gè)創(chuàng)始人默認(rèn)在所有人之上的。

S：好的。那么，十分感謝Marty和Evrhet接受我們的采訪！

第二頁是英文采訪實(shí)錄。

#p# 

s: hi Marty, Evrhet, thanks for taking our interview!

You call yourself a cloud ninja. Does this have anything to do with cloud computing?

M: yes. When we do our work, the control system is all on the cloud. And it just sounds funny. When there are lots of problems going on there, you have to be a little bit ninja.

S: you know architect, devops, dba, security, and everything else to keep pinterest running. And you started off as a coder. When did you learn all these things?

M: on the job. both of us. i was a compiler guy, and i didn't do scaling for web. i did big scaling on the other side of the game, like making java run faster, but that's on the other side. i didn't do things quite on that level. so that's the way we are going. and we do a little bit of everything, architect, devops, db guy, product feature...eventually we get segment as we get more engineers. i do spam. Evr, how about you talk what you do?

e: i'm focused on engagement growth, so trying to make sure more users are using our product, and continue to use it, day by day. so just features, making people enjoy the product, help them find contents they are interested in.

s: so a lot on the product side?

e: mine is a lot more product focused, so we do things to make sure you have good recommendations, they are pretty to look at, make users happy when seeing their homefeed.

m: every eyeball still keeps an eye on the website, in case it breaks, get a backup and get it back running.

s: do you have a product manager?

m: not yet. no official individual product manager yet, but i think we'll get one.

s: so just the engineers doing everything?

m: oh we have a founder who is focused on the vision. we have some great designers. and people focused on the business. we have people focused on internalization, and help us understand what we need to engineer that. so we have product management in implicit way, but not a specific product manager yet.

s: so how is the engineering team in pinterest like? how do you distribute work?

e: i think we have over 30 engineers. and we are splitting them into more and more teams. we have originally an integrity team, and a growth team, and then we had even smaller teams, like the spam team, recently a team related to feed quality, and we have a platform team, that just worries about the apis, and making the platform available for partners and third-party apps. so we just have a lot of small teams flying around different features and products we have. and we have a mobile team, too.

m: and in terms of the type of people that we look for, 30% of our interview is on technical aspects, to make sure they are good engineers, but a 30% is on cultural - how good they can fit into our culture. are they motivated, are they fun, and they like to joke around. do they want to build things and make them run, and create. if they are not like that, if they can only write code but they are not fun, we won't hire them.

s: so you interviewed them?

m: yes, we all interview the new people coming in, that's the culture we want going forward, so that to make sure people coming in match the current culture, in the way that you like, i think it's productive.

s: that's quite unique.

m: yes, i think that barrier is common at the early stage of companies. make sure the people bring in a having lots of fun. build things.

s: so do all the members of the teams know everything?

m: most people know a little bit about every piece, but they may not know, day to day, about mysql, we are a little bit segmented. but a lot of people we hire have just done different pieces, some of them know well about mysql if that's what they previously did. but people who know mysql are most probably the ops guys, and they need to know code, need to know how to fix things when it breaks.

s: are you regarded as specialized in certain fields, then?

m: i'm working on spam right now, we tend to jump on whatever needs to be fixed now and then. i'm actually on the team that we call "blackups", it sounds cool, but what it does is if there is something broken and there is not a specific team to fix it, we'll go and fix it. we are the people that make sure everything is fluid and working. so this role requires us to know a little bit about everything.

e: we do have some people who are focused on data and recommendations, they are a bit more specialized than the rest of us. they have knowledge of phds, on machine learning or things like that, or analytical knowledge. most of us are like generalist engineers, and can come to deal with scaling servers, add a feature, and things like that. usually we need people who can jump onto anything.

s: that's interesting. so marty, before you joined pinterest you were doing jvm. how about eve?

e: i was at yelp. a generalist engineer that was doing credit cards protection, data warehousing, and locations.

s: so what difference do you think there is between working in pinterest and working in your previous post?

m: i think i really learned how to scale things on a larger scale, how things break down, i get to see things from a top level. i really learned how to engineer very large systems. it's the same problem, but also a very different problem from scaling mysql. from a compiler level, your system(?) either work or doesn't work, it's 1 or 0. for mysql, you find yourself surrounded by technology that works 95%. you have to write things that are very tolerant to that, tolerant to the bugs. some of them don't work, some of them break away, so part of our job is to make things tolerant - make them work no matter what is happening. i think the compiler work helped a lot for me, just for debugging regular problems, beating them down to the ground to make them work.

e: yelp does pretty much the same thing, running a website that has a lot of users. so i have a lot of similar experience that i have now. but i think yelp has a lot of knowledge in scaling a team. making sure the code is maintainable, has unit tests, is documented, so that the rest of the team can carry on with the code. yelp had a good job in doing that, and i learned a lot of these processes, like hr processes, getting more engineers, getting the right people for the job.

m: one thing i think is interesting is that, it is just two of us for wild scaling, when pinterest is scaling fast, a lot of code is not matured. some code there is just really nasty, and if you don't slow down a bit, it will just become worse and worse. so some engineers we brought in, their work is to stablize, so they come in and rework a lot of the codes to make them cleaner, and more scalable for later.

s: cool. so the important thing is to get the right team.

m: yes. in january we had 5 engineers, and now we have 30. so we are getting bigger in size. it actually doesn't matter what scale you are at, but about how fast things move. so when we moved from 6 to 15 engineers, and 15 to 60, or 100 to 150, or any doubling the size, you have to rework everything that you used to. so we have to make sure the culture matches, and we have to make sure we have properly related our team. we have to make sure their coding doesn't fly too fast, but at the same time we don't want to place too many rules around the place, that will slow down the fun, slow down the feeling that you are getting something done. bigger companies have this problem: i want to make a change, i have to go three levels for approval. right now we are still fast, but not super loose. keep it fun.

s: cool. so you have been engineers for quite a long time, do you think things have changed a lot in these 2-3 years?

m: it certainly is moving fast for me.

s: and do you think this has made things more difficult for developers?

m: i think we get the right people who likes things to be fast moving. it makes things better for them. we get a lot of people who come from big big companies that have slowed down. and they want to come back to the fast.

s: so they want the fast changes?

m: yes. they want to be in small startups that move fast. we've got a lot of people who say "i'm sick of all the management. i want to make code and move. i want to make another product and move. pinterest is still young enough so they can get that opportunity. so fast is what we thrive on.

e: oh yes, that's one of the reasons i joined pinterest. there are so many opportunities to work on new products. Right after I joined, it was probably the most stressful two weeks of my life. Thing were breaking all the time, you keep going back to a fixing mode - i've never experienced that before in the two years at yelp. it's really different. it can be stressful, keep solving problems, but it is really satisfying, very very fun at the end of the day. so in pinterest we have all the right people, like families, sleep and hangout together, i really enjoyed it.

m: one thing is that we kept watching that graph of growing number of users that keeps rising. so no matter what happens, whether we worked up till 3am or even all night every week or the other week, as long as you saw that graph, it's like, oh, it's worth it, it's worth it~ and also it's a fun product to build. it's a really beautiful product, we get really good designers, visionaries, and it's really simple. a lot of people had so much fun with it, and i like going on to twitter and reading all these people saying "i love pinterest". a lot of people would go in the end of stressful days and say, oh at least i've got my pinterest and my key. reading those is really satisfying. you know a lot of people is relaxing, you know, my stress! at least my stress is going to relax somebody:)

s: cool. so last question. your session mentioned choosing sharding over clustering. so who make those decisions? what's the process like? do you just go ahead and decide, or do some data, testing before you went for it?

m: so early on, when there were only 2 or 3 of us, it was pretty much like "we hope this will work", because we just didn't have the room to test and bring up a lot of technologies that didn't work. and we knew what was working, so we sort of extract it, and hope for the best thing we got. and in a way we got lucky as it turned out to be working. now we do have the capacity to do a bit of testing, but we are still kind of build the system and start push it to a little bit of entity(?) and see how it does, and if there are problems, fix them. For the follower service we started off pushing 10% and see, ok we've found a few problems, fix those. push 20%, ok a few more problems, and so on. so we have this bring things to production, but just push some things to it. and when it's 100%, you are done. so in a way we are kind of test in production, but in control. and you can always push loads to the system, you can see how the system responds, but it never ever matches the real load.

e: also i think it's up to the team you have. when new ideas come up, engineers are going to talk about it. are they excited about it? good ideas tend to have lots of chats, and people will start bringing them in. test them. deploy them. they have to be excited about the new idea - good solutions to the problems we have.

s: so do you have some kind of a leader?

m: we have a head of engineers. and then, every team has a team leader who is very down into the code - he is not managing, just code along and lead the teams. and we have a founder, two founders who are automatically on top of everything.