Max Schireson's blog

An employee at my company (10gen, the company behind MongoDB) sent me this link to a video of Eric Ries talking about the “5 whys” and how technology problems are often people problems.

For those of you not familiar with the notion of “5 whys”, the idea is to ask why 5 times until you get to the root cause. While I think asking why and finding root cause can be important, a few issues jump out at me: first, why do you need to ask exactly 5 times?  Second and perhaps more importantly, when there are multiple answers to a given why, which one do you follow for further investigation.

In the example Eric gives, they start with a server crash and end with a manager who doesn’t believe in training. I would argue this is an attempt to reduce technical management to something that general managers reading Harvard Business Review can do, which is dangerous.

What really needs to be asked? The first why is straightforward.

1. Q: Why did the server crash?

A: There was a bad call to some new API

Now what’s the root cause? Is the problem with the API or the program that called it? Is it a problem of programmer competence, communication, or training? Or is it an architecture problem which is actually bigger than this specific API usage? Was the problem in design, implementation, testing, or documentation? Was it a process problem, a training problem, or a hiring/management problem?  I don’t think 3 or 4 more whys will mechanically answer that. One or two more whys might answer it, but the key is focusing on the right areas.

What technical manager at a startup really thinks that training will prevent a developer from calling an internally developed API incorrectly in a way that causes a crash? Its not impossible, but I think pretty unlikely. Better API docs could help, and better communications could definitely help. Or a more robust API, or more thought about whether that API needs to be public if it can’t be made more robust, or broader test coverage…

I think a better policy might be “2 or 3 whys, but the right ones”. I don’t think it will catch on as a management slogan, but I think it is more likely to yield useful answers.

– Max

A friend of mine sent me a link to an article by Danny Hillis on elections. Danny is a smart guy who has done good work in many areas, but his analysis struck me as a) radically oversimplified and b) wrong in some important ways. But it got me thinking about how our changing understanding of human rationality (or lack thereof) in economics might spill over into political science. Let me start with how human behavior differs from classical economic models and move from there to how elections are effected.

Traditionally, economics was based on the notion of rational actors. Each individual makes decisions to maximize their utility functions (think about  a modern version of quantifying the notion of “greatest good” in Utilitarianism). Person A might value summers off more highly than a chance at millions of dollars and therefore go into teaching. Person B might dream of having their own private jet and be willing to work nights and weekends to keep even a small hope of their dream alive. But all of these are rational decisions based on utility functions which describe how much each economic actor values each benefit.

It seems reasonable (to a quantitative person such as myself) to think that human economic behavior works this way and the mysteries of the different choices people make is encapsulated in this unknown and in all practical respects unknowable utility function. Unfortunately for those of us who like simple mathematical explanations of human behavior, people don’t actually behave this way. In 1953 Maurice Allais wrote a paper (warning: it is mostly in French, 44 pages, and costs $10 to download) describing a paradox which showed that even if you look at simple financial wealth, people do not rationally follow a utility function. His paradox attacked the notion that if some percentage of the time (say 98%) two options produce the same outcome, we should base our choice between the options on the 2% of the time when they produce a different outcome (in economics-geek-speak, this is the Independence axiom in the Von Neumann-Morgenstern utility theorem). It seems rational (and official, now that it is part of a theorem), but in fact we care very much what happens in the 98% – we do not ignore the identical part of the two scenarios in evaluating them, irrational as that may seem.

Detailed example:

In scenario 1a, you receive $1 million 100% of the time. In scenario 1b, you take a 1% risk of losing your $1 million prize to gain a 1% chance of increasing its value to $3 million. 1% chance of nothing, 98% chance of $1 million, 1% chance of $3 million.

In scenario 2a, you have an 2% chance of receiving $1 million, in the other 98% you receive nothing. In scenario 2b, again you take a 1% chance of losing your $1 million prize to gain a 1% chance of increasing its value to $3 million. 1% chance of 3 million, 98% chance of nothing.

Some people (a reasonable number I would guess) would choose 1a over 1b but 2b over 2a. Mathematically speaking, both decisions are identical: you are giving up a 1% chance at $1 million to increase a $1 million prize to $3 million 1% of the time. The other 98% doesn’t matter. But psychologically, it matters a lot. When you are guaranteed $1 million dollars, risking it sounds irresponsible. And if that 1% chance happens, you will live forever with the knowledge that your greed to get $3 million cost you $1 million. In the other situation, if you don’t get anything, you aren’t surprised and you are fairly confident that it had nothing to do with your decision to increase the odds of getting nothing from 98% to 99%.

Now, on to politics.

Hillis lays out a simple left-right political spectrum. He puts two candidates on it and models voting behavior as people voting for whichever candidate is closer to their beliefs. From this, we can deduce all sorts of things, some of which – like elections being close – sometimes model reality pretty well. Then again, Herbert Hoover never participated win or lose in a close presidential election, and how many electoral votes did Walter Mondale get in 1984? 13. Which is 5 more electoral votes than Alf Landon received in 1936!

The problem is that elections are much more complicated than Hillis’s model. Lets put aside for a second that there are multiple issues involved so it is not a simple one-dimensional spectrum. Lets assume there are only two candidates in the election. Lets also put aside candidates personal charm or lack thereof and their potentially scandalous sexual, military, or substance consumption histories. This model is an still an oversimplification in two critical respects:

1. Voters do not have a simple choice between voting for the Democrat and voting for the Republican. They can stay home. Or they can volunteer or donate money or attempt to harangue their friends into voting for the candidate of their choice. Imagine an (exaggerated, to make a point) election where:

a) 60% of the electorate prefers Smith to Jones, but only by the narrowest of margins; they are generally dissatisfied with both candidates

b) 40% of the electorate strongly prefers Jones to Smith – in fact they thing Jones is close to their ideal candidate and Smith is so nightmarish a candidate they would consider leaving the country if he were elected

What happens in that election? Maybe a 48% turnout; 36% of population (90% of the Jones supporters) vote for Jones and 12% of the population (20% of the Smith supporters) vote for Smith. Jones wins in a landslide, with 75% of the vote. Smith, despite a 20 point margin in polls of registered voters, can’t even carry his home state.

Is this exaggerated? Yes, but intensity matters.

2. Just as the 98% that was unchanged effected the outcome in Allais’s paradox, candidates not on the ballot effect voters. Again, here’s a scenario:

The Yellow party and the Purple party each have 50% support. Both of them nominate a candidate who is free of scandal but not particularly personally charming. Each candidate represents the mainstream of their party – put them just slightly to the center of the middle of their party’s 50%, at say the 30th and 70th percentile respectively.

So far, it sounds like a close election.

However, in the Yellow Party primary Mr. Gold had to duke it out with Mr. Canary. Canary was just a few points further from center than Gold, so the primary was very close, and neither party had a majority of committed delegates at the convention. Mr. Gold did have a slight lead over Mr. Canary, so when the super delegates broke strongly for Gold (whom they felt was more electable) they felt they weren’t overturning the will of the people. But the Canary camp felt like if you included the Canary, Maize and Buff delegates (who endorsed Canary when they dropped out, but such endorsements aren’t binding on their delegates so they were counted separately) against the Gold and Ecru delegates (Ecru having thrown his support to Gold), Canary held a lead before the super delegates voted.

The Purple party primary, however, was a different story. Mrs. Plum was the only mainstream candidate; Mrs. Indigo and Mrs. Magenta were really fringe candidates who generated very little support. The party became unified behind Mrs. Plum in February.

How does this election turn out? Not close at all. About half the Yellow party voters are profoundly upset with the Gold campaign and their party. Many of those voters don’t have an ideological problem with Gold per se, but they are so disappointed with how the process played out for Canary (who wasn’t picked as VP) that they are not at all motivated to vote. Despite being weary from the primary, the half of the Yellow party that supported Gold turns out 60% of their voters giving gold 30 million votes, but only 40% of the Canary voters turn out at the general election. Those that do vote reluctantly for Gold, giving him another 20 million votes for a total of 50 million. Meanwhile, 60% of the Purple party voters show up, giving Mrs Plum 60 million votes and a 10 million vote margin.

If that seems farfetched, I can introduce you to some Hilary Clinton supporters who are strongly pro-choice, support gay marriage, and voted for John McCain.

Now, I constructed both of these scenarios while leaving in place many oversimplifications. Overlay multiple issues of varying intensity and varying degrees of organized lobbying with ethnic, religious, and gender identification and elections become very complex things. Certainly the practical modeling of individual factors is quite advanced; I am curious what the intellectual underpinnings are, and to what extent they have embraced the latest thinking in human economic behavior.

– Max

A few weeks ago I posted a puzzle about two mathematicians guessing the results of coin tosses. Dwight asked me about generalizations to more than two mathematicians. While the problem may seem very different on the surface, in my opinion the underlying issue is actually quite similar and the changes are necessary to generalize from two mathematicians to N.

I posted one version a while ago here which I will repeat as a warmup for the harder version

16 mathematicians are in a room. They are each assigned a hat, either black or white. Each hat is assigned totally independently of all the other hats and has a 50% chance of being either color. Each mathematician can see everyone else’s hat but not his/her own hat. The mathematicians all have to independently and simultaneously guess the color of their own hat. They have an hour before the hats are assigned to make a plan, then one minute to view the hats, then they each go into a voting booth to vote for the color of their own hat. While they are viewing the hats they can not communicate or signal in any manner to the other mathematicians.

The success or failure of the mathematicians is judged as a team: if every single one guesses their own hat color correctly, the team wins. If even a single mathematician guesses incorrectly, the team loses. What are the odds of their success, and what strategy should they employ to achieve it? [Hint: the odds are much better than you might first think they are. Really. If you are sure you can't improve them I am happy to find a jurisdiction where we can play this for high stakes!]

Second version (if you thought the first version was too easy):

Everything is just like the first version, except that when the mathematicians enter the voting booth they can vote “White”, “Black”, or “Don’t know”. If anyone is wrong, they all lose. If everyone passes, they all lose. If at least one mathematician chooses a color and all mathematicians who choose colors are correct, they win. Again, what are the odds of their success, and what strategy should they employ to achieve it? Again, you can do better than you might first think!

– Max

16 mathematicians are imprisoned by the evil emperor. They are told that in the morning they will each be assigned a hat, either white or black. Each hat is assigned randomly (50/50) and independently. Each mathematician will be able to see everyone else’s hat but not their own. After one minute to observe the hats, they will simultaneously be sent back to their private cells where they have to guess the color of their own hat. If a single mathematician guesses incorrectly, they will all be executed. If all 16 guess correctly, they go free.

What is the probability that the mathematicians will be set free, and how should they guess?

If you’d like a hint, look at this problem. Thanks to Dwight for suggesting more complex problems in that spirit; if this isn’t hard enough I’ll add another twist next week.

– Max

The king is very angry at his two best mathematicians, but rather than executing them immediately institutes the following process:

They are banished to opposite ends of the kingdom, kept under 24 hour guard. Every day  in both Westerville and Eastgate where the mathematicians are being kept one of the guards flips a coin and records the result. Each mathematician must then guess the result of the flip at the other end of the kingdom. A messenger is then dispatched from each location to the royal palace to share the results of the flip and their mathematician’s guess. As long as one of the two mathematicians guesses right, the lives of both mathematicians are spared until the next day. The first day that both mathematicians are wrong, they are both executed.

The king imagined this game would not last more than a week or so. The mathematicians, however, hatched a clever plan as they were being ushered out the door of the castle.

What do the mathematicians do to live as long a life as they can, and on average how long should the king have to wait to execute them?

– Max

I just got an email from a friend describing the interview offer process of a mutual friend joining a technology startup. The newly minted employee has an offer but they won’t tell him how many shares are outstanding. He was told what percentage of the “employee pool” he was getting, but not the overall percentage. What exactly is included in the “employee pool” is a question – is it just unissued options, is it akk issued and unissued options, is it all common shares including founders???

As I’ve said before when talking about stock options (see this post), it is impossible to evaluate your grant without knowing how many total shares are outstanding. Proof by analogy: how much would you pay for partial ownership of my house? I can create an “investor pool” which represents some ownership of my house. For $50,000, you can own 10% of the investor pool. Sorry, but I am unable to disclose how much of the ownership of my house will be assigned to the investor pool. Please contact me if you are interested! I don’t expect to sell much of my house this way; why should I expect employees to be excited about options offered in the same manner?

Not only does the lack of data make it hard to evaluate the stock grant, but the lack of transparency is concerning as well. One of the things that makes startups great is a team working together towards a common goal. For me, a big part of creating that is giving the team a lot of transparency about the company they are building. I view the employees in an early stage company as stakeholders, and as investors in a sense: they are investing their time, their passion, and their reputation in the company. To not share basic information, especially at an early stage, demeans that investment and undermines the sense of team.

Will my friend still join? Maybe. There’s a lot to like about the opportunity. But the lack of data and the lack of transparency is a red flag. At a minimum, it’s caused an extra round of digging/assessment. As an employer, why would you want to give a potential recruit a reason to question your offer?

– Max

The industry has a tendency to attach everything to the latest hot trend/buzzword, but I just can’t accept Autonomy and Endeca as “big data” vendors.

Autonomy is a search engine vendor that added various content technologies like Zantaz (email archiving) and Interwoven (web content management). Yes, it has “unstructured” in common with big data, but that’s about it.

Endeca started as an ecommerce search engine with aspirations of general-purpose BI tool. So it has “analytics” in it but again, I don’t see it as big data.

I’m not saying that either Autonomy or Endeca is a bad technology, if they are big data then maybe Sharepoint is big data too.

Does anyone out there really think these enterprise search-in-disguise vendors are big data?

– Max

I love free software, and Richard Stallman has done much to advance it, but this time he’s really gone too far.

From his blog:

“Steve Jobs, the pioneer of the computer as a jail made cool, designed to sever fools from their freedom, has died.

As Chicago Mayor Harold Washington said of the corrupt former Mayor Daley, “I’m not glad he’s dead, but I’m glad he’s gone.” Nobody deserves to have to die – not Jobs, not Mr. Bill, not even people guilty of bigger evils than theirs. But we all deserve the end of Jobs’ malign influence on people’s computing.

Unfortunately, that influence continues despite his absence. We can only hope his successors, as they attempt to carry on his legacy, will be less effective.”

I can’t speak on behalf of the entire open source community, but I can say that Richard Stallman certainly does not on this point.

My quick take:

Apple, under Jobs leadership, built some great products. They weren’t free (like beer or speech), but they moved the industry forward. Would it have been even better if they were free? Sure. But that’s a different world. I’m glad that folks have the freedom to make great software which is free and great software which is not. “Freedom” can be its own tyranny.

Love to hear your thoughts…

– Max

A room is 30 feet long, 12 feet wide, and 12 feet high.

An ant is one foot from the ceiling in the middle of the 12 foot long wall on the north end of the room.

A drop of honey is in the middle of the 12 foot long wall on the south end of the room, one foot from the floor.

How far does the ant have to walk to get the honey.

Hint: if the obvious path were the shortest, I wouldn’t have bothered to post the problem.

Have fun!

– Max

I’ve gotten a lot of questions from press, analysts, and customers about Oracle’s NoSQL offering. Our booth at Oracle Open World has been mobbed (next year we need a bigger booth – this year we had to make do with spilling over about 8 feet into the walkway on either side of our booth, sorry for the blockage for anyone trying to get past us!). I thought I’d share some further thoughts on Oracle’s now-announced offering here.

Disclaimer: I work for 10gen, the company that sponsors MongoDB, so I have some obvious bias. That said, I strongly believe that solid offerings from big players will help the market overall. With that bias disclosed, I have done my best to be fair and intellectually honest in my analysis of their offering.

About the offering

Personally, I think the Oracle offering is reasonable. Their approach to consistency looks sensible, as do their replication and sharding designs. It looks like it will be available open source, so I expect it will get a reasonable amount of adoption. I don’t think it breaks new ground, but if you want a key-value store from Oracle, it looks like it will do the job. My prediction is that it won’t crack the top 3 in NoSQL, but it looks solid and with Oracle behind it I’d expect it to wind up somewhere around # 5 or so in a very competitive market. This is just based on looking at their white papers; final judgement will have to wait until the product is released.

One plus for their offering is that it is based very heavily on BerkeleyDB. This technology has been around for at least 15 years, so it should be pretty stable. The main addition appears to be hash-based sharing; this is the part that customers need to ensure is stable before adopting it, but I’d expect the rest of it to be quite solid.

On criticism for flip-flopping

There have been a lot of harsh words about Oracle’s switch from writing a white paper on “Debunking the NoSQL Hype” in May to announcing a NoSQL offering in October. I wish it were otherwise, but welcome to the world of high-tech marketing. When a big company feels pain in the market and doesn’t have a product, they have their marketing guys write something that says the market isn’t important. You can be grumpy that the industry works that way and think whoever decided to do that is a hypocrite, but it really has no bearing on Oracle’s commitment to the product now that they have built it. Time to move on and evaluate the product on its merits.

So, congrats to Oracle on entering an exciting and growing market segment with what looks to be a solid offering. I look forward to seeing how it evolves over its next few releases and to competing with them in marketplace!

– Max