When Sun described their fallacies of distributed computing in 1994, they went into detail.  It is an interesting read because it assumes something like Terracotta cannot exist.  Why?  Because Sun engineers assumed the entire heap would be replicated and, as I said in my talk, we do not replicate everything as much as we can replicate almost anything.  Specifically, they say transparency will suffer from 4 problems:
1. network latency
2. Memory address spaces cannot transparently translate
3. concurrency semantics won't translate
4. failure cannot be efficiently dealt with in a cluster.

The paper says to ignore #1 since the network will just get faster (10MBits when they wrote the paper is now 10Gbits 12 years later...)
2.  We can ignore #2 since object identity is  preserved with Terracotta and we don't have strange copies of objects appearing where the developer hadn't expected references to end up generating clones on deserialization
3.  Sun didn't assert that making locks clustered is a bad idea.  They only asserted that synchronization does not become clustered and thus, clustered programming cannot be transparent.  Again, Terracotta did not exist at the time.

4. The paper asserts that #1-3 can be quite easily dealt with (as Terracotta has done--arguably not transparently but with config).  This is the one they say that is insurmountable.  But #4 assumes a peer-to-peer cluster and Sun engineers are talking about network partitioning.  Terracotta's server makes recovery from failure as easy as waiting for another Terracotta Server; there is no such thing as a partition.  its not perfect by any means (Oracle Active / Passive clusters fail to fail over all the time).  But it is O(1) to find a backup place to write your data.  O(1) to find a backup server is what Amazon, EBay, and Google do with their decentralized architectures and these seem to work, no?

I don't mean to come off as defensive.  I just want to get the facts about "fallacies of distributed computing" written down somewhere.

The config is the key.  Without it, Sun is correct.  With it, we are still programming clustering into our app, just not through saying "implements Serializable" and "get()" or "put()."

--Ari