BYO Fusion Boot Drive

After reading about how the new Fusion Drive in the latest Macs work, I was surprised to find that it's not a hardware level algorithm (similar to my Seagate Momentus XT) that writes to both the SSD and hardrive but rather a tiered implementation that stores frequently read AND write data on the faster media alone. I have an SSD and a WD Caviar Black hardrive in my Mac Pro at work and having to constantly delete downloaded files and/or move things to the slower drive (that doesn't always work since, for instance, your Maven home needs to be on one drive and I know for a fact that I probably access only 5% of the artifacts frequently), I decided to give it a try and followed the instructions on http://jollyjinx.tumblr.com/ to roll my own Fusion BYO drive on my 2011 Mac Pro. I wanted my bootdrive (as opposed to a secondary drive) to be one single unified drive that automatically move things around based on my usage. I have a gut feeling that with 128GB of SSD, almost everything that I need quick access to would be "hot".

Obviously, this whole setup requires 10.8.2.

First thing, I backed up my computer using Time Machine twice, it's already backing up to a Drobo FS device on the network but just in case, I backed it up again to an external drive. It's nice that Time Machine can actually handle having more than one backup drive which is nice for those who are paranoid about data loss I suppose. I found out however, that a complete system backup exists only on the Drobo device and not the external drive but since I am restoring after a reinstall anyways, it doesn't really matter.

First off, I need to create have OSX Recovery on a disk or an external drive since my Mac does not have internet recovery (or it won't trigger as long as my local disk has the recovery partition). I just repartitioned my external drive to have a 1GB partition for that purpose and downloaded OSX recovery assistant http://support.apple.com/kb/DL1433 to copy the local recovery partition to the external drive.

Then, holding the "Option" key while restarting, boot to that recovery partition. Then, I ran the commands that jollyjinx wrote in a terminal (Utilities -> Terminal).

First run diskutil list to figure out which physical disks will form the logical volume.

Then, diskutil cs create NAME_OF_LOGICAL_VOLUME_GROUP FIRST_DISK SECOND_DISK

For instance, I typed diskutil cs create fusion disk1 disk2

Afterwards, run diskutil cs list to get the logical volume group UUID to create the logical volume:

diskutil cs list

diskutil cs createVolume LOGICAL_VOLUME_GROUP_UUID jhfs+ fusion 100%

This creates a new volume (unencrypted) called fusion with journaled HFS+ occupying the entire drive.

Now, after that, close the terminal (you need to actually do Quit from the menu bar) and then reinstall (do not restore from Time Machine since it wouldn't know how to create the recovery partition) Mountain Lion. It checks for your App Store login at this point you make sure you use one that has the purchase associated (if not, just pay Apple another $19.99).

It'll download the installer again (mine took about 35mins) and install Mountain Lion to your local machine. It will restart after downloading and then install OS X on the disk "fusion" (what I named the volume). Takes about 20 minutes on my mac.

Afterwards, boot into mac, use Migration Assistant or otherwise, restore your files. I chose to just re-install everything and copy the relevant documents back from the Time Machine sparsebundle. FileVault works and I can partition the drive via diskutil again.

Mission Accomplished. :)

What Scala?

After 2 years with Scala, our company (GLMX) decided to finally say goodbye to the language that was (or is still perhaps) the future of Java. We did the unthinkable of converting all the code back to Java and was pleasantly surprised that it wasn't as bloated, non-intuitive and verbose as we thought it would be. On the contrary, the code is what it is (what-you-code-is-what-you-get), no implicit conversions, inferencing, traits that modified behavior (that really should have been done with composition), coupled with the fact that there are nice Java libraries out there (Google Guava for one) that encapsulates a lot of the functional aspects that we liked about Scala (never mind the fact that functions are really classes in Scala), it's actually refreshing to know that we don't have to do a whole lot of ".seqAsJavaList", ".bigDecimal" and teach other frameworks (e.g. Spring) to understand what's a scala Int). It would have been nice if all frameworks were indeed written in Scala but alas, we live in a world where interoperability is a requirement, not a nice-to-have.

Couple of learnings about Scala (copied from a chat I had with an ex-Googler friend of mine asking about my thoughts about it):

1. It's hard to read the code quickly (code concepts / 100 char line) with all the implicit conversions, language specific peculiarities, compositions, etc., you used to be able to read code by glancing, with scala you can't because the density of logic vs. characters can suddenly jump within a single line (e.g. one line you have an assignment to a val, the next one is the reversal of a list, implicitly converted from a java list, then a flatmap on the output that extracts a collection, partitioning by some predicate, ... all on one line)
2. Build tools don't work perfectly with scala (some application servers don't like the code it generates, but that's not to blame the language itself), it doesn't co-exist with java in some corner cases (meaning we need to write java anyways, JPA for one, if you need typed criteria query meta-model classes)
3. Hard to find scala programmers and training them takes a while (from java)
4. The actual productivity gains are not that much after a while; an IDE is good for eliminating a lot of the cruft of java and you realize that you don't actually save that much from coding
5. The libraries encourage you to trash the heap quite a bit without too much thought (since it's so easy), with java though, you are actually more aware of allocations and performance since there's less magic happening

Then the more specific issues:

• XML support in scala is nice but when you need true marshalling and unmarshalling, you are back to some java-based framework
• The provided types are nice but once you start mixing java frameworks and scala frameworks, you have to be real careful passing in a Scala list of BigDecimal since even if the conversion is done properly to yield a Java list, the BigDecimals in it still needs another round of unwrapping
• It's actually nice to just use iteration rather than making it real "scala-ly" and issue a chain of function calls on collections to do what you want. I understand that you can still do iteration in Scala but it's just too tempting to be cool and instead instantiate a whole bunch of functions (which, again, are actual classes in the JVM) and create a bunch of throw-away intermediate collections
• Then, there's always the debate about whether to pass a Buffer, List, Seq, Collection, Traversable, TraversableOnce around. :) With Java, it's easy to suggest that functions should take in the most flexible inputs while returning the most specific outputs (i.e. taking in Collection<T> and returning a List<T>)

Okay, I still really do like the fact that you don't need () everywhere and must end a line with a semi-colon though. That and the fact that you are more inclined to use "final" variables.

Finally, a working Nest Auto-Away with Google Latitude

This is a long time coming, I needed Indigo 5 (used as an event engine in this case), the Google Latitude Bridge I whipped up earlier (https://latitudebridge.appspot.com), the Nest for Indigo Plugin (https://github.com/johnray/Indigo-Nest-Thermostat-Plugin/blob/master/README.md) and my own Google Latitude for Indigo Script (http://www.perceptiveautomation.com/filelib/index.php?keywords=google+latitude).

Background: My Nest's Auto-Away never really worked properly since I live in a townhouse with 3 floors and I am the only person there. It never said that it detected enough activity and even when I tried to fool it (by purposefully "generating activity" in front of it when it's learning), it's still spotty at best. In any case, it will only turn on auto-away when no activity has been detected for 2 hours.

I would love to have it automatically detect when I am away accurately (saving me money) and I realized that everything is there technically: 1) an internet enabled thermostat, 2) a location tracking server and 3) a device that follows me all the time.

So, with a bit of tinkering, my Nest now has a couple new tricks: when I am away (>1km from home) for 5 minutes, it engages away status immediately; when I am back (<1km from home) for 5 minutes, it disengages away status. With Indigo, it actually says it out loud. :) When I am back in the house, it would say "away status for thermostat disengaged". Cool :)

Hibernate Envers Bug HHH-7157

While helping out a co-worker today and looking at a stack trace, I got a deja-vu feeling that I have seen the bug before. Turns out it was the same manifestation where if you have a Many-to-One and One-to-Many relationship between two entities (one using embedeed ids) and they are audited, Hibernate generates a query that's not valid (talk about ORM errors), it would supply too many query parameters to the underlying database resulting in a nasty exception that you think typically don't exist anymore if you use an ORM framework.

Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Invalid value "7" for parameter "parameterIndex" [90008-143]


For those interested, I saw the bug back in March and filed a bug against Hibernate with a test case reproducing the bug on github: https://hibernate.onjira.com/browse/HHH-7157. Unfortunately, it's still not fixed in the latest release 4.1.5.SP1. The workaround is to add @NotAudited to the @OneToMany side of the relationship (which does not inhibit any auditing as the OneToMany side of the relationship isn't really represented in a database row anyways).

Full stack trace below (different DBMS will produce a different error message though since it's technically an error with the actual query):


javax.persistence.RollbackException: Error while committing the transaction
at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:90)
at org.hibernate.clementp.EnversTest.testAuditedEmbeddedId(EnversTest.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:76)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:182)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Invalid value "7" for parameter "parameterIndex" [90008-143]
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1360)
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1288)
at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:78)
... 31 more
Caused by: org.hibernate.exception.GenericJDBCException: Invalid value "7" for parameter "parameterIndex" [90008-143]
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:129)
at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81)
at $Proxy22.setLong(Unknown Source)
at org.hibernate.type.descriptor.sql.BigIntTypeDescriptor$1.doBind(BigIntTypeDescriptor.java:57)
at org.hibernate.type.descriptor.sql.BasicBinder.bind(BasicBinder.java:92)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:305)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:300)
at org.hibernate.type.ComponentType.nullSafeSet(ComponentType.java:358)
at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate(AbstractEntityPersister.java:2611)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2854)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3298)
at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:88)
at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:362)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:354)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:275)
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:326)
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1127)
at org.hibernate.envers.synchronization.AuditProcess.doBeforeTransactionCompletion(AuditProcess.java:157)
at org.hibernate.engine.spi.ActionQueue$BeforeTransactionCompletionProcessQueue.beforeTransactionCompletion(ActionQueue.java:662)
at org.hibernate.engine.spi.ActionQueue.beforeTransactionCompletion(ActionQueue.java:307)
at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:524)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.beforeTransactionCommit(JdbcTransaction.java:105)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:175)
at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:73)
... 31 more
Caused by: org.h2.jdbc.JdbcSQLException: Invalid value "7" for parameter "parameterIndex" [90008-143]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.getInvalidValueException(DbException.java:213)
at org.h2.jdbc.JdbcPreparedStatement.setParameter(JdbcPreparedStatement.java:1254)
at org.h2.jdbc.JdbcPreparedStatement.setLong(JdbcPreparedStatement.java:541)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:122)
... 55 more

Google Latitude OAuth Bridge

As part of my project to use Indigo 5 (http://www.perceptiveautomation.com/indigo/index.html) to automate my house (and also remind me when I am forgetful of doing things like closing the garage door when I leave the house...), I decided to integrate Google Latitude into the server so that a variable "isAtHome" would be exposed and various triggers can decide on what to do (e.g. emailing me when nobody's at home but there's activity detected in the house).

Anyways, it turned out to be a bit of a difficult task since the Google Latitude API requires OAuth 2.0 with tokens that live for an hour. That wouldn't really work with something like Indigo which would have required me to store a whole bunch of stuff (client ids, client secrets, tokens, refresher tokens) and execute multiple round-trips against Google auth servers before you can get your location securely.

Well, decided that I might as well do what I do best and whipped up a Google App Engine + Play Framework + Siena + bootstrap combo that would allow me to generate a permanent URL for the JSON query.

After about 2 hours, http://latitudebridge.appspot.com is up. :)

Twitter's Bootstrap project is awesome!

Basically, you do the OAuth once and it generates a unique URL for you to query your current position. Just like the old days. :)

Next up, get the applescript written so that it can do a curl and figure out the rest.

First Post!

Finally decided to start a public blog on more technical stuff that would likely bore my Facebook friends but probably useful to others out there . :)