Unable to acquire bytes of memory running Spark SQL1.5

Sometimes, when we run complex sql in Spark1.5, e.g. with filtering, join, aggregation.
It will hit a bug in Tungsten. java.io.IOException: Unable to acquire 8388608 bytes of memory

Solution:
Disable the tungsten execution engine.
val sqlContext = new SQLContext(sc)
sqlContext.setConf("spark.sql.tungsten.enabled", "false")

Spark 1.6 resolved this issue.

16/01/14 14:27:00 ERROR Executor: Exception in task 0.0 in stage 9.0 (TID 52)

java.io.IOException: Unable to acquire 8388608 bytes of memory

        at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPage(UnsafeExternalSorter.java:368)

        at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.<init>(UnsafeExternalSorter.java:138)

        at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.create(UnsafeExternalSorter.java:106)

        at org.apache.spark.sql.execution.UnsafeExternalRowSorter.<init>(UnsafeExternalRowSorter.java:68)

        at org.apache.spark.sql.execution.TungstenSort.org$apache$spark$sql$execution$TungstenSort$$preparePartition$1(sort.scala:146)

        at org.apache.spark.sql.execution.TungstenSort$$anonfun$doExecute$3.apply(sort.scala:169)

        at org.apache.spark.sql.execution.TungstenSort$$anonfun$doExecute$3.apply(sort.scala:169)

        at org.apache.spark.rdd.MapPartitionsWithPreparationRDD.prepare(MapPartitionsWithPreparationRDD.scala:50)

       

Reference:
https://issues.apache.org/jira/browse/SPARK-10309