Scala Java 8 SAM

Java 8 introduces SAM(Single Abstract Method) type to embrace functional programming.
SAM type is enabled by -Xexperimental flag in scala 2.11.x flags. In build.sbt, add below:

scalacOptions := Seq("-unchecked", "-deprecation", "-Xexperimental")

Interoperating with Java requires SAM, which also generates more efficient byte code since SAM has a native byte code counterpart. Using anonymous class for event handler or callback can be more pleasant in Scala just as in Java.

So for the stream example, if compiler has the -Xexperimental flag, scala will automatically change the function to java’s function, which grant scala user a seamless experience with the library.

Reference:

https://herringtondarkholme.github.io/2015/01/24/scala-sam/

Install Java8 and Scala in Ubuntu

1. Install Java

sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update 

 sudo apt-get install oracle-java8-installer

export JAVA_HOME=/usr/lib/jvm/java-8-oracle

2. Install Scala and Sbt

sudo apt-get remove scala-library scala
sudo wget www.scala-lang.org/files/archive/scala-2.10.4.deb
sudo dpkg -i scala-2.10.4.deb
sudo apt-get update
sudo apt-get install scala
wget http://scalasbt.artifactoryonline.com/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.12.4/sbt.deb
sudo dpkg -i sbt.deb
sudo apt-get update
sudo apt-get install sbt

3. Run a scala program with external config file

$ nohup java -Dconfig.file=application.conf  -classpath scala-program.jar com.alvin.ServiceBootStrap

Scala List Operation

Given a list: val list1 = List(1,3,4,0,-1,6)

1. Filter
The filter operator takes as operands a list of type List[T] and one function of type T => Boolean called predicate. This operator returns a new list with all elements of the original list for which the predicate is true.

scala> val list2 = list1 filter (_ > 0)

res0: List[Int] = List(1, 3, 4, 6)

2. Find

Returns the first element for which the predicate is true.

scala> list1 find (_ > 0)

res1: Option[Int] = Some(1)

3. Partition

The partition operator returns a pair of lists. The first one includes all elements that satisfies the predicate.

scala> list1 partition (_ > 0)

res3: (List[Int], List[Int]) = (List(1, 3, 4, 6),List(0, -1))

4. TakeWhile

The takeWhile operator iterates the list until it finds one element that doesn’t satisfy the predicate.

It returns the longest prefix such that every element satisfies the predicate.

scala> list1 takeWhile (_ > 0)

res4: List[Int] = List(1, 3, 4)

5. DropWhile

The dropWhile operator iterates the list until it finds one element that doesn’t satisfy the predicate,

It drops the longest prefix such that every element satisfies the predicate.

scala> list1 dropWhile (_ > 0)

res5: List[Int] = List(0, -1, 6)

6. Convert List[Option[T]] to List[T]
There is an implicit conversion from Option[A] to Iterable[A].

scala> val list1 = List(Some(1), None, Some(2))
scala> val list2 = list1.flatten
res5: List[Int] = List(1,2)

7. Pattern Matching
Match the list with unknown length.

object Names {
    def unapplySeq(name: String): Option[(String, String, Seq[String])] = {
        val names = name.trim.split("")
        if (names.size < 2) None
        else Some((names.last, names.head, names.drop(1).dropRight(1)))
    }
}


def greet(fullName: String) = fullName match {
    case Names(lastName, firstName, _*) = firstName + " " + lastName
    case _ = "Welcome"
}

Explicitly-typed-self vs. Inheritance

In Scala it is possible to tie a class to another type (which will be implemented in future) by giving self reference self the other type explicitly.

trait T {
  self : T2 =>
  ...
}

In the body, self is an alias for this but has the more precise type T with T2.
The trait T has been mixed in to an appropriate type, and makes those methods available.

Although both the explicit self-type annotation and the simple extends keyword describe an "is-a" relationship between two types, that relationship is not externally visible in the explicit self-type.

trait T
class C {this: T =>}
implicitly[C <:< T]

In the cake pattern you don't want your "module" object to be inadvertently, polymorphically used as one of the traits on which it depends.
When you use inheritance, you make decisions about initialization order. When you use self types, you leave that open.


Reference:
http://docs.scala-lang.org/tutorials/tour/explicitly-typed-self-references.html

Setting Optional Sections in the Typesafe Config File

If your configuration lets users invent new sections, you may not have all paths up front,
 and may be unable to set up defaults in reference.conf for dynamic paths.

In Scala, you could write an enrichment class to use the idiomatic Option syntax to optionally set an item in config file:

implicit class RichConfig(val underlying: Config) extends AnyVal {
  def getOptionalString(path: String): Option[String] = 
  if (underlying.hasPath(path)) {
     Some(underlying.getBoolean(path))
  } else {
    None
}}
val item = myConfig.getOptionalString("option_item")

Functors, Monads and Monoids in Scala

1. Functors

A function X => Y is transformed to another function List[X]=>List[Y] by a higher order function called functor.



def functor[X, Y](f: X=>Y): List[X]=>List[Y] = {
      def fun: List[X] => List[Y] = (arg: List[X]) => arg match {
            case Nil => Nil
            case x :: xs => f(x) :: fun(xs)
      }
      fun
}

functor return a function that takes List[X], invokes f on each element, and returns List[Y].

Example: Map

The map method takes a function as a parameter and applies it to each element in a container in order to return a new container. The output container will be the same time as the input container.
This is because of a builder "canBuildFrom" builds an appropriate builder for the type of input collection.

2. Monads

A function X => List[Y] is transformed to another function List[X] => List[Y] by a high order function called Monad.

Monads are containers. They support higher order functions and can be combined

together. Roughly speaking, if a type F is a monad, you can combine multiple instances
of F , where some are chosen based on the values located inside the others.

def monad[X, Y](f: X=>List[Y]): List[X] => List[Y] = {
   def fun: List[X] => List[Y] = (arg: List[X]) => arg match {
          case Nil => Nil
          case x :: xs => f(x) ::: fun(xs)
   }
   fun
}

Example: FlatMap

The flatMap method takes a function (X => List[Y]) as a parameter, applies it to each element in a container, and flattens the overall result.

3. Monoids

Monoid needs to be associative and should have an identity value. Associative paves the way to parallelize operations.



Example: FoldLeft.

Reference:

http://scabl.blogspot.ca/2013/02/monads-in-scala-1.html

http://bytes.codes/2015/04/10/a-skeptics-guide-to-scalaz-part-1-disjunctions/

http://stackoverflow.com/questions/16526282/use-scalaz-in-console-repl-without-creating-a-project

Sealed Trait

A sealed trait can only be extended within the file in which it is defined.

The compiler can use this to perform exhaustiveness checking. 

Another idiom related to sealed traits is to provide an alternative to enums.

object ScalaEnum extends App {
  sealed trait WeekDay
  case object Sun extends WeekDay
  case object Mon extends WeekDay
  case object Tue extends WeekDay
  case object Wed extends WeekDay
  case object Thu extends WeekDay
  case object Fri extends WeekDay
  case object Sat extends WeekDay

  def m(p: WeekDay) = println(p)
  m(Sat) // prints Sat
}

The case object is used when there is no instance-specific state! It is a kind of a singleton.

Scala compiler issues in IntelliJ

1. scalac object scala.runtime in compiler mirror not found

scala.reflect.internal.MissingRequirementError: object scala.runtime in compiler mirror not found.

Solution:

In IntelliJ IDE,
File -> Settings -> Build,Execution,Deployment -> Scala Compiler -> select sub project -> Additional Compiler options: remove dependecyFile

2. Error:scalac: Error: assertion failed: List(object package$DebugNode, object package$DebugNode)

      java.lang.AssertionError: assertion failed: List(object package$DebugNode, object package$DebugNode)
       at scala.reflect.internal.Symbols$Symbol.suchThat(Symbols.scala:1678)
       at scala.reflect.internal.Symbols$ClassSymbol.companionModule0(Symbols.scala:2988)
       at scala.reflect.internal.Symbols$ClassSymbol.companionModule(Symbols.scala:2991)
       at scala.tools.nsc.backend.jvm.GenASM$JPlainBuilder.genClass(GenASM.scala:1371)




Solution: mvn clean compile, then rebuild.

Reference:

http://stackoverflow.com/questions/24165184/scalac-assertion-failed-while-run-scalatest-in-idea

http://apache-spark-user-list.1001560.n3.nabble.com/spark-github-source-build-error-td10532.html

Parsing Map Structure from Config File

In some cases, we want to set a map relationship in config file, and keep their orders in our app.
In application.conf, we have a map like below.
Note: The key must be a string, can't contain any variables.
If it contains '.', a double quota is needed.

pairs = [
    {"c.json": ccc},
    {"a.json": aaa},
    {"b.json": bbb}
  ]

import scala.collection.immutable.ListMap
lazy val inputPairs : ListMap[String, String] = {

      val list : Iterable[ConfigObject] = stageConfig
        .getObjectList("pairs")
        .toList

     val map = for {
        item : ConfigObject = list
        entry : Entry[String, ConfigValue] = item.entrySet().toList
        col = entry.getKey
        func =  entry.getValue.unwrapped().toString
      } yield (col, func)

      map.map(identity)(collection.breakOut)
    }  

Note: Here we use immutable.ListMap to keep the input order.
Otherwise, Map will return uncertain orders.

In the definition of map:

def map[B, That](f : (A) => B)(implicit bf : CanBuildFrom[Repr, B, That]) : That 

The first is your function and the second is an implicit. If you do not provide that implicit, Scala will choose the most specific one available.

breakOut can help to skip the intermediary List and collect the results directly into a Map

The definition of breakOut:

def breakOut[From, T, To](implicit b : CanBuildFrom[Nothing, T, To]) =
  new CanBuildFrom[From, T, To] {
    def apply(from: From) = b.apply() ; def apply() = b.apply()
  }

Reference:

http://stackoverflow.com/questions/3827964/scala-for-comprehension-returning-an-ordered-map

http://stackoverflow.com/questions/1715681/scala-2-8-breakout

https://lauris.github.io/map-order-scala/

http://stackoverflow.com/questions/4531856/why-is-there-no-mutable-treemap-in-scala

Fetching AppInfo from Manifest

Sometimes, we want to dynamically fetch App info without hardcoding them.
We can get these info from manifest.

Add below plugin in the parent pom.xml
In the child pom.xml, setup below
<artifactId>com.test.app</artifactId>
<name>AppName</name>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.5</version>
<configuration>
<archive>
     <manifest>
<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
     </manifest>
<manifestEntries>
<Implementation-Build>${maven.build.timestamp}</Implementation-Build>
<APP-ID>${artifactId}</APP-ID>
<APP-NAME>${name}</APP-NAME>
<APP-VERSION>${version}</APP-VERSION>
        </manifestEntries>
</archive>
</configuration>
</plugin>

def getAppInfo: (String, String) = {

    val p = getClass.getPackage
    val appName = Option(p.getImplementationTitle)
                  .getOrElse("MyApp")
    val version = Option(p.getImplementationVersion)
                  .getOrElse("SNAPSHOT")
    (appName, version)
  }

Existential Types

Existential types are a way of abstracting over types. They let you assert that some
type "exists" without specifying exactly what it is.

The type parameters of generics are erased in JVM byte code. For example, when a
List[Int] is created, the Int type is not available in the byte code, so at runtime it's not
possible to distinguish between List[Int] and a List[String], based on the known type
information.

object Doubler{

    def double(seq: Seq[_]): Seq[Int] = 

         seq match{
     case Nil = Nil
     case head +: tail = (toInt(head)*2)+:double(tail)
 }

     private def toInt(x:Any):Int = 

         x match{
     case i:Int = i
     case s:String = s.toInt
     case x = throw new RuntimeException(s"Unexpected list element $x")
 }
}

The expression Seq[_] is actually shorthand for the existential type, Seq[T] forSome {type T}.
T can be any subtype of Any. Existential types exist primarily to support Java generics while preserving correctness in Scala's type system.

Scala Best Practices

1. SHOULD NOT declare abstract "var" members

It's a bad practice to declare abstract vars in abstract classes or traits.
Do not do this:
trait Foo {
  var value: String
}

Instead, prefer to declare abstract things as def:
trait Foo {
  def value: String
}

// can then be overridden as anything
class Bar(val value: String) extends Foo

The reason has to do with the imposed restriction - a var can only be overridden with a var. The way to allow freedom to choose on inheritance is to use def for abstract members. And why would you impose the restriction to use a var on those that inherit from your interface. def is generic so use it instead.

2. MUST NOT use Option.get

You might be tempted to do this:
val someValue: Option[Double] = ???

// val result = someValue.get + 1
Don't ever do that, since your trading a NullPointerException for a NoSuchElementException and that defeats the purpose of using Option in the first place.
Alternatives:

1. using Option.getOrElse
2. using Option.fold
3. using pattern matching and dealing with the None branch explicitly
4. not taking the value out of its optional context

As an example for (4), not taking the value out of its context means this:
val result = someValue.map(_ + 1)

Instead of working with Any, think about the generic type you want and the set of sub-types you need, and come up with an Algebraic Data-Type:

sealed trait JsValue

case class JsNumber(v: Double) extends JsValue
case class JsBool(v: Boolean) extends JsValue
case class JsString(v: String) extends JsValue
case class JsObject(map: Map[String, JsValue]) extends JsValue
case class JsArray(list: Seq[JsValue]) extends JsValue
case object JsNull extends JsValue

Now, instead of operating on Any, we can do pattern matching on JsValue and the compiler can help us here on missing branches, since the choice is finite.


3. SHOULD NOT define case classes nested in other classes

It is tempting, but you should almost never define nested case classes inside another object/class because it messes with Java's serialization. The reason is that when you serialize a case class it closes over the "this" pointer and serializes the whole object, which if you are putting in your App object means for every instance of a case class you serialize the whole world.
And the thing with case classes specifically is that:

1. one expects a case class to be immutable (a value, a fact) and hence
2. one expects a case class to be easily serializable

Prefer flat hierachies.

4. MUST NOT include classes, traits and objects inside package objects

Classes, including case classes, traits and objects do not belong inside package objects. It is unnecessary, confuses the compiler and is therefore discouraged.
For example, refrain from doing the following:

package foo
package object bar {
  case object FooBar
}

The same effect is achieved if all artifacts are inside a plain package:

package foo.bar
case object FooBar

Package objects should only contain value, method and type alias definitions, etc. Scala allows multiple public classes in a single file, and the convention is to have the first letter of the filename be lowercase in such cases.


Reference:
https://github.com/AlvinCJin/scala-best-practices/blob/master/sections/2-language-rules.md

Package Object

The standard Scala package also has its package object. Because scala._ is automatically imported into every Scala file, the definitions of this object are available without prefix.

As you can see, the main purpose of this object is to make a number of often-used definitions nested in subpackages available from the scala package. 

For instance, the List type is used so often that it makes sense to put it in the scala package thereby making it accessible without an import or name qualification. For example,

  package object Errors {

    case class InvalidConfigurationException(val msg: String) extends RuntimeException(msg)
    case class UnsupportedFormatException(val msg: String) extends RuntimeException(msg)

}

Reference:

http://www.scala-lang.org/docu/files/packageobjects/packageobjects.html