Monday 14 March 2016

Extracting the same columns from a list of Dataframs

We have a list of dataframes, which have only one row, with the same schema.

If we want to extract two specific columns from all dataframes then return values in two list.

val listPair: (List[String], List[String]) =
    headerDfs.map{ header=
            val row = header
                .select(field1, field2)
                .head
            (row(0).toString, row(1).toString)
         }.unzip[String, String]

Another approach is to union all dataframes, then do the select columns.

No comments:

Post a Comment