Spark SQL IQmulus Library

A library for reading and writing Lidar point cloud collections in PLY, LAS and XYZ formats from Spark SQL.

Requirements

This Spark package is for Spark 1.6.2

Usage

The spark-iqmulus library is published as a Spark package, which can be added to Spark jobs launched through spark-shell or spark-submit by using the --packages command line option. For example, to include it when starting the spark shell:

$ bin/spark-shell --packages IGNF:spark-iqmulus:0.1.0-s_2.10

Unlike using --jars, using --packages ensures that this library and its dependencies will be added to the classpath. The --packages argument can also be used with bin/spark-submit. For now this library is only published for Scala 2.10.

Scala API

Examples below require the creation of an SQLContext and the following imports, depending on which aspect of the library is being used:

val sqlContext = new SQLContext(sc)

import fr.ign.spark.iqmulus.ply._
import fr.ign.spark.iqmulus.las._
import fr.ign.spark.iqmulus.xyz._

Reading files as Spark dataframes:

val df1 = sqlContext.read.ply("myfile.ply")
val df2 = sqlContext.read.las("myfile.las")
val df3 = sqlContext.read.xyz("myfile.xyz")

Writing back Spark dataframes as files:

similarly, for XYZ and LAS :

val df = ...
df.write.ply("/tmp/ply")
df.write.las("/tmp/las")
df.write.xyz("/tmp/xyz")

Acknowledgments

The development of this library is partly supported by:

• the EU FP7 Project IQmulus, which leverages the information hidden in large heterogeneous geospatial data sets and make them a practical choice to support reliable decision making (N. ICT- 2011-318787).
• IGN, the French National Mapping Agency and its MATIS research lab.

References

To our knowledge this library has been used in the following research:

• M. Brédif, B. Vallet, B. Ferrand. Distributed dimensionality-based rendering of lidar point clouds. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XL-3/W3, 559-564, doi:10.5194/isprsarchives-XL-3-W3-559-2015, 2015. Paper Presentation Poster