Databricks Runtime 3.4

Databricks released this image in November, 2017.

Important

This release was deprecated on July 30, 2018. For more information about the Databricks Runtime deprecation policy and schedule, see Databricks Runtime Support Lifecycle.

The following release notes provide information about the Databricks Runtime 3.4 powered by Apache Spark.

Changes and Improvements

  • Introduced new vectorized UDFs for PySpark.
  • Capable of limiting the max number of partitions in query watchdog.
  • Improved error message communication when installing CRAN packages using Databricks Library UI and API.
  • MySQL JDBC driver has been replaced by MariaDB JDBC driver.
  • Upgraded Python library setuptools from 36.2.7 to 36.6.0, wheel from 0.30.0a0 to 0.30.0.
  • Upgraded R from 3.2.3 to 3.4.2.
  • Upgraded several pre-installed R libraries. Refer to Pre-installed R Libraries for details.
  • Added AES encryption for customers who require HIPAA-compliant clusters

Apache Spark

The Databricks Runtime 3.4 includes Apache Spark 2.2.0. Databricks Runtime 3.4 also includes the following extra bug fixes and improvements made to Spark:

  • [SPARK-21718][SQL] Heavy log of type: “Skipping partition based on stats …”
  • [SPARK-22278][SS] Expose current event time watermark and current processing time in GroupState
  • [SPARK-22187][SS] Update unsaferow format for saved state such that we can set timeouts when state is null
  • [SPARK-22136][SS] Evaluate one-sided conditions early in stream-stream joins
  • [SPARK-20396][SQL][PYSPARK] groupby().apply() with pandas udf
  • [SPARK-22124][SQL] Sample and Limit should also defer input evaluation under codegen
  • [SPARK-21782][CORE] Repartition creates skews when numPartitions is a power of 2
  • [SPARK-21527][CORE] Use buffer limit in order to use JAVA NIO Util’s buffercache
  • [SPARK-22176][SQL] Fix overflow issue in Dataset.show
  • [SPARK-21410][CORE] Create less partitions for RangePartitioner if RDD.count() is less than partitions
  • [SPARK-21517][CORE] Avoid copying memory when transfer chunks remotely
  • [SPARK-22238]Fix plan resolution bug caused by EnsureStatefulOpPartitioning
  • [SPARK-22159][SQL] Make config names consistently end with “enabled”.
  • [SPARK-20331][SQL] Enhanced Hive partition pruning predicate pushdown
  • [SPARK-22223][SQL] ObjectHashAggregate should not introduce unnecessary shuffle
  • [SPARK-22263][SQL] Refactor deterministic as lazy value
  • [SPARK-22257][SQL] Reserve all non-deterministic expressions in ExpressionSet
  • [SPARK-21907][CORE] Fix oom during spill
  • [SPARK-22252][SPARK-21165]FileFormatWriter bug fixes
  • [SPARK-22230]Swap per-row order in state store restore.
  • [SPARK-21988][SS] Implement StreamingRelation.computeStats to fix explain
  • [SPARK-22170][SQL] Reduce memory consumption in broadcast joins
  • [SPARK-22206][SQL][SPARKR] gapply in R can’t work on empty grouping columns
  • [SPARK-20466][CORE] HadoopRDD#addLocalConfiguration throws NPE
  • [SPARK-22178][SQL] Refresh Persistent Views by REFRESH TABLE Command
  • [SPARK-22158][SQL] convertMetastore should not ignore table property
  • [SPARK-22146]FileNotFoundException while reading ORC files containing special characters
  • [SPARK-22161][SQL] Add Impala-modified TPC-DS queries
  • [SPARK-22129][SPARK-22138] Release script improvements
  • [SPARK-22143][SQL] Fix memory leak in OffHeapColumnVector
  • [SPARK-22140]Add TPCDSQuerySuite
  • [SPARK-22214][SQL] Refactor the list hive partitions code
  • [SPARK-22136][SS] Implement stream-stream outer joins.
  • [SPARK-22053][SS] Stream-stream inner join in Append Mode
  • [SPARK-22153][SQL] Rename ShuffleExchange -ShuffleExchangeExec
  • [SPARK-22160][SQL] Make sample points per partition (in range partitioner) configurable and bump the default value up to 100
  • [SPARK-22159][SQL] Make config names consistently end with “enabled”.
  • [SPARK-22141][SQL] Propagate empty relation before checking Cartesian products
  • [SPARK-22120][SQL] TestHiveSparkSession.reset() should clean out Hive warehouse directory
  • [SPARK-22083][CORE] Release locks in MemoryStore.evictBlocksToFreeSpace
  • [SPARK-22107]Change as to alias in python quickstart
  • [SPARK-22109][SQL] Resolves type conflicts between strings and timestamps in partition column
  • [SPARK-22092]Reallocation in OffHeapColumnVector.reserveInternal corrupts struct and array data
  • [SPARK-18136]Fix SPARK_JARS_DIR for Python pip install on Windows
  • [SPARK-22094][SS] processAllAvailable should check the query state
  • [SPARK-21928][CORE] Set classloader on SerializerManager’s private kryo
  • [SPARK-22076][SQL] Expand.projections should not be a Stream
  • [SPARK-22052]Incorrect Metric assigned in MetricsReporter.scala
  • [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles
  • [SPARK-21953]Show both memory and disk bytes spilled if either is present
  • [SPARK-21985][PYSPARK] PairDeserializer is broken for double-zipped RDDs
  • [SPARK-18608][ML][FOLLOWUP] Fix double caching for PySpark OneVsRest.
  • [SPARK-21980][SQL] References in grouping functions should be indexed with semanticEquals
  • [SPARK-22103]Move HashAggregateExec parent consume to a separate function
  • [SPARK-22141][SQL] Propagate empty relation before checking Cartesian products
  • [SPARK-21977]SinglePartition optimizations break certain Streaming Stateful Aggregation requirements

Known Issues

  • Log links on the executor page are not set correctly. Please use the worker page to access stdout and stderr links of an executor for now.

System Environment

  • Operating System: Ubuntu 16.04.3 LTS
  • Java: 1.8.0_131
  • Scala: 2.10.6 (Scala 2.10 cluster version)/2.11.8 (Scala 2.11 cluster version)
  • Python: 2.7.12 (or 3.5.2 if Python 3 support is enabled)
  • R: R version 3.4.2 (2017-09-28)
  • For GPU clusters, the following NVIDIA GPU libraries are installed:
    • CUDA 8.0
    • CUDNN 6.0

Pre-installed Python Libraries

Library Version Library Version Library Version
ansi2html 1.1.1 argparse 1.2.1 backports-abc 0.5
boto 2.42.0 boto3 1.4.1 botocore 1.4.70
brewer2mpl 1.4.1 certifi 2016.2.28 cffi 1.7.0
chardet 2.3.0 colorama 0.3.7 configobj 5.0.6
cryptography 1.5 cycler 0.10.0 Cython 0.24.1
decorator 4.0.10 docutils 0.14 enum34 1.1.6
et-xmlfile 1.0.1 freetype-py 1.0.2 funcsigs 1.0.2
fusepy 2.0.4 futures 3.1.1 ggplot 0.6.8
html5lib 0.999 idna 2.1 ipaddress 1.0.16
ipython 2.2.0 ipython-genutils 0.1.0 jdcal 1.2
Jinja2 2.8 jmespath 0.9.0 llvmlite 0.13.0
lxml 3.6.4 MarkupSafe 0.23 matplotlib 1.5.3
mpld3 0.2 msgpack-python 0.4.7 ndg-httpsclient 0.3.3
numba 0.28.1 numpy 1.11.1 openpyxl 2.3.2
pandas 0.18.1 pathlib2 2.1.0 patsy 0.4.1
pexpect 4.0.1 pickleshare 0.7.4 Pillow 3.3.1
pip 9.0.1 ply 3.9 prompt-toolkit 1.0.7
psycopg2 2.6.2 ptyprocess 0.5.1 py4j 0.10.3
pyarrow 0.4.1 pyasn1 0.1.9 pycparser 2.14
Pygments 2.1.3 PyGObject 3.20.0 pyOpenSSL 16.0.0
pyparsing 2.2.0 pypng 0.0.18 Python 2.7.12
python-dateutil 2.5.3 python-geohash 0.8.5 pytz 2016.6.1
requests 2.11.1 s3transfer 0.1.9 scikit-learn 0.18.1
scipy 0.18.1 scour 0.32 seaborn 0.7.1
setuptools 36.6.0 simplejson 3.8.2 simples3 1.0
singledispatch 3.4.0.3 six 1.10.0 statsmodels 0.6.1
tornado 4.5.2 traitlets 4.3.0 urllib3 1.19.1
virtualenv 15.0.1 wcwidth 0.1.7 wheel 0.30.0
wsgiref 0.1.2        

Pre-installed R Libraries

Library Version Library Version Library Version
abind 1.4-5 assertthat 0.2.0 backports 1.1.1
base 3.4.2 BH 1.65.0-1 bindr 0.1
bindrcpp 0.2 bit 1.1-12 bit64 0.9-7
bitops 1.0-6 blob 1.1.0 boot 1.3-20
brew 1.0-6 car 2.1-5 caret 6.0-77
chron 2.3-51 class 7.3-14 cluster 2.0.6
codetools 0.2-15 colorspace 1.3-2 commonmark 1.4
compiler 3.4.2 crayon 1.3.4 curl 3.0
CVST 0.2-1 data.table 1.10.4 datasets 3.4.2
DBI 0.7 ddalpha 1.3.1 DEoptimR 1.0-8
desc 1.1.1 devtools 1.13.3 dichromat 2.0-0
digest 0.6.12 dimRed 0.1.0 doMC 1.3.4
dplyr 0.7.4 DRR 0.0.2 foreach 1.4.3
foreign 0.8-69 gbm 2.1.3 ggplot2 2.2.1
git2r 0.19.0 glmnet 2.0-13 glue 1.1.1
gower 0.1.2 graphics 3.4.2 grDevices 3.4.2
grid 3.4.2 gsubfn 0.6-6 gtable 0.2.0
h2o 3.14.0.3 httr 1.3.1 hwriter 1.3.2
hwriterPlus 1.0-3 ipred 0.9-6 iterators 1.0.8
jsonlite 1.5 kernlab 0.9-25 KernSmooth 2.23-15
labeling 0.3 lattice 0.20-35 lava 1.5.1
lazyeval 0.2.0 littler 0.3.2 lme4 1.1-14
lubridate 1.6.0 magrittr 1.5 mapproj 1.2-5
maps 3.2.0 MASS 7.3-47 Matrix 1.2-11
MatrixModels 0.4-1 memoise 1.1.0 methods 3.4.2
mgcv 1.8-22 mime 0.5 minqa 1.2.4
ModelMetrics 1.1.0 munsell 0.4.3 mvtnorm 1.0-6
nlme 3.1-131 nloptr 1.0.4 nnet 7.3-12
numDeriv 2016.8-1 openssl 0.9.7 parallel 3.4.2
pbkrtest 0.4-7 pkgconfig 2.0.1 pkgKitten 0.1.4
plogr 0.1-1 plyr 1.8.4 praise 1.0.0
pROC 1.10.0 prodlim 1.6.1 proto 1.0.0
purrr 0.2.3 quantreg 5.33 R.methodsS3 1.7.1
R.oo 1.21.0 R.utils 2.5.0 R6 2.2.2
randomForest 4.6-12 RColorBrewer 1.1-2 Rcpp 0.12.13
RcppEigen 0.3.3.3.0 RcppRoll 0.2.2 RCurl 1.95-4.8
recipes 0.1.0 reshape2 1.4.2 rlang 0.1.2
robustbase 0.92-7 RODBC 1.3-15 roxygen2 6.0.1
rpart 4.1-11 rprojroot 1.2 Rserve 1.7-3
RSQLite 2.0 rstudioapi 0.7 scales 0.5.0
sfsmisc 1.1-1 sp 1.2-5 SparkR 2.2.0
SparseM 1.77 spatial 7.3-11 splines 3.4.2
sqldf 0.4-11 statmod 1.4.30 stats 3.4.2
stats4 3.4.2 stringi 1.1.5 stringr 1.2.0
survival 2.41-3 tcltk 3.4.2 TeachingDemos 2.10
testthat 1.0.2 tibble 1.3.4 tidyselect 0.2.0
timeDate 3012.100 tools 3.4.2 utils 3.4.2
viridisLite 0.2.0 whisker 0.3-2 withr 2.0.0
xml2 1.1.1        

Pre-installed Java and Scala libraries (Scala 2.10 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.7.3
com.amazonaws aws-java-sdk-autoscaling 1.11.126
com.amazonaws aws-java-sdk-cloudformation 1.11.126
com.amazonaws aws-java-sdk-cloudfront 1.11.126
com.amazonaws aws-java-sdk-cloudhsm 1.11.126
com.amazonaws aws-java-sdk-cloudsearch 1.11.126
com.amazonaws aws-java-sdk-cloudtrail 1.11.126
com.amazonaws aws-java-sdk-cloudwatch 1.11.126
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.126
com.amazonaws aws-java-sdk-codedeploy 1.11.126
com.amazonaws aws-java-sdk-cognitoidentity 1.11.126
com.amazonaws aws-java-sdk-cognitosync 1.11.126
com.amazonaws aws-java-sdk-config 1.11.126
com.amazonaws aws-java-sdk-core 1.11.126
com.amazonaws aws-java-sdk-datapipeline 1.11.126
com.amazonaws aws-java-sdk-directconnect 1.11.126
com.amazonaws aws-java-sdk-directory 1.11.126
com.amazonaws aws-java-sdk-dynamodb 1.11.126
com.amazonaws aws-java-sdk-ec2 1.11.126
com.amazonaws aws-java-sdk-ecs 1.11.126
com.amazonaws aws-java-sdk-efs 1.11.126
com.amazonaws aws-java-sdk-elasticache 1.11.126
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.126
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.126
com.amazonaws aws-java-sdk-elastictranscoder 1.11.126
com.amazonaws aws-java-sdk-emr 1.11.126
com.amazonaws aws-java-sdk-glacier 1.11.126
com.amazonaws aws-java-sdk-iam 1.11.126
com.amazonaws aws-java-sdk-importexport 1.11.126
com.amazonaws aws-java-sdk-kinesis 1.11.126
com.amazonaws aws-java-sdk-kms 1.11.126
com.amazonaws aws-java-sdk-lambda 1.11.126
com.amazonaws aws-java-sdk-logs 1.11.126
com.amazonaws aws-java-sdk-machinelearning 1.11.126
com.amazonaws aws-java-sdk-opsworks 1.11.126
com.amazonaws aws-java-sdk-rds 1.11.126
com.amazonaws aws-java-sdk-redshift 1.11.126
com.amazonaws aws-java-sdk-route53 1.11.126
com.amazonaws aws-java-sdk-s3 1.11.126
com.amazonaws aws-java-sdk-ses 1.11.126
com.amazonaws aws-java-sdk-simpledb 1.11.126
com.amazonaws aws-java-sdk-simpleworkflow 1.11.126
com.amazonaws aws-java-sdk-sns 1.11.126
com.amazonaws aws-java-sdk-sqs 1.11.126
com.amazonaws aws-java-sdk-ssm 1.11.126
com.amazonaws aws-java-sdk-storagegateway 1.11.126
com.amazonaws aws-java-sdk-sts 1.11.126
com.amazonaws aws-java-sdk-support 1.11.126
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.126
com.amazonaws jmespath-java 1.11.126
com.carrotsearch hppc 0.7.1
com.chuusai shapeless_2.10 2.3.2
com.clearspring.analytics stream 2.7.0
com.databricks Rserve 1.8-3
com.databricks dbml-local_2.10 0.2.2-db1-spark2.2
com.databricks dbml-local_2.10-tests 0.2.2-db1-spark2.2
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.10 0.4.15-9
com.databricks.scalapb scalapb-runtime_2.10 0.4.15-9
com.esotericsoftware kryo-shaded 3.0.3
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.0.0
com.fasterxml.jackson.core jackson-annotations 2.6.7
com.fasterxml.jackson.core jackson-core 2.6.7
com.fasterxml.jackson.core jackson-databind 2.6.7.1
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.6.7
com.fasterxml.jackson.datatype jackson-datatype-joda 2.6.7
com.fasterxml.jackson.module jackson-module-paranamer 2.6.7
com.fasterxml.jackson.module jackson-module-scala_2.10 2.6.7.1
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.rwl jtransforms 2.4.0
com.google.code.findbugs jsr305 2.0.1
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.h2database h2 1.3.174
com.jamesmurty.utils java-xmlbuilder 1.0
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.mchange c3p0 0.9.5.1
com.mchange mchange-commons-java 0.2.10
com.microsoft.azure azure-data-lake-store-sdk 2.0.11
com.microsoft.sqlserver mssql-jdbc 6.1.0.jre8
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.thoughtworks.paranamer paranamer 2.6
com.trueaccord.lenses lenses_2.10 0.3
com.twitter chill-java 0.8.0
com.twitter chill_2.10 0.8.0
com.twitter parquet-hadoop-bundle 1.6.0
com.twitter util-app_2.10 6.23.0
com.twitter util-core_2.10 6.23.0
com.twitter util-jvm_2.10 6.23.0
com.typesafe config 1.2.1
com.typesafe scalalogging-slf4j_2.10 1.1.0
com.univocity univocity-parsers 2.2.1
com.vlkan flatbuffers 1.2.0-3f79e055
com.zaxxer HikariCP 2.4.1
commons-beanutils commons-beanutils 1.7.0
commons-beanutils commons-beanutils-core 1.8.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 2.2
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.7
io.dropwizard.metrics metrics-core 3.1.2
io.dropwizard.metrics metrics-ganglia 3.1.2
io.dropwizard.metrics metrics-graphite 3.1.2
io.dropwizard.metrics metrics-healthchecks 3.1.2
io.dropwizard.metrics metrics-jetty9 3.1.2
io.dropwizard.metrics metrics-json 3.1.2
io.dropwizard.metrics metrics-jvm 3.1.2
io.dropwizard.metrics metrics-log4j 3.1.2
io.dropwizard.metrics metrics-servlets 3.1.2
io.netty netty 3.9.9.Final
io.netty netty-all 4.0.43.Final
io.prometheus simpleclient 0.0.16
io.prometheus simpleclient_common 0.0.16
io.prometheus simpleclient_dropwizard 0.0.16
io.prometheus simpleclient_servlet 0.0.16
io.prometheus.jmx collector 0.7
javax.activation activation 1.1.1
javax.annotation javax.annotation-api 1.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.validation validation-api 1.1.0.Final
javax.ws.rs javax.ws.rs-api 2.0.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.11
joda-time joda-time 2.9.3
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
mx4j mx4j 3.0.2
net.hydromatic eigenbase-properties 1.1.5
net.iharder base64 2.3.8
net.java.dev.jets3t jets3t 0.9.3
net.jpountz.lz4 lz4 1.3.0
net.razorvine pyrolite 4.13
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt oncrpc 1.0.7
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr antlr4-runtime 4.5.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 0.4.0
org.apache.arrow arrow-memory 0.4.0
org.apache.arrow arrow-vector 0.4.0
org.apache.avro avro 1.7.7
org.apache.avro avro-ipc 1.7.7
org.apache.avro avro-ipc-tests 1.7.7
org.apache.avro avro-mapred-hadoop2 1.7.7
org.apache.calcite calcite-avatica 1.2.0-incubating
org.apache.calcite calcite-core 1.2.0-incubating
org.apache.calcite calcite-linq4j 1.2.0-incubating
org.apache.commons commons-compress 1.4.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.5
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.6.0
org.apache.curator curator-framework 2.6.0
org.apache.curator curator-recipes 2.6.0
org.apache.derby derby 10.10.2.0
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.3
org.apache.hadoop hadoop-auth 2.7.3
org.apache.hadoop hadoop-client 2.7.3
org.apache.hadoop hadoop-common 2.7.3
org.apache.hadoop hadoop-hdfs 2.7.3
org.apache.hadoop hadoop-mapreduce-client-app 2.7.3
org.apache.hadoop hadoop-mapreduce-client-common 2.7.3
org.apache.hadoop hadoop-mapreduce-client-core 2.7.3
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.3
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.3
org.apache.hadoop hadoop-yarn-api 2.7.3
org.apache.hadoop hadoop-yarn-client 2.7.3
org.apache.hadoop hadoop-yarn-common 2.7.3
org.apache.hadoop hadoop-yarn-server-common 2.7.3
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.2
org.apache.httpcomponents httpcore 4.4.4
org.apache.ivy ivy 2.4.0
org.apache.parquet parquet-column 1.8.2-databricks1
org.apache.parquet parquet-common 1.8.2-databricks1
org.apache.parquet parquet-encoding 1.8.2-databricks1
org.apache.parquet parquet-format 2.3.1
org.apache.parquet parquet-hadoop 1.8.2-databricks1
org.apache.parquet parquet-jackson 1.8.2-databricks1
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.9.3
org.apache.xbean xbean-asm5-shaded 4.4
org.apache.zookeeper zookeeper 3.4.6
org.bouncycastle bcprov-jdk15on 1.51
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.0
org.codehaus.janino janino 3.0.0
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.3.11.v20160721
org.eclipse.jetty jetty-continuation 9.3.11.v20160721
org.eclipse.jetty jetty-http 9.3.11.v20160721
org.eclipse.jetty jetty-io 9.3.11.v20160721
org.eclipse.jetty jetty-jndi 9.3.11.v20160721
org.eclipse.jetty jetty-plus 9.3.11.v20160721
org.eclipse.jetty jetty-proxy 9.3.11.v20160721
org.eclipse.jetty jetty-security 9.3.11.v20160721
org.eclipse.jetty jetty-server 9.3.11.v20160721
org.eclipse.jetty jetty-servlet 9.3.11.v20160721
org.eclipse.jetty jetty-servlets 9.3.11.v20160721
org.eclipse.jetty jetty-util 9.3.11.v20160721
org.eclipse.jetty jetty-webapp 9.3.11.v20160721
org.eclipse.jetty jetty-xml 9.3.11.v20160721
org.fusesource.jansi jansi 1.4
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.4.0-b34
org.glassfish.hk2 hk2-locator 2.4.0-b34
org.glassfish.hk2 hk2-utils 2.4.0-b34
org.glassfish.hk2 osgi-resource-locator 1.0.1
org.glassfish.hk2.external aopalliance-repackaged 2.4.0-b34
org.glassfish.hk2.external javax.inject 2.4.0-b34
org.glassfish.jersey.bundles.repackaged jersey-guava 2.22.2
org.glassfish.jersey.containers jersey-container-servlet 2.22.2
org.glassfish.jersey.containers jersey-container-servlet-core 2.22.2
org.glassfish.jersey.core jersey-client 2.22.2
org.glassfish.jersey.core jersey-common 2.22.2
org.glassfish.jersey.core jersey-server 2.22.2
org.glassfish.jersey.media jersey-media-jaxb 2.22.2
org.hibernate hibernate-validator 5.1.1.Final
org.iq80.snappy snappy 0.2
org.javassist javassist 3.18.1-GA
org.jboss.logging jboss-logging 3.1.3.GA
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.jpmml pmml-model 1.2.15
org.jpmml pmml-schema 1.2.15
org.json4s json4s-ast_2.10 3.2.11
org.json4s json4s-core_2.10 3.2.11
org.json4s json4s-jackson_2.10 3.2.11
org.mariadb.jdbc mariadb-java-client 2.1.2
org.mockito mockito-all 1.9.5
org.objenesis objenesis 2.1
org.postgresql postgresql 9.4-1204-jdbc41
org.roaringbitmap RoaringBitmap 0.5.11
org.rocksdb rocksdbjni 5.2.1
org.rosuda.REngine REngine 2.1.0
org.scala-lang jline 2.10.6
org.scala-lang scala-compiler_2.10 2.10.6
org.scala-lang scala-library_2.10 2.10.6
org.scala-lang scala-reflect_2.10 2.10.6
org.scala-lang scalap_2.10 2.10.6
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.10 1.12.5
org.scalamacros quasiquotes_2.10 2.0.0
org.scalanlp breeze-macros_2.10 0.13.2
org.scalanlp breeze_2.10 0.13.2
org.scalatest scalatest_2.10 2.2.6
org.slf4j jcl-over-slf4j 1.7.16
org.slf4j jul-to-slf4j 1.7.16
org.slf4j slf4j-api 1.7.16
org.slf4j slf4j-log4j12 1.7.16
org.spark-project.hive hive-beeline 1.2.1.spark2
org.spark-project.hive hive-cli 1.2.1.spark2
org.spark-project.hive hive-exec 1.2.1.spark2
org.spark-project.hive hive-jdbc 1.2.1.spark2
org.spark-project.hive hive-metastore 1.2.1.spark2
org.spark-project.spark unused 1.0.0
org.spire-math spire-macros_2.10 0.13.0
org.spire-math spire_2.10 0.13.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.0
org.typelevel machinist_2.10 0.6.1
org.typelevel macro-compat_2.10 1.1.1
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.2.6
org.yaml snakeyaml 1.16
oro oro 2.0.8
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52

Pre-installed Java and Scala libraries (Scala 2.11 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.7.3
com.amazonaws aws-java-sdk-autoscaling 1.11.126
com.amazonaws aws-java-sdk-cloudformation 1.11.126
com.amazonaws aws-java-sdk-cloudfront 1.11.126
com.amazonaws aws-java-sdk-cloudhsm 1.11.126
com.amazonaws aws-java-sdk-cloudsearch 1.11.126
com.amazonaws aws-java-sdk-cloudtrail 1.11.126
com.amazonaws aws-java-sdk-cloudwatch 1.11.126
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.126
com.amazonaws aws-java-sdk-codedeploy 1.11.126
com.amazonaws aws-java-sdk-cognitoidentity 1.11.126
com.amazonaws aws-java-sdk-cognitosync 1.11.126
com.amazonaws aws-java-sdk-config 1.11.126
com.amazonaws aws-java-sdk-core 1.11.126
com.amazonaws aws-java-sdk-datapipeline 1.11.126
com.amazonaws aws-java-sdk-directconnect 1.11.126
com.amazonaws aws-java-sdk-directory 1.11.126
com.amazonaws aws-java-sdk-dynamodb 1.11.126
com.amazonaws aws-java-sdk-ec2 1.11.126
com.amazonaws aws-java-sdk-ecs 1.11.126
com.amazonaws aws-java-sdk-efs 1.11.126
com.amazonaws aws-java-sdk-elasticache 1.11.126
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.126
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.126
com.amazonaws aws-java-sdk-elastictranscoder 1.11.126
com.amazonaws aws-java-sdk-emr 1.11.126
com.amazonaws aws-java-sdk-glacier 1.11.126
com.amazonaws aws-java-sdk-iam 1.11.126
com.amazonaws aws-java-sdk-importexport 1.11.126
com.amazonaws aws-java-sdk-kinesis 1.11.126
com.amazonaws aws-java-sdk-kms 1.11.126
com.amazonaws aws-java-sdk-lambda 1.11.126
com.amazonaws aws-java-sdk-logs 1.11.126
com.amazonaws aws-java-sdk-machinelearning 1.11.126
com.amazonaws aws-java-sdk-opsworks 1.11.126
com.amazonaws aws-java-sdk-rds 1.11.126
com.amazonaws aws-java-sdk-redshift 1.11.126
com.amazonaws aws-java-sdk-route53 1.11.126
com.amazonaws aws-java-sdk-s3 1.11.126
com.amazonaws aws-java-sdk-ses 1.11.126
com.amazonaws aws-java-sdk-simpledb 1.11.126
com.amazonaws aws-java-sdk-simpleworkflow 1.11.126
com.amazonaws aws-java-sdk-sns 1.11.126
com.amazonaws aws-java-sdk-sqs 1.11.126
com.amazonaws aws-java-sdk-ssm 1.11.126
com.amazonaws aws-java-sdk-storagegateway 1.11.126
com.amazonaws aws-java-sdk-sts 1.11.126
com.amazonaws aws-java-sdk-support 1.11.126
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.126
com.amazonaws jmespath-java 1.11.126
com.carrotsearch hppc 0.7.1
com.chuusai shapeless_2.11 2.3.2
com.clearspring.analytics stream 2.7.0
com.databricks Rserve 1.8-3
com.databricks dbml-local_2.11 0.2.2-db1-spark2.2
com.databricks dbml-local_2.11-tests 0.2.2-db1-spark2.2
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.11 0.4.15-9
com.databricks.scalapb scalapb-runtime_2.11 0.4.15-9
com.esotericsoftware kryo-shaded 3.0.3
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.0.0
com.fasterxml.jackson.core jackson-annotations 2.6.7
com.fasterxml.jackson.core jackson-core 2.6.7
com.fasterxml.jackson.core jackson-databind 2.6.7.1
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.6.7
com.fasterxml.jackson.datatype jackson-datatype-joda 2.6.7
com.fasterxml.jackson.module jackson-module-paranamer 2.6.7
com.fasterxml.jackson.module jackson-module-scala_2.11 2.6.7.1
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.rwl jtransforms 2.4.0
com.google.code.findbugs jsr305 2.0.1
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.h2database h2 1.3.174
com.jamesmurty.utils java-xmlbuilder 1.0
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.mchange c3p0 0.9.5.1
com.mchange mchange-commons-java 0.2.10
com.microsoft.azure azure-data-lake-store-sdk 2.0.11
com.microsoft.sqlserver mssql-jdbc 6.1.0.jre8
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.thoughtworks.paranamer paranamer 2.6
com.trueaccord.lenses lenses_2.11 0.3
com.twitter chill-java 0.8.0
com.twitter chill_2.11 0.8.0
com.twitter parquet-hadoop-bundle 1.6.0
com.twitter util-app_2.11 6.23.0
com.twitter util-core_2.11 6.23.0
com.twitter util-jvm_2.11 6.23.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging-api_2.11 2.1.2
com.typesafe.scala-logging scala-logging-slf4j_2.11 2.1.2
com.univocity univocity-parsers 2.2.1
com.vlkan flatbuffers 1.2.0-3f79e055
com.zaxxer HikariCP 2.4.1
commons-beanutils commons-beanutils 1.7.0
commons-beanutils commons-beanutils-core 1.8.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 2.2
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.7
io.dropwizard.metrics metrics-core 3.1.2
io.dropwizard.metrics metrics-ganglia 3.1.2
io.dropwizard.metrics metrics-graphite 3.1.2
io.dropwizard.metrics metrics-healthchecks 3.1.2
io.dropwizard.metrics metrics-jetty9 3.1.2
io.dropwizard.metrics metrics-json 3.1.2
io.dropwizard.metrics metrics-jvm 3.1.2
io.dropwizard.metrics metrics-log4j 3.1.2
io.dropwizard.metrics metrics-servlets 3.1.2
io.netty netty 3.9.9.Final
io.netty netty-all 4.0.43.Final
io.prometheus simpleclient 0.0.16
io.prometheus simpleclient_common 0.0.16
io.prometheus simpleclient_dropwizard 0.0.16
io.prometheus simpleclient_servlet 0.0.16
io.prometheus.jmx collector 0.7
javax.activation activation 1.1.1
javax.annotation javax.annotation-api 1.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.validation validation-api 1.1.0.Final
javax.ws.rs javax.ws.rs-api 2.0.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.11
joda-time joda-time 2.9.3
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
mx4j mx4j 3.0.2
net.hydromatic eigenbase-properties 1.1.5
net.iharder base64 2.3.8
net.java.dev.jets3t jets3t 0.9.3
net.jpountz.lz4 lz4 1.3.0
net.razorvine pyrolite 4.13
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt oncrpc 1.0.7
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr antlr4-runtime 4.5.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 0.4.0
org.apache.arrow arrow-memory 0.4.0
org.apache.arrow arrow-vector 0.4.0
org.apache.avro avro 1.7.7
org.apache.avro avro-ipc 1.7.7
org.apache.avro avro-ipc-tests 1.7.7
org.apache.avro avro-mapred-hadoop2 1.7.7
org.apache.calcite calcite-avatica 1.2.0-incubating
org.apache.calcite calcite-core 1.2.0-incubating
org.apache.calcite calcite-linq4j 1.2.0-incubating
org.apache.commons commons-compress 1.4.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.5
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.6.0
org.apache.curator curator-framework 2.6.0
org.apache.curator curator-recipes 2.6.0
org.apache.derby derby 10.10.2.0
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.3
org.apache.hadoop hadoop-auth 2.7.3
org.apache.hadoop hadoop-client 2.7.3
org.apache.hadoop hadoop-common 2.7.3
org.apache.hadoop hadoop-hdfs 2.7.3
org.apache.hadoop hadoop-mapreduce-client-app 2.7.3
org.apache.hadoop hadoop-mapreduce-client-common 2.7.3
org.apache.hadoop hadoop-mapreduce-client-core 2.7.3
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.3
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.3
org.apache.hadoop hadoop-yarn-api 2.7.3
org.apache.hadoop hadoop-yarn-client 2.7.3
org.apache.hadoop hadoop-yarn-common 2.7.3
org.apache.hadoop hadoop-yarn-server-common 2.7.3
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.2
org.apache.httpcomponents httpcore 4.4.4
org.apache.ivy ivy 2.4.0
org.apache.parquet parquet-column 1.8.2-databricks1
org.apache.parquet parquet-common 1.8.2-databricks1
org.apache.parquet parquet-encoding 1.8.2-databricks1
org.apache.parquet parquet-format 2.3.1
org.apache.parquet parquet-hadoop 1.8.2-databricks1
org.apache.parquet parquet-jackson 1.8.2-databricks1
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.9.3
org.apache.xbean xbean-asm5-shaded 4.4
org.apache.zookeeper zookeeper 3.4.6
org.bouncycastle bcprov-jdk15on 1.51
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.0
org.codehaus.janino janino 3.0.0
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.3.11.v20160721
org.eclipse.jetty jetty-continuation 9.3.11.v20160721
org.eclipse.jetty jetty-http 9.3.11.v20160721
org.eclipse.jetty jetty-io 9.3.11.v20160721
org.eclipse.jetty jetty-jndi 9.3.11.v20160721
org.eclipse.jetty jetty-plus 9.3.11.v20160721
org.eclipse.jetty jetty-proxy 9.3.11.v20160721
org.eclipse.jetty jetty-security 9.3.11.v20160721
org.eclipse.jetty jetty-server 9.3.11.v20160721
org.eclipse.jetty jetty-servlet 9.3.11.v20160721
org.eclipse.jetty jetty-servlets 9.3.11.v20160721
org.eclipse.jetty jetty-util 9.3.11.v20160721
org.eclipse.jetty jetty-webapp 9.3.11.v20160721
org.eclipse.jetty jetty-xml 9.3.11.v20160721
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.4.0-b34
org.glassfish.hk2 hk2-locator 2.4.0-b34
org.glassfish.hk2 hk2-utils 2.4.0-b34
org.glassfish.hk2 osgi-resource-locator 1.0.1
org.glassfish.hk2.external aopalliance-repackaged 2.4.0-b34
org.glassfish.hk2.external javax.inject 2.4.0-b34
org.glassfish.jersey.bundles.repackaged jersey-guava 2.22.2
org.glassfish.jersey.containers jersey-container-servlet 2.22.2
org.glassfish.jersey.containers jersey-container-servlet-core 2.22.2
org.glassfish.jersey.core jersey-client 2.22.2
org.glassfish.jersey.core jersey-common 2.22.2
org.glassfish.jersey.core jersey-server 2.22.2
org.glassfish.jersey.media jersey-media-jaxb 2.22.2
org.hibernate hibernate-validator 5.1.1.Final
org.iq80.snappy snappy 0.2
org.javassist javassist 3.18.1-GA
org.jboss.logging jboss-logging 3.1.3.GA
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.jpmml pmml-model 1.2.15
org.jpmml pmml-schema 1.2.15
org.json4s json4s-ast_2.11 3.2.11
org.json4s json4s-core_2.11 3.2.11
org.json4s json4s-jackson_2.11 3.2.11
org.mariadb.jdbc mariadb-java-client 2.1.2
org.mockito mockito-all 1.9.5
org.objenesis objenesis 2.1
org.postgresql postgresql 9.4-1204-jdbc41
org.roaringbitmap RoaringBitmap 0.5.11
org.rocksdb rocksdbjni 5.2.1
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.11 2.11.8
org.scala-lang scala-library_2.11 2.11.8
org.scala-lang scala-reflect_2.11 2.11.8
org.scala-lang scalap_2.11 2.11.8
org.scala-lang.modules scala-parser-combinators_2.11 1.0.2
org.scala-lang.modules scala-xml_2.11 1.0.2
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.11 1.12.5
org.scalanlp breeze-macros_2.11 0.13.2
org.scalanlp breeze_2.11 0.13.2
org.scalatest scalatest_2.11 2.2.6
org.slf4j jcl-over-slf4j 1.7.16
org.slf4j jul-to-slf4j 1.7.16
org.slf4j slf4j-api 1.7.16
org.slf4j slf4j-log4j12 1.7.16
org.spark-project.hive hive-beeline 1.2.1.spark2
org.spark-project.hive hive-cli 1.2.1.spark2
org.spark-project.hive hive-exec 1.2.1.spark2
org.spark-project.hive hive-jdbc 1.2.1.spark2
org.spark-project.hive hive-metastore 1.2.1.spark2
org.spark-project.spark unused 1.0.0
org.spire-math spire-macros_2.11 0.13.0
org.spire-math spire_2.11 0.13.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.0
org.typelevel machinist_2.11 0.6.1
org.typelevel macro-compat_2.11 1.1.1
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.2.6
org.yaml snakeyaml 1.16
oro oro 2.0.8
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52