Connecting Spark to HAWQ via JDBC driver


November 2018


1.3k time


Trying to connect to HAWQ from Spark, using greenplum's odbc/jdbc drivers (downloaded from the proper Pivotal page).

Using Spark 1.4, here's the sample code written in python: (All capitals have proper variable assignments) ...

from pyspark import SparkContext
from pyspark.sql import SQLContext

sc = SparkContext()
sqlContext = SQLContext(sc)

df ='jdbc',\


Spark submit command appends the odbc driver to the classpath. I've done a 'hello world' with the basic sqlContext instantiation and everything runs fine on the cluster. But when I try to actually connect to the HAWQ postgresql db it will not run.


py4j.protocol.Py4JJavaError: An error occurred while calling o24.load.
: java.util.NoSuchElementException: key not found: path
        at scala.collection.MapLike$class.default(MapLike.scala:228)
        at org.apache.spark.sql.sources.CaseInsensitiveMap.default(ddl.scala:467)
        at scala.collection.MapLike$class.apply(MapLike.scala:141)
        at org.apache.spark.sql.sources.CaseInsensitiveMap.apply(ddl.scala:467)
        at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:273)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(
        at java.lang.reflect.Method.invoke(
        at py4j.reflection.MethodInvoker.invoke(
        at py4j.reflection.ReflectionEngine.invoke(
        at py4j.Gateway.invoke(
        at py4j.commands.AbstractCommand.invokeMethod(
        at py4j.commands.CallCommand.execute(

Any thoughts or suggestions? I've tried at least 20 combinations of the "df =" definition but to no avail.

1 answers


Правильный формат для JDBC должно быть что-то вроде этого

df ='jdbc').options(url='jdbc:postgresql://node1:5432/postgres?user=myuser&password=password',dbtable='test').load()

Другой пример здесь