Questions tagged [spark-notebook]

1

votes
0

answer
54

Views

Spark Scala - Connect to MySQL over SSH using Key Pair

I wanted to understand if there is a method one can connect to MySQL database over SSH using Private-Public Key pair using Spark notebook for scala? I have been trying to modify this code to no avail Connect to MySQL over SSH using Java
Leothorn
0

votes
0

answer
3

Views

How to import one databricks notebook into another?

I have a python notebook A in Azure Databricks having import statement as below: import xyz, datetime, ... I have another notebook xyz being imported in notebook A as shown in above code. When I run notebook A, it throws the following error: ImportError: No module named xyz Both notebooks are in...
user39602
1

votes
1

answer
212

Views

Scheduler for jobs executing Apache Spark SQL on Bluemix

I am using Apache Spark in Bluemix. I want to implement scheduler for sparksql jobs. I saw this link to a blog that describes scheduling. But its not clear how do I update the manifest. Maybe there is some other way to schedule my jobs.
Yakov
1

votes
2

answer
4.6k

Views

Evaluating Spark-Notebook

I am evaluating Spark Notebook and found three different products; 1. Hue 3.9 comes with Spark notebook (beta) 2. Apache zeppelin 3. andypetrella/spark-notebook. Can you please help me understand pros and cons of each product Thanks Pani
Pani Dhakshnamurthy
2

votes
2

answer
531

Views

How to connect Spark-Notebook to Hive metastore?

This is a cluster with Hadoop 2.5.0, Spark 1.2.0, Scala 2.10, provided by CDH 5.3.2. I used a compiled spark-notebook distro It seems Spark-Notebook cannot find the Hive metastore by default. How to specify the location of hive-site.xml for spark-notebook so that it can load the Hive metastore? Here...
Rex
3

votes
1

answer
1.7k

Views

Is it possible to embed the HTML output of a Zeppelin Notebook so that the output can be looked at when the server hosting the Notebook isn't active?

I have a Zeppelin Notebook producing interactive graphs. I don't want to have to host the notebook indefinitely but I want to have that interactive output appear on another website. I understand that I can 'link to this paragraph' and then embed the output in an iframe, but that requires the noteboo...
3

votes
0

answer
43

Views

Spark notebooks is quicker than executing a jar

I have finished some code in spark notebook, I tried to move it into a real project, and use sbt to generate a jar, then use the spark-submit to execute it. Problem: It takes just 10 minutes to get the result in spark notebooks, but it takes almost 3 hours to get the result when I use the command sp...
Leyla Lee
2

votes
0

answer
530

Views

p.nettyException - Exception caught in Netty java.lang.NoSuchMethodError:

I compiled spark-notebook from sources and am getting an error when trying to run it. Something is wrong with netty version. Well, there are a lot of components in spark-notebooks. And those components require different netty versions. I tried to force sbt to use some specific version like libraryDe...
Mike Pakhomov
7

votes
3

answer
2k

Views

How to import libraries in Spark Notebook

I'm having trouble importing magellan-1.0.4-s_2.11 in spark notebook. I've downloaded the jar from https://spark-packages.org/package/harsha2010/magellan and have tried placing SPARK_HOME/bin/spark-shell --packages harsha2010:magellan:1.0.4-s_2.11 in the Start of Customized Settings section of the s...
Curtis Chong
3

votes
2

answer
431

Views

odd error when populating accumulo 1.6 mutation object via spark-notebook

using spark-notebook to update an accumulo table. employing the method specified in both the accumulo documentation and the accumulo example code. Below is verbatim what I put into notebook, and the responses: val clientRqrdTble = new ClientOnRequiredTable val bwConfig = new BatchWriterConfig val ba...
David Holiday
1

votes
3

answer
13.7k

Views

What are SparkSession Config Options

I am trying to use SparkSession to convert JSON data of a file to RDD with Spark Notebook. I already have the JSON file. val spark = SparkSession .builder() .appName('jsonReaderApp') .config('config.key.here', configValueHere) .enableHiveSupport() .getOrCreate() val jread = spark.read.json('search-r...
Sha2b
4

votes
3

answer
540

Views

How to run spark-notebook on docker on MacOS X?

Running the spark-notebook using docker on OSX (via boot2docker) doesn't seem to do anything. Here's the output [email protected]:~/apps/spark-notebook$ docker run -p 9000:9000 andypetrella/spark-notebook:0.1.4-spark-1.2.0-hadoop-1.0.4 Play server process ID is 1 SLF4J: Class path contains multiple SLF4J bi...
juniper-
4

votes
0

answer
418

Views

Want to run Spark(scala) kernel inside Jupyter Notebook. Getting OSError: [WinError 193] %1 is not a valid Win32 application

Traceback (most recent call last): File 'c:\users\rdx\anaconda3\lib\runpy.py', line 184, in _run_module_as_main '__main__', mod_spec) File 'c:\users\rdx\anaconda3\lib\runpy.py', line 85, in _run_code exec(code, run_globals) File 'C:\Users\RDX\Anaconda3\Scripts\ipython.exe\__main__.py', line 9, in F...
Darshan
5

votes
2

answer
4k

Views

How do I create a Spark RDD from Accumulo 1.6 in spark-notebook?

I have a Vagrant image with Spark Notebook, Spark, Accumulo 1.6, and Hadoop all running. From notebook, I can manually create a Scanner and pull test data from a table I created using one of the Accumulo examples: val instanceNameS = 'accumulo' val zooServersS = 'localhost:2181' val instance: Insta...
David Holiday