Questions tagged [scala]

48864 questions
1

votes
0

answer
204

Views

Scala reflection error while registering a Spark UDF

I'm using Spark UDFs all over my code, but there is one registration the fails intermittently with the following error: scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving package at scala.reflect.internal.Symbols$TypeSymbol.tpe(Symbols.scala:2768) at scala.reflect.in...
Hagai
1

votes
0

answer
117

Views

How to find implicit function(s) using Scala metaprogramming?

In scala language, the implicit scope of a class is erased in runtime, e.g. if defining: case class A(i: Int) object A { implicit def toInt(a: A) = a.i } Then for an array: val arr = Array[Any](A(1), 2) it is impossible to write: arr.map(_ + 3) as type A and its implicit scope is removed after being...
tribbloid
1

votes
1

answer
48

Views

Scala: Printing the ouput in proper data format using scala

I would like to display the data in proper format, I have the below code val maplist=List(Map('id' -> '1', 'Name' -> 'divya'), Map('id' -> '2', 'Name' -> 'gaya') ) val header=maplist.flatMap(_.keys).distinct val data=maplist.map(_.values) println(header.mkString(' ')) data.foreach(x => println(x.mk...
lak
0

votes
0

answer
2

Views

Scala Spark Dataset Error on Nested Object

I am trying to test dataframe(dataset) code with strongly typed nested case classes into dataframe to then pass over my functions. The serialize/creation of the dataframe keeps failing and I do not have enough experience to know what is going on in scala or spark. I think that I am trying to determi...
vfrank66
1

votes
0

answer
8

Views

CreateDirectStream with messages avro

In a first moment, I had to process the information from a text file: C1_4,C2_4,C1______10,01/12/2015,30/12/2015,123456789,S,12345 Now, I need to process the same information but in format avro. How can I do it ? Before I used this code: createDirectStream[String, String, StringDecoder, StringDecode...
user2140391
1

votes
0

answer
205

Views

Sbt ( new version 1.0.4) assembly failure

I have been trying to build fat jar for some time now. I got assembly.sbt in project folder and it looks like below addSbtPlugin('com.eed3si9n' % 'sbt-assembly' % '0.14.6') and my build.sbt looks like below name := 'cool' version := '0.1' scalaVersion := '2.11.8' resolvers += 'Hortonworks Repositor...
ram
1

votes
1

answer
74

Views

Removing elements from LinkedHashMap and ListBuffer of ListBuffer, used as vertex attributes into a graph structure

I have built a graph in Scala using the GraphX API. In my graph, each vertex has a LinkedHashMap[Int, ListBuffer[ListBuffer[Int]]] as attribute: every couple (key, value) of the LinkedHashMap represents: key: id of a node - as Int value: all the possible paths to reach the node - every path is a Lis...
Andrea
1

votes
1

answer
85

Views

separate select clause and insert clause from sql string

I am trying to solve spark sql small files problem using the above approach , in my organization users pass in sql statements for example. insert overwrite table x partition (column1) select * from y; I can run the above sql in spark sql problem is it creates small files when compared with hive whic...
1

votes
1

answer
571

Views

Preserving Manifest.mf when building via sbt assembly

When I build my project using sbt assembly, I get java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF. The solution that google recommends, is to use MergeStrategy.discard. That works in getting the project to build - but it crashes at runtime, because Dropwizard (a dependency) relies...
Click Upvote
1

votes
1

answer
308

Views

Is akka really a solution for a deadlock?

As Described here While locks seem to be the natural remedy to uphold encapsulation with multiple threads, in practice, they are inefficient and easily lead to deadlocks in any application of real-world scale. My question simply, is akka really a solution for a deadlock. I know as long as akka actor...
user2810472
1

votes
0

answer
68

Views

Calculate depart flights from sorted data using Spark

I've a dataset of flights in the form of +----------------+----------+-------------+ |flightID |depart_ts |arrival_ts | +----------------+----------+-------------+ |1 |1451603468| 1451603468| |2 |1451603468| 1451603468| |3 |1451603468| 1451603...
Assem
1

votes
0

answer
206

Views

Spark: join with a cached RDD takes too long

I have a key-value RDD and I need to join several keysets with this RDD. Key-value RDD is big (100GB), keysets are relatively small (but not small enough to broadcast it) I assign the same partitioner to all RDDs and call join. Expected behavior: After repartitioning all the data to be joined is col...
simpadjo
1

votes
0

answer
180

Views

Slick3.2 Error: No matching Shape found

I'm not sure what is wrong here. The following code block is throwing error: (for { (e,r) r.col1 === e.col3) } yield (e.id) Error No matching Shape found. [error] Slick does not know how to map the given types. [error] Possible causes: T in Table[T] does not match your * projection, [error] you us...
Sujit Baniya
1

votes
0

answer
295

Views

Spark (Scala) - how to optimize objective function parameters

I have function f going from R2 to R which takes 2 parameters (a and b) and returns a scalar. I would like to use an optimizer to estimate the values of a and b for which the value returned by f is maximized (or minimized, I can work with -f). I have looked into the LBFGS optimizer from mllib, see:...
Raphvanns
1

votes
0

answer
743

Views

How to inject json coming from feeder inside body with gatling

I have a gatling scenario which create a Http object create a post request with a body set up the scenario I can inject a body from a json file containing this: [{'id':'1'}] At this point I have no problem. My question is on the second point. I want users to have theirs body feed with different id....
a.moussa
1

votes
1

answer
1.3k

Views

Did not find value which can be converted into java.lang.String

I have a JSON input which is to be written into Hbase an example of this is { 'contributors': null, 'truncated': false, 'text': '#Care #MichelleNunn #Stop #Pimping #Haiti #HurricaneMatthew #Give #Answers #Blacklivesmatter https:\/\/t.co\/mmlOawLdCy', 'is_quote_status': false, 'in_reply_to_status_id...
Ram
1

votes
1

answer
139

Views

Play json absent-sensitive Reads

I'd like to have a reads like Reads[Patch[T]] for a case class like this sealed trait Patch[+T] case class Update[+T](value: T) extends Patch[T] case object Delete extends Patch[Nothing] case object Ignore extends Patch[Nothing] where a missing json value reads to Ignore, a null json value reads to...
kag0
1

votes
1

answer
431

Views

Get date months with iteration over Spark dataframe

I have a problem case to iterate for last 36 months based on an input date. Currently using Scala, through a DataFrame I am getting the max value of a timestamp field. For example: val vGetDate = hc.read.format('filodb.spark').option('database','YYYYY').option('dataset','XXX').load().agg(max('inv_da...
Rajdip
1

votes
1

answer
206

Views

overloaded method value aggregate with alternatives

I have following function, that does not compile: private def save(pea: KStream[String, String]): Unit = { pea .groupByKey() .aggregate(() => '''{folder: ''}''', (_: String, _: String, value: String) => value, EventStoreTopology.Store) } the error message is: [error] [VR](x$1: org.apache.kafka.str...
zero_coding
1

votes
1

answer
25

Views

not able to store result in hdfs when code runs for second iteration

Well I am new to spark and scala and have been trying to implement cleaning of data in spark. below code checks for the missing value for one column and stores it in outputrdd and runs loops for calculating missing value. code works well when there is only one missing value in file. Since hdfs does...
Yogesh Patel
1

votes
0

answer
42

Views

Log file is not generated after running JAR

I have logback.xml file saved in src/main/resources of my Maven project. The path where the log file should be saved is defined as follows: file>/my/path/myLog.log. I package the JAR file using IntelliJ IDEA. When I run this JAR file, myLog.log does not appear in /my/path/. The problem is that every...
Markus
1

votes
2

answer
107

Views

Scala - composing generic types

I want to create a generic interface which hides the implementation of the hash function on the key of a product (which is used to lookup products and for various caching and calculations outside the scope of the product itself). I want to tie the concrete implementation of the hash function to the...
Avner Barr
1

votes
0

answer
45

Views

How to calculate connections of the node in Spark 2

I have the following DataFrame df: val df = Seq( (1, 0, 1, 0, 0), (1, 4, 1, 0, 4), (2, 2, 1, 2, 2), (4, 3, 1, 4, 4), (4, 5, 1, 4, 4) ).toDF('from', 'to', 'attr', 'type_from', 'type_to') +-----+-----+----+---------------+---------------+ |from |to |attr|type_from |type_to | +-----+-----...
Markus
1

votes
1

answer
1.3k

Views

How can I install older version (0.13.x) sbt on my MAC?

I need sbt version 0.13.x for a course in Scala and I am having difficulties installing it. I tried with the 'brew install [email protected]' command, but when I then check with 'sbt about' it gives a message 'No such file or directory'.  However, if I again try and do 'brew install [email protected]', it gives 'War...
kris
1

votes
1

answer
83

Views

Unable to override java generic function with scala parameterized type

I am getting the error - Method read has incompatible type ; when I try to override a java generic function name with a Scala parameterized type. This is the java abstract function I am trying to override. public abstract class SRType{ //Other abstract functions public abstract T read()throws IOEx...
Pratyush Das
1

votes
0

answer
215

Views

Using JBDC to read sql file in spark scala collecting Warehouse error

I am trying to read MySQL file using Spark Scala. Following is the code I tried val dataframe_mysql = sqlContext.read.format('jdbc') .option('url','jdbc:mysql://xx.xx.xx.xx:xx') .option('driver', 'com.mysql.jdbc.Driver') .option('dbtable', 'schema.xxxx') .option('user', 'xxxx').option('password', '...
Fouad Haddud
1

votes
2

answer
1.3k

Views

change a dataframe row value with dynamic number of columns spark scala

I have a dataframe (contains 10 columns) for which I want to change the value of a row (for the last column only). I have written following code for this: val newDF = spark.sqlContext.createDataFrame(WRADF.rdd.map(r=> { Row(r.get(0), r.get(1), r.get(2), r.get(3), r.get(4), r.get(5), r.get(6), r.get(...
Pardeep Naik
1

votes
0

answer
64

Views

How to handle error on batch tasks in flink

In my flink job there is a specific task 'GroupReduce' that involve external system (elasticsearch in this case) that can encounter an timeout exception. A simple try and catch is not enought because i don't want to just skip the groupreduce task if an error occure. There is a way to, once cought th...
Michael Sogos
1

votes
1

answer
345

Views

Using Spark CombineByKey with set of values

I have the following dataset: val data = sc.parallelize(Array( ('group1','value1'),('group1','value2'),('group1', 'value1'), ('group2','value1'),('group1','value1'),('group3', 'value3') )) I'm using Spark, and I would like to obtain the following result: Array(('group1','value1',3),('group1', 'value...
Pep's
1

votes
0

answer
139

Views

How do I inject ReactiveMongoApi trait in a Repository with macwire?

I'm new to Play framework and all the Scala stack. I'm trying to implement dependency injection using macwire and ReactiveMongo to handle my MongoDB database. I have this classes to define repositories. trait BaseRepository extends ReactiveMongoComponents { /** * collection to retrieve documents fro...
Marcos
1

votes
0

answer
69

Views

Covariance annotation raises incorrect implicit ambiguity error

The code below raises an implicit not found error that the -Xlog-implicits sbt option explains as an ambiguity, which I pasted at the end: import scala.language.higherKinds trait JsonOFormat[T] trait BsonOFormat[T] trait Codec[CTXO[_]] implicit def jsonCodec: Codec[JsonOFormat] = new Codec[JsonOForm...
shayan
1

votes
0

answer
66

Views

Scala Swing scrollbar for TextArea

I'm trying to add a scrollbar to a text area component. I could use FlowPanel, but I need to use GridBagPanel for my other components and when I tried to add a flowpanel inside the gridbagpanel it was empty and like 2x2 pixels. Is there a way to add two panels to a MainFrame or add the scrollbar so...
Alqio
1

votes
0

answer
286

Views

Convert an Array[org.apache.spark.sql.Row] to a DenseMatrix[Double] in scala

I have this solution which works but I wonder if there's a clever way to do it, avoiding the explicit for loop and the toString.toDouble conversion import scala.collection.mutable.ArrayBuffer val rows: Array[Row] = df2.collect() //rows: Array[org.apache.spark.sql.Row] = Array([-1.35980713367,... var...
MLeiria
1

votes
0

answer
145

Views

MongoController with Play

I am using Play! Framework with current version and i am trying to integrate with MongoDB via ReactiveMongo, but I am getting following error 'illegal inheritance; superclass Object is not a subclass of the superclass AbstractController of the mixin trait MongoController' Here is my controller code;...
amit
1

votes
0

answer
228

Views

Getting “null is disabled” warning for log messages with scala wart remover

Wart remover complains about log messages. Why does it complain? How to get rid of wart remover warnings for log messages? [wartremover:Null] null is disabled [warn] logger.error(e) [warn] ^
Gowrav
1

votes
0

answer
325

Views

Spark CosmosDB Sink: org.apache.spark.sql.AnalysisException: 'write' can not be called on streaming Dataset/DataFrame

I am reading a data stream from Event Hub in Spark (using Databricks). My goal is to be able to write the streamed data to a CosmosDB. However I get the following error: org.apache.spark.sql.AnalysisException: 'write' can not be called on streaming Dataset/DataFrame. Is this scenario not supported?...
fbeltrao
1

votes
0

answer
560

Views

Spark job dataframe write to Oracle using jdbc failing

When writing spark dataframe to Oracle database (Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit), the spark job is failing with the exception java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection. The scala code is dataFrame.write.mode(...
Abhishek Joshi
1

votes
1

answer
285

Views

How to create an Akka flow with backpressure and Control

I need to create a function with the following Interface: import akka.kafka.scaladsl.Consumer.Control object ItemConversionFlow { def build(config: StreamConfig): Flow[Item, OtherItem, Control] = { // Implementation goes here } My problem is that I don't know how to define the flow in a way that it...
Stoecki
1

votes
0

answer
159

Views

How to perform Welch's Ttest in Spark 2.0.0 using StreamingTest

I want to try Welch's T-test in Spark 2.0.0 As I know I can use StreamingTest() from mllib on this website. [https://spark.apache.org/docs/2.0.0/api/scala/index.html#org.apache.spark.mllib.stat.test.StreamingTest] this is my code in spark-shell import org.apache.spark.mllib.stat.test.{BinarySample,...
Jchanho
1

votes
0

answer
73

Views

Setting 'testFrameworks' in build.sbt of a scalajs project causes java.lang.NullPointerException when running test

I am new to scala and scalajs. I am trying to add unit tests to my scalajs project using minitest. I followed instructions at https://github.com/monix/minitest#usage-in-sbt, so this is what my build.sbt looks like: enablePlugins(ScalaJSPlugin, WorkbenchPlugin) name := 'Test' version := '0.1-SNAPSHOT...
qcabeza

View additional questions