Questions tagged [scala]
48864 questions
1
votes
0
answer
204
Views
Scala reflection error while registering a Spark UDF
I'm using Spark UDFs all over my code, but there is one registration the fails intermittently with the following error:
scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving package
at scala.reflect.internal.Symbols$TypeSymbol.tpe(Symbols.scala:2768)
at scala.reflect.in...
1
votes
0
answer
117
Views
How to find implicit function(s) using Scala metaprogramming?
In scala language, the implicit scope of a class is erased in runtime, e.g. if defining:
case class A(i: Int)
object A {
implicit def toInt(a: A) = a.i
}
Then for an array:
val arr = Array[Any](A(1), 2)
it is impossible to write:
arr.map(_ + 3)
as type A and its implicit scope is removed after being...
1
votes
1
answer
48
Views
Scala: Printing the ouput in proper data format using scala
I would like to display the data in proper format, I have the below code
val maplist=List(Map('id' -> '1', 'Name' -> 'divya'),
Map('id' -> '2', 'Name' -> 'gaya')
)
val header=maplist.flatMap(_.keys).distinct
val data=maplist.map(_.values)
println(header.mkString(' '))
data.foreach(x => println(x.mk...
0
votes
0
answer
2
Views
Scala Spark Dataset Error on Nested Object
I am trying to test dataframe(dataset) code with strongly typed nested case classes into dataframe to then pass over my functions. The serialize/creation of the dataframe keeps failing and I do not have enough experience to know what is going on in scala or spark.
I think that I am trying to determi...
1
votes
0
answer
8
Views
CreateDirectStream with messages avro
In a first moment, I had to process the information from a text file:
C1_4,C2_4,C1______10,01/12/2015,30/12/2015,123456789,S,12345
Now, I need to process the same information but in format avro. How can I do it ?
Before I used this code:
createDirectStream[String, String, StringDecoder, StringDecode...
1
votes
0
answer
205
Views
Sbt ( new version 1.0.4) assembly failure
I have been trying to build fat jar for some time now. I got assembly.sbt in project folder and it looks like below
addSbtPlugin('com.eed3si9n' % 'sbt-assembly' % '0.14.6')
and my build.sbt looks like below
name := 'cool'
version := '0.1'
scalaVersion := '2.11.8'
resolvers += 'Hortonworks Repositor...
1
votes
1
answer
74
Views
Removing elements from LinkedHashMap and ListBuffer of ListBuffer, used as vertex attributes into a graph structure
I have built a graph in Scala using the GraphX API. In my graph, each vertex has a LinkedHashMap[Int, ListBuffer[ListBuffer[Int]]] as attribute: every couple (key, value) of the LinkedHashMap represents:
key: id of a node - as Int
value: all the possible paths to reach the node - every path is a Lis...
1
votes
1
answer
85
Views
separate select clause and insert clause from sql string
I am trying to solve spark sql small files problem using the above approach , in my organization users pass in sql statements for example.
insert overwrite table x partition (column1) select * from y;
I can run the above sql in spark sql problem is it creates small files when compared with hive whic...
1
votes
1
answer
571
Views
Preserving Manifest.mf when building via sbt assembly
When I build my project using sbt assembly, I get java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF. The solution that google recommends, is to use MergeStrategy.discard.
That works in getting the project to build - but it crashes at runtime, because Dropwizard (a dependency) relies...
1
votes
1
answer
308
Views
Is akka really a solution for a deadlock?
As Described here
While locks seem to be the natural remedy to uphold encapsulation with multiple threads, in practice, they are inefficient and easily lead to deadlocks in any application of real-world scale.
My question simply, is akka really a solution for a deadlock.
I know as long as akka actor...
1
votes
0
answer
68
Views
Calculate depart flights from sorted data using Spark
I've a dataset of flights in the form of
+----------------+----------+-------------+
|flightID |depart_ts |arrival_ts |
+----------------+----------+-------------+
|1 |1451603468| 1451603468|
|2 |1451603468| 1451603468|
|3 |1451603468| 1451603...
1
votes
0
answer
206
Views
Spark: join with a cached RDD takes too long
I have a key-value RDD and I need to join several keysets with this RDD.
Key-value RDD is big (100GB), keysets are relatively small (but not small enough to broadcast it)
I assign the same partitioner to all RDDs and call join.
Expected behavior:
After repartitioning all the data to be joined is col...
1
votes
0
answer
180
Views
Slick3.2 Error: No matching Shape found
I'm not sure what is wrong here.
The following code block is throwing error:
(for {
(e,r) r.col1 === e.col3)
} yield (e.id)
Error
No matching Shape found.
[error] Slick does not know how to map the given types.
[error] Possible causes: T in Table[T] does not match your * projection,
[error] you us...
1
votes
0
answer
295
Views
Spark (Scala) - how to optimize objective function parameters
I have function f going from R2 to R which takes 2 parameters (a and b) and returns a scalar.
I would like to use an optimizer to estimate the values of a and b for which the value returned by f is maximized (or minimized, I can work with -f).
I have looked into the LBFGS optimizer from mllib, see:...
1
votes
0
answer
743
Views
How to inject json coming from feeder inside body with gatling
I have a gatling scenario which
create a Http object
create a post request with a body
set up the scenario
I can inject a body from a json file containing this:
[{'id':'1'}]
At this point I have no problem. My question is on the second point. I want users to have theirs body feed with different id....
1
votes
1
answer
1.3k
Views
Did not find value which can be converted into java.lang.String
I have a JSON input which is to be written into Hbase
an example of this is
{
'contributors': null,
'truncated': false,
'text': '#Care #MichelleNunn #Stop #Pimping #Haiti #HurricaneMatthew #Give #Answers #Blacklivesmatter https:\/\/t.co\/mmlOawLdCy',
'is_quote_status': false,
'in_reply_to_status_id...
1
votes
1
answer
139
Views
Play json absent-sensitive Reads
I'd like to have a reads like Reads[Patch[T]] for a case class like this
sealed trait Patch[+T]
case class Update[+T](value: T) extends Patch[T]
case object Delete extends Patch[Nothing]
case object Ignore extends Patch[Nothing]
where a missing json value reads to Ignore, a null json value reads to...
1
votes
1
answer
431
Views
Get date months with iteration over Spark dataframe
I have a problem case to iterate for last 36 months based on an input date. Currently using Scala, through a DataFrame I am getting the max value of a timestamp field. For example:
val vGetDate = hc.read.format('filodb.spark').option('database','YYYYY').option('dataset','XXX').load().agg(max('inv_da...
1
votes
1
answer
206
Views
overloaded method value aggregate with alternatives
I have following function, that does not compile:
private def save(pea: KStream[String, String]): Unit = {
pea
.groupByKey()
.aggregate(() => '''{folder: ''}''',
(_: String, _: String, value: String) => value,
EventStoreTopology.Store)
}
the error message is:
[error] [VR](x$1: org.apache.kafka.str...
1
votes
1
answer
25
Views
not able to store result in hdfs when code runs for second iteration
Well I am new to spark and scala and have been trying to implement cleaning of data in spark. below code checks for the missing value for one column and stores it in outputrdd and runs loops for calculating missing value. code works well when there is only one missing value in file. Since hdfs does...
1
votes
0
answer
42
Views
Log file is not generated after running JAR
I have logback.xml file saved in src/main/resources of my Maven project. The path where the log file should be saved is defined as follows: file>/my/path/myLog.log.
I package the JAR file using IntelliJ IDEA. When I run this JAR file, myLog.log does not appear in /my/path/.
The problem is that every...
1
votes
2
answer
107
Views
Scala - composing generic types
I want to create a generic interface which hides the implementation of the hash function on the key of a product (which is used to lookup products and for various caching and calculations outside the scope of the product itself). I want to tie the concrete implementation of the hash function to the...
1
votes
0
answer
45
Views
How to calculate connections of the node in Spark 2
I have the following DataFrame df:
val df = Seq(
(1, 0, 1, 0, 0), (1, 4, 1, 0, 4), (2, 2, 1, 2, 2),
(4, 3, 1, 4, 4), (4, 5, 1, 4, 4)
).toDF('from', 'to', 'attr', 'type_from', 'type_to')
+-----+-----+----+---------------+---------------+
|from |to |attr|type_from |type_to |
+-----+-----...
1
votes
1
answer
1.3k
Views
How can I install older version (0.13.x) sbt on my MAC?
I need sbt version 0.13.x for a course in Scala and I am having difficulties installing it. I tried with the 'brew install [email protected]' command, but when I then check with 'sbt about' it gives a message 'No such file or directory'.
However, if I again try and do 'brew install [email protected]', it gives 'War...
1
votes
1
answer
83
Views
Unable to override java generic function with scala parameterized type
I am getting the error - Method read has incompatible type ; when I try to override a java generic function name with a Scala parameterized type.
This is the java abstract function I am trying to override.
public abstract class SRType{
//Other abstract functions
public abstract T read()throws IOEx...
1
votes
0
answer
215
Views
Using JBDC to read sql file in spark scala collecting Warehouse error
I am trying to read MySQL file using Spark Scala. Following is the code I tried
val dataframe_mysql = sqlContext.read.format('jdbc')
.option('url','jdbc:mysql://xx.xx.xx.xx:xx')
.option('driver', 'com.mysql.jdbc.Driver')
.option('dbtable', 'schema.xxxx')
.option('user', 'xxxx').option('password', '...
1
votes
2
answer
1.3k
Views
change a dataframe row value with dynamic number of columns spark scala
I have a dataframe (contains 10 columns) for which I want to change the value of a row (for the last column only). I have written following code for this:
val newDF = spark.sqlContext.createDataFrame(WRADF.rdd.map(r=> {
Row(r.get(0), r.get(1),
r.get(2), r.get(3),
r.get(4), r.get(5),
r.get(6), r.get(...
1
votes
0
answer
64
Views
How to handle error on batch tasks in flink
In my flink job there is a specific task 'GroupReduce' that involve external system (elasticsearch in this case) that can encounter an timeout exception.
A simple try and catch is not enought because i don't want to just skip the groupreduce task if an error occure.
There is a way to, once cought th...
1
votes
1
answer
345
Views
Using Spark CombineByKey with set of values
I have the following dataset:
val data = sc.parallelize(Array(
('group1','value1'),('group1','value2'),('group1', 'value1'),
('group2','value1'),('group1','value1'),('group3', 'value3')
))
I'm using Spark, and I would like to obtain the following result:
Array(('group1','value1',3),('group1', 'value...
1
votes
0
answer
139
Views
How do I inject ReactiveMongoApi trait in a Repository with macwire?
I'm new to Play framework and all the Scala stack. I'm trying to implement dependency injection using macwire and ReactiveMongo to handle my MongoDB database.
I have this classes to define repositories.
trait BaseRepository extends ReactiveMongoComponents {
/**
* collection to retrieve documents fro...
1
votes
0
answer
69
Views
Covariance annotation raises incorrect implicit ambiguity error
The code below raises an implicit not found error that the -Xlog-implicits sbt option explains as an ambiguity, which I pasted at the end:
import scala.language.higherKinds
trait JsonOFormat[T]
trait BsonOFormat[T]
trait Codec[CTXO[_]]
implicit def jsonCodec: Codec[JsonOFormat] = new Codec[JsonOForm...
1
votes
0
answer
66
Views
Scala Swing scrollbar for TextArea
I'm trying to add a scrollbar to a text area component. I could use FlowPanel, but I need to use GridBagPanel for my other components and when I tried to add a flowpanel inside the gridbagpanel it was empty and like 2x2 pixels.
Is there a way to add two panels to a MainFrame or add the scrollbar so...
1
votes
0
answer
286
Views
Convert an Array[org.apache.spark.sql.Row] to a DenseMatrix[Double] in scala
I have this solution which works but I wonder if there's a clever way to do it, avoiding the explicit for loop and the toString.toDouble conversion
import scala.collection.mutable.ArrayBuffer
val rows: Array[Row] = df2.collect()
//rows: Array[org.apache.spark.sql.Row] = Array([-1.35980713367,...
var...
1
votes
0
answer
145
Views
MongoController with Play
I am using Play! Framework with current version and i am trying to integrate with MongoDB via ReactiveMongo, but I am getting following error 'illegal inheritance; superclass Object is not a subclass of the superclass AbstractController of the mixin trait MongoController'
Here is my controller code;...
1
votes
0
answer
228
Views
Getting “null is disabled” warning for log messages with scala wart remover
Wart remover complains about log messages. Why does it complain? How to get rid of wart remover warnings for log messages?
[wartremover:Null] null is disabled
[warn] logger.error(e)
[warn] ^
1
votes
0
answer
325
Views
Spark CosmosDB Sink: org.apache.spark.sql.AnalysisException: 'write' can not be called on streaming Dataset/DataFrame
I am reading a data stream from Event Hub in Spark (using Databricks). My goal is to be able to write the streamed data to a CosmosDB. However I get the following error:
org.apache.spark.sql.AnalysisException: 'write' can not be called on streaming Dataset/DataFrame.
Is this scenario not supported?...
1
votes
0
answer
560
Views
Spark job dataframe write to Oracle using jdbc failing
When writing spark dataframe to Oracle database (Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit), the spark job is failing with the exception java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection. The scala code is
dataFrame.write.mode(...
1
votes
1
answer
285
Views
How to create an Akka flow with backpressure and Control
I need to create a function with the following Interface:
import akka.kafka.scaladsl.Consumer.Control
object ItemConversionFlow {
def build(config: StreamConfig): Flow[Item, OtherItem, Control] = {
// Implementation goes here
}
My problem is that I don't know how to define the flow in a way that it...
1
votes
0
answer
159
Views
How to perform Welch's Ttest in Spark 2.0.0 using StreamingTest
I want to try Welch's T-test in Spark 2.0.0
As I know I can use StreamingTest() from mllib
on this website.
[https://spark.apache.org/docs/2.0.0/api/scala/index.html#org.apache.spark.mllib.stat.test.StreamingTest]
this is my code in spark-shell
import org.apache.spark.mllib.stat.test.{BinarySample,...
1
votes
0
answer
73
Views
Setting 'testFrameworks' in build.sbt of a scalajs project causes java.lang.NullPointerException when running test
I am new to scala and scalajs. I am trying to add unit tests to my scalajs project using minitest. I followed instructions at https://github.com/monix/minitest#usage-in-sbt, so this is what my build.sbt looks like:
enablePlugins(ScalaJSPlugin, WorkbenchPlugin)
name := 'Test'
version := '0.1-SNAPSHOT...