Questions tagged [apache]

72206 questions
1

votes
1

answer
92

Views

Java Spark: com.mongodb.spark.config.writeconfig issue

I am trying to connect with MongoDB via java spark connector and I am getting an error 'com.mongodb.spark.config.writeconfig', when I submit the jar and run the jar in spark shell. Here the error screenshot: Could you please help me to resolve this issue. I have tried this as well, but no success. $...
Tom Swayer
1

votes
1

answer
724

Views

Hierarchical data manipulation in Apache Spark

I am having a Dataset in Spark (v2.1.1) with 3 columns (as shown below) containing hierarchical data. My target objective is to assign incremental numbering to each row based on the parent-child hierarchy. Graphically it can be said that the hierarchical data is a collection of trees. As per be...
Sridher
1

votes
0

answer
204

Views

Scala reflection error while registering a Spark UDF

I'm using Spark UDFs all over my code, but there is one registration the fails intermittently with the following error: scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving package at scala.reflect.internal.Symbols$TypeSymbol.tpe(Symbols.scala:2768) at scala.reflect.in...
Hagai
1

votes
2

answer
700

Views

Apache Camel redelivery: how to use the attempt number

I have a Camel Route with the onException clause: at each redelivery I want to increase the redeliveryDelay. How do I get the attemptNumber? The DefaultErrorHandler clearly stores it somewhere because it prints it in the log e.g. 'On delivery attempt: 1 caught' onException(MyException.class) .handle...
Gep
1

votes
0

answer
41

Views

How to make Spark Worker read data from local mongodb with mongodb-spark-connector?

I have got two 'mongodb' on two computers. And there is also a 'Spark Worker' on each computer. But when I run 'spark', it doesn't read data from its local 'mongodb'. Instead, it reads from one of them. Therefore, only got partial data. There is a page. https://docs.mongodb.com/spark-connector/maste...
BobXWu
1

votes
0

answer
280

Views

apache httpd LogFormat not honoring strftime format

i need a 8601 timestamp in my httpd logs, but httpd appears to not be honoring its time formatting contract. i'm using apache httpd 2.4.6. i have a logging conf as follows: ErrorLogFormat '{\ \'level\': \'%l\',\ ... \'timestamp\':\'%{%Y-%m-%dT%H:%M:%S%z}t\'\ }' ErrorLog /dev/stderr LogLevel info Lo...
cdaringe
1

votes
3

answer
57

Views

Is there a way to rewrite my urls without using mod_rewrite and .htaccess?

Is there a way to rewrite my urls from: http://www.website.com/notification.php to http://www.website.com/notification I dont have the permission to turn on mod_rewrite and our organisation doesnt want it either. Is there a way to realise this with php only? Thanks in advance
Khaled Tlili
1

votes
0

answer
484

Views

Request address for sparkDriver failing

while running pyspark on my terminal There is a new issue appeared in pyspark initiation: 17/12/28 10:31:59 ERROR SparkContext: Error initializing SparkContext. java.net.BindException: Can't assign requested address: Service 'sparkDriver' failed after 16 retries (starting from 0)! Consider explicit...
mermi
1

votes
0

answer
1.2k

Views

X-Frame-Options Header Not Set in Apache Tomcat 8.5.9

I am using Apache Tomcat 8.5.9 server for Java Web application with struts2, spring and spring-security. While doing security testing using 'Zap 2.7.0 security scanning Tool' I got following errors in a scanning report of my web application. X-Frame-Options Header Not Set Web Browser XSS Protection...
Prakash Krishnakumar
1

votes
0

answer
224

Views

Spark UI storage tab shows more RDDs

I am running a spark streaming application. The app uses MapWithStateRDD to manage state across batches. App and setup details: Nodes: 2 Memory per executor: 3 GB Num of partitions for MapWithStateRDD: 200 Standalone mode Batch size: 20 sec Timeout duration: 1 minute Checkpointing enabled Checkpoint...
scorpio
1

votes
1

answer
50

Views

apache IndexOptions not working

Can anyone tell me what I've missed? I can't get the IndexOptions directive to work. I have included my virtual host *.conf file. I've done this before, with an almost identical virtual host settings file and it worked fine. However, in the past I have used ubuntu server and this time I am using...
brendon1981
1

votes
0

answer
41

Views

Why does a Capitalization error work locally…?

I've encountered this situation a couple of times now where everything is working fine locally, then when I push to remote server, I get errors due to simple Capitalization error. 'App\Useraction' vs 'App\UserAction The same error exists locally, so why does everything still work? Is there an Apac...
BizzyBob
1

votes
0

answer
41

Views

Is there intermediate computation optimization when using functions.window [Spark]

I am using functions.window to create sliding window computation using Spark and Java. Example code: Column slidingWindow = functions.window(singleIPPerRow.col('timestamp'), '3 hours', '1 seconds'); Dataset aggregatedResultsForWindow = singleIPPerRow.groupBy(slidingWindow, singleIPPerRow.col('area')...
Anton.P
1

votes
1

answer
253

Views

Run Elastic Search on pdf and ppts

I am new to elastic search. I have read its tutorials. But need guidance on my problem: I have a collection of pdf documents and power point files on my system. I need to build a system using elastic search where I can retrieve these files on the basis of keywords present in this file. Can someone p...
Astha Sachdev
1

votes
1

answer
667

Views

Setting the font in XSL-FO with fop

I have in the .fonts/Dinarra/ folder the files Dinarra LT Std-Roman.otf and Dinarra LT Std-Italic.otf. The font configuration in fop.xconf is as follow: ... ... I have registered the fonts in fop with fop -c ~/.fop/fop.xconf. Now I have the following XSL-FO code in the file test.fob: S.E. Reverendo...
Reverendo Asperso
1

votes
0

answer
273

Views

Transaction management using SQLAlchemy Core and Flask Python fails with Apache

Have a Flask web application with API calls and transaction management using SQLAlchemy Core. While inserting/updating into multiple tables simultaneously, the code excerpt that takes care of it: import os from sqlalchemy import create_engine DB_URL='postgresql://tasksys:[email protected]/mydb...
user956424
1

votes
1

answer
70

Views

iDempiere multiple launching while updating any of OSGI plugins by Felix console

I hope you can help me to solve the problem. I’ve got a Linux server with iDempiere 3.1 and Java 1.8. Only iDempiere schedulers are launched on this server. Each scheduler has its own launch frequency (start time). I noticed that while updating any of OSGI plugins by Felix console, time of the nex...
Max Gabderakhmanov
1

votes
1

answer
1.2k

Views

How to merge small files in spark while writing into hive orc table

I am reading csv files from s3 and writing into a hive table as orc. While writing, it is writing lot of small files. I need to merge all these files. I have following properties set: spark.sql('SET hive.merge.sparkfiles = true') spark.sql('SET hive.merge.mapredfiles = true') spark.sql('SET hive.mer...
doitright
1

votes
1

answer
265

Views

How is spark.streaming.blockInterval related to RDD partitions?

What is the difference between blocks in spark.streaming.blockInterval and RDD partitions in Spark Streaming? Quoting Spark Streaming 2.2.0 documentation: For most receivers, the received data is coalesced together into blocks of data before storing inside Spark’s memory. The number of blocks in...
Rasika Gayani
1

votes
0

answer
32

Views

how do I remove http to https redirects cent OS wordpress LAMP stack

I have a website that has redirects when I moved to https. I have since removed some of those pages but the redirects are still there an giving me 404 error. I have checked the .htaccess file and the redirects are not there. Where are these redirects kept? the answered provided in the post is on how...
Michael
1

votes
0

answer
320

Views

Flink 1.4.0 ClassDefNotFoundError … S3ErrorResponseHandler

Working on setting up a local test of Flink 1.4.0 that writes to s3 and I'm getting the following error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.internal.S3ErrorResponseHandler at org.apache.flink.fs.s3presto.shaded.com...
moku
1

votes
1

answer
360

Views

java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ClassTag$$anon$1

I am using Spark, GraphX 2.0.2 and IntelliJ. I got the error: Class is not registered: org.apache.spark.graphx.impl.GraphImpl So I added: kryo.register(classOf[GraphImpl[Object,Object]]) but I got this error: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ClassTag$$anon$1...
DaliMidou
1

votes
1

answer
151

Views

camel rest servlet 404 instead of 405

Using the sample code from Camel in github in 1 I am getting 404 instead of 405. This is a summary of the code in 1 rest('/provider').description('Provider rest service').consumes('application/json').produces('application/json').get('/{id}').description('Find provider by id').outType(Provider.class)...
Bill
1

votes
0

answer
378

Views

MacOS Dnsmasq High Sierra 403 Forbidden error

I tried to configure dnsmasq with a fresh clean macOS High Sierra 10.13.2 Apache Version Server version: Apache/2.4.28 (Unix) Server built: Oct 9 2017 19:54:20 apachectl configtest Syntax OK I used to have it and worked fine. But I think I'm missing something, because I'm getting 403 forbidden er...
Absolutkarlos
1

votes
1

answer
101

Views

How mod_jk handling node failure

We have configured mod_jk with two tomcat servers with 2 apache web servers. We wanted to know how mod_jk handling node failure or how it will do a health check.?
Pramod Gouda
1

votes
0

answer
293

Views

The requested URL … was not found on this server

The home page of my wordpress website seems to be wroking correctly but if you click through to any of the other pages I get the following error message: The requested URL /about was not found on this server. There is following htaccess code in the permalink setting: RewriteEngine On RewriteBase / R...
Sanjeev Thakur
1

votes
1

answer
81

Views

How to remove negative index error in groovy code?

I have code logic inside nifi processor (executeScript processor) which will reduce log files(in this case in my log files i have same text so i want to remove duplicates and i try to choose them by name and file size),but i sometimes ( not always) got negative index...
titan titan
1

votes
0

answer
280

Views

Apache storm: tick tuple not working

In my Storm based application I need to query oracle table periodically So I thought to use Tick tuple of storm. But it's not giving correct result and tick tuple is not producing. My storm version is 1.0.1.2.5.3.0-37 I tried as below, Added getComponentConfiguration method in bolt as http://www.mic...
parag dharmadhikari
1

votes
0

answer
1.3k

Views

Convert DOC [HWPFDocument] to pdf [with font and images] using java

converting doc file to pdf I am using the following code : POIFSFileSystem fs = null; Document Pdfdocument = new Document(); fs = new POIFSFileSystem(new FileInputStream(srcFile)); HWPFDocument doc = new HWPFDocument(fs); WordExtractor we = new WordExtractor(doc); PdfWriter writer = PdfWriter.getIns...
Kishan C S
1

votes
1

answer
140

Views

Use of countByKeyApprox() for Partial manual broadcast hash join

I read about Partial manual broadcast hash join which can be used while joining Pair RDD in Spark. This is suggested to be useful if one key is so large that it can’t fit on a single partition. In this case you can use countByKeyApprox on the large RDD to get an approximate idea of which keys woul...
Amol T K
1

votes
2

answer
410

Views

Apache 2.2, Django, use Python 3.5

I want to run django with apache2.2 and python 3.6, after making changes in wsgy.py and virtuahost still running python 2.6 Apache/2.2.34 (Unix) DAV/2 mod_wsgi/3.2 Python/2.6.9 configured -- resuming normal operations Here wsgi.py import os, sys sys.path.append('/home/app/myapp/sivale') sys.path.app...
Arturo Alm
1

votes
1

answer
14

Views

Use regex_extract to retrieve the score number in a string text column

I need to extract the float number after score. {'reason_desc': { 'score':'0.1', 'numOfIndicatrix':'0', 'indicatrix':[]}, 'success':true, 'id':'1555039965661065S427A2DCF5787920' } I expect the output of 0.1 or any number enclosed by ''.
JYWQ
0

votes
0

answer
2

Views

Scala Spark Dataset Error on Nested Object

I am trying to test dataframe(dataset) code with strongly typed nested case classes into dataframe to then pass over my functions. The serialize/creation of the dataframe keeps failing and I do not have enough experience to know what is going on in scala or spark. I think that I am trying to determi...
vfrank66
1

votes
0

answer
8

Views

CreateDirectStream with messages avro

In a first moment, I had to process the information from a text file: C1_4,C2_4,C1______10,01/12/2015,30/12/2015,123456789,S,12345 Now, I need to process the same information but in format avro. How can I do it ? Before I used this code: createDirectStream[String, String, StringDecoder, StringDecode...
user2140391
1

votes
1

answer
1.5k

Views

How to write parquet files from streaming query?

I'm reading from a CSV file using Spark 2.2 structured streaming. My query for writing the result to the console is this: val consoleQuery = exceptions .withWatermark('time', '5 years') .groupBy(window($'time', '1 hour'), $'id') .count() .writeStream .format('console') .option('truncate', value = f...
Matthias Mueller
1

votes
1

answer
183

Views

XGBoost does not use enough all resources while running Spark in AWS EMR

I'm trying to make a binary classification on a big dataset (5million rows x 450 features) using XGBoost Spark lib in AWS EMR. I've attempted setting many different configurations like: Number of XGboost workers, nthreads, spark.task.cpus, spark.executor.instances, spark.executor.cores. Even though...
Bruno Brito
1

votes
1

answer
22

Views

Merging records by 1+ common elements

I have a hive table with the following schema: int key1 # this is unique array key2_list Now I want to merge records here if their key2_lists have any common element(s). For example, if record A has (10, [1,3,5]) and B has (12, [1,2,4]), I want to merge it as ([10,12], [1,2,3,4,5]) or ([1,2,3,4,5])....
kee
1

votes
1

answer
283

Views

Spark UDF written in Java Lambda raises ClassCastException

Here's the exception: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to ... of type org.apache.spark.sql.api.java.UDF2 in instance of ... If I don't implement the UDF by Lambda expression, it's ok. Like: private UDF2 funUdf = new UDF2() { @Override public S...
secfree
1

votes
0

answer
68

Views

Calculate depart flights from sorted data using Spark

I've a dataset of flights in the form of +----------------+----------+-------------+ |flightID |depart_ts |arrival_ts | +----------------+----------+-------------+ |1 |1451603468| 1451603468| |2 |1451603468| 1451603468| |3 |1451603468| 1451603...
Assem
1

votes
1

answer
20

Views

Redirecting and changing URI using .htaccess

I have requests like https://example.net/files/public/file.html which I would like to redirect to https://example.com/domain/public/file.html via htaccess. In theory I would have to write an if condition and then remove the files part from the URI and then redirect to the new domain. But in practice...
Johannes

View additional questions