Questions tagged [apache-nifi]

1

votes
1

answer
775

Views

Apache Nifi - ConvertJSONToSQL - JSON Does not have a value for the required column

I am trying to experiment with a tutorial I came across online, and here is its template: While the template ended with converting CSV to JSON, i want to go ahead and dump this into a MySQL Table. So i create a new processor 'ConvertJSONToSQL'. Here are its properties: And these are the controller s...
Shashank
1

votes
2

answer
138

Views

How to wait for GenerateTableFetch queries to finish

My use case is like this. I have some X tables to be pulled from MySQL. I am splitting them using SplitText to put each table in a individual flow file and pull using GenerateTableFetch and ExecuteSQL. And I want to be notified or put some other action when import is done for all the tables. At Spl...
pratpor
1

votes
1

answer
138

Views

Apache Nifi Expression Language - toDate formatting

I am trying to format a date string using the Apache Nifi expression language and the Replace Text processor(regex). Given a date string date_str : '2018-12-05T11:44:39.717+01:00', I wish to convert this to: correct_mod_date_str: '2018-12-05 10:44:39.717', (notice how the date is converted to UTC,...
irrelevantUser
1

votes
1

answer
31

Views

How to replace the content of a flow file that exists between [ and ] in Nifi?

I wanted to remove the entire content between the brackets of a flow file attribute. Attached is my sample Flow file and in which I wanted to remove the content between [ and ]. May I know the search and replacement value to be used from ReplaceText Processor ?Flow File content
SPK
1

votes
2

answer
43

Views

Access AWS services from Apache Nifi running on AWS

I have a Nifi instance running on an EC2 machine and I am trying to access a restricted s3 bucket. Because generating access keys manually is not recommended, I want to give the machine the proper IAM role for accessing the outside bucket. I gave the EC2 machine a role which seems to work for every...
Ethan McCue
1

votes
1

answer
43

Views

How to use Aws Temporary credentials in Nifi

I have to use aws temporary credentials AccessKey, SecretKey and Token within nifi process to access S3 objects. AccessKey, SecretKey and Token will be provided by an Api call. How to use these temperory credentials in nifi ListS3 Object etc? One of the options I found is using AWSCredentialsProvid...
Ani
0

votes
0

answer
3

Views

Jolt Transform Producing Arrays in NiFi

I've never used Jolt Transform before and I'm not sure how to fix what I've done, so I apologize if this is actually a really easy fix. I have two XML files (dummy versions below, actual contains PII) that I merged together using MergeRecord in NiFi. Because of the way that the output is (one flow f...
carousallie
1

votes
1

answer
214

Views

Is it possible to use Apache NiFi as dataflow engine for MS SQL Server in-database computations?

Is it possible to use Apache NiFi as dataflow engine for MS SQL Server in-database computations? In other words, I would like to use Apache NiFi as if I used Alteryx. For example, can I execute two or more MS SQL procedures sequentially from within Apache NiFi? Can you suggest alternative tools?
user2530062
1

votes
2

answer
156

Views

How to use invoke http to perform GET request in nifi?

I need to perform a get request from nifi to couchbase. The curl command is: curl http://HOST:PORT/query/service -d 'statement=select item.Date from bucket unnest bucket as item' -u USER:PASSWORD I tried using InvokeHttp and ExecuteStreamCommand but it keeps returning errors(status code 400). The fu...
Gal Itzhak
1

votes
1

answer
39

Views

Difference Between ExecuteSQL and ExecuteSQLRecord

Experts, i am Nifi beginner and trying to understand the difference between 2 processors. ExecuteSQL and ExecuteSQLRecord (V1.8 onwards). Is there something extra which later provides and ExecuteSQL doesn't? One thing i have already seen is that ExecuteSQL doesn't mantain 'state'. Wondering if Execu...
Sidd
1

votes
2

answer
58

Views

Cannot delete a flow in Nifi registry via http call when using git as storage

When using DELETE /buckets/{bucketId}/flows/{flowId}, it cannot delete a flow which has version 0. for example, something like the following flow cannot be deleted via the above http call. { 'bucketIdentifier': '6a0f624c-3163-496e-bb60-9ca4fe287654', 'bucketName': 'flow_bucket', 'createdTimestamp'...
Akira Sendoh
1

votes
2

answer
38

Views

I have two Json payload. I want to merge them in a single Json object

I have two payloads and want to merge them into single JSON object (streaming join). At few places people are suggesting to use AttributesToJSON, but as one of the JSON does not have fix set of attributes I guess that would not be possible. First payload is { 'title':'API-Actions Documentation', '...
Anurag Pathak
1

votes
0

answer
38

Views

Wait-Notify for parallel & sequential processing in Nifi

I have a requirement where I need to execute 4 jobs parallel and when same items job is done in all 4 processors parallely then trigger the next processor for this I have used wait-notify Flow is like 4 parallel jobs -> notify (release signal identifier = ${itemid}, signal counter name = ${processor...
happy
1

votes
1

answer
81

Views

How to remove negative index error in groovy code?

I have code logic inside nifi processor (executeScript processor) which will reduce log files(in this case in my log files i have same text so i want to remove duplicates and i try to choose them by name and file size),but i sometimes ( not always) got negative index...
titan titan
1

votes
0

answer
107

Views

NiFi DecryptContent - the cipher stream is too small to contain the salt

I'm testing with NiFi and I've run into an error that I can't find explained anywhere. I am using the EncryptContent processor with MD5_AES128, NiFi Legacy KDF, and a password. I'm encrypting something about 500KB, and it encrypts fine. If I immediately pipe the output of that operation into a seco...
Dan
1

votes
0

answer
478

Views

Unable to connect to remote sql server using nifi

I have NiFi running on HDFnode-2 and sl running on HDPnode-1. Using ExecuteSQl processor I am trying to fetch some data but getting below error: ExecuteSQL[id=e50be381-0160-1000-0000-00006ee4f593] Unable to execute SQL select query select * from READ_ACCESS_LOG LIMIT 10; due to org.apache.nifi.proce...
Saahil Gupta
1

votes
0

answer
198

Views

“Value” causing issues in schema generation

I have an object like this: 'Meta': { 'Type': 10, 'Key': 'Meta', 'Value': {
Matt
1

votes
0

answer
293

Views

Validate XML with Nifi

What is the proper way to validate an XML file with Nifi processors? Is there an example of a schema for the validteXML processor? I am new to the Nifi world.
CMPE
1

votes
0

answer
52

Views

Is it possible to schedule and monitor MR and spark jobs using Apache NIFI?

I would like to use Apache NiFi for the complete data pipeline scheduling and monitoring for both batch(Map reduce, Hive) and streaming jobs(Spark) Is it possible to schedule and monitor MR and spark jobs using Apache NIFI? If it so what are the implementation steps ?
prady
1

votes
1

answer
587

Views

json formation in nifi using jolt transformjson

I am new to nifi. I need help to make new json template using jolttransform processor. could anyone please help to make joltspec for this requirement. From convertavrotojson processor flow file am getting the following attributes name, address, id,status And from its content file, am getting mobile...
user4540719
1

votes
1

answer
68

Views

Sync database extraction with Hadoop

Lets say you have periodic task that extract data from a database and loads that data into Hadoop. How does Apache Sqoop/Nifi mantain database sync between the source database (SQL or NoSQL) with destination storage(Hadoop HDFS or HBASE, even S3)? For example, lets say that at time A the database h...
Diego Gallegos
1

votes
1

answer
501

Views

Read Avro file with python to create a SQL table

I'm trying to create an SQL table from AVRO file which contains the structure of my table : { 'type' : 'record', 'name' : 'warranty', 'doc' : 'Schema generated by Kite', 'fields' : [ { 'name' : 'id', 'type' : 'long', 'doc' : 'Type inferred from '1'' }, { 'name' : 'train_id', 'type' : 'long', 'doc' :...
amira khalifa
1

votes
1

answer
755

Views

Apache Nifi: PutHiveStreaming is not connecting

I have a simple process flow that follows the example: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html The flow looks like this: The flow files go through the entire process but then fails to write to the Hive DB in the processor 'Stream CSV to Hi...
lordoku
1

votes
0

answer
145

Views

Using Selenium on Apache NiFi

Has anyone automated Apache NiFi using Selenium? I can't seem to find helpful guides that, its seems like a challenges because I believe its built using AngularJS and waiting for the elements to load is challenging when the GUI is not consistent. My main objective is to take screenshots on every pro...
Zahidul Islam
1

votes
0

answer
133

Views

Apache Nifi HBASE lookup

I am new on Apache Nifi We create Nifi flow, which is consuming json data from kafka and the results are being sent to another kafka topic after enrichment. However HBase lookup does not return value of key. Instead it returns key, value pair like MapRecord[{SERIAL_NUM=123456789}]. But I require on...
erkan.oktay
1

votes
1

answer
315

Views

Secure communication between Ingress Controller (Traefik) and backend service on Kubernetes

I'm trying to secure Nifi in a Kubernetes cluster, behind a Traefik proxy. Both are running as services in K8S. Traefik is secured with a public certificate. I want it to redirect calls to nifi, while securing the communication between Traefik (as an Ingress Controller) and the backend pods : Nifi....
Olivier
1

votes
0

answer
154

Views

NiFi avro schema using regex to validate a string

I have an avro schema in NiFi which validates the columns of a CSV file, all is working well, however I'd like to ideally have an extra level of validation on certain string column to test that they adhere to specific patterns. For example ABC1234-X, or whatever. Here's the wrinkle though, the avro...
Mark Balmer
1

votes
1

answer
358

Views

Nifi ExecuteSQL failed to process session due to createSchema: Unknown SQL type 1111

I'm working with Nifi and I would like to transfer data from a postgresql's table to Json files. I have a table in postgresql database with schema: CREATE TABLE 'public'.'users' ( 'id' int4 NOT NULL DEFAULT nextval('users_id_seq'::regclass), 'email' varchar(255) COLLATE 'pg_catalog'.'default' NOT NU...
Wilson Ho
1

votes
1

answer
81

Views

Synchronize NiFi process groups or flows that don't/can't connect?

Like the question states, is there some way to synchronize NiFi process groups or pipelines that don't/can't connect in the UI? Eg. I have a process where I want to getFTP->putHDFS->moveHDFS (which ends up actually being getFTP->putHDFS->listHDFS->moveHDFS, see https://stackoverflow.com/a/50166151/8...
1

votes
1

answer
312

Views

NiFi ListHDFS cannot find directory, FileNotFoundException

Have pipeline in NiFi of the form listHDFS->moveHDFS, attempting to run the pipeline we see the error log 13:29:21 HSTDEBUG01631000-d439-1c41-9715-e0601d3b971c ListHDFS[id=01631000-d439-1c41-9715-e0601d3b971c] Returning CLUSTER State: StandardStateMap[version=43, values={emitted.timestamp=1525468790...
1

votes
2

answer
855

Views

Do messages get deleted from the queue after a read operation in IBM MQ?

I am using Nifi to get data from IBM MQ. It is working fine. My question is once the message is read from an MQ queue, does it get deleted from the queue? How to just read messages from the queue without deleting them from the queue?
trougc
1

votes
0

answer
37

Views

Schema validation of multi reference chainied schema

I want to do three things Validate JSON against a JSON-Schema Create JSON-Schema to AVRO Schema converter Create JSON-Schema to Hive Table converter The problem I'm facing is the Schema has a referencing chain. I'm trying to use this JSON Schema Validator which resolves reference and validates but...
Sam
1

votes
1

answer
97

Views

Nifi GetTwitter Processors with Proxy

I am using NiFi at work. I'm behind the proxy. I can't get data from twitter because of error cannot establish connection properly and connect time out Did someone what I have to do to fix this error? Or maybe someone know the configuration for GetTwitter Processor with Proxy? Thanks
tkris
1

votes
0

answer
561

Views

NiFi merge CSV files using MergeRecord

i have a stream of JSON records that i convert it into CSV record successfully with this instruction. but now i want to merge this CSV records into one CSV file. below is that flow: at step 5 i face with around 9K csv record, how do i merge it into one csv file using MergeRecord processor? my csv he...
meh
1

votes
0

answer
36

Views

Warning when deleting processors using Nifi 1.6

I recently upgraded the test environment to Nifi 1.6 on Centos 7. I am running a three node cluster using the Nifi internal zookeeper version. If I drag in a new processor and then delete it I get the following stack trace in the logs. WARN StandardStateManagerProvider Component with ID {} was...
dmarrow4
1

votes
0

answer
180

Views

Securing Nifi registry with nginx Error

I followed this tutorial to set up a secure version of Nifi registry: https://community.hortonworks.com/content/kbentry/170966/setting-up-a-secure-apache-nifi-registry.html I am working on an ubuntu server. I do not have the possibility to generate the keychain and to access the graphical interface...
amira khalifa
1

votes
0

answer
61

Views

Why are there duplicates generated by Apache Nifi CaptureChangeMySQL?

Bin log format is ROW. Set concurrent tasks to 1. Set Execution on to 'Primary Node'. Run schedule - 200 millis Retrieve all records : false Include ddl events : false Include Begin/Commit events : false Configured CDCMapCache on localhost. Started the processor and updated a couple of records in...
Sachin Sharma
1

votes
0

answer
281

Views

Process nifi flow files present in incoming queue, all together

Need help to process multiple nifi flow files together. I am using Wait/Notify -> ExecuteScript in my flow. Wait/Notify will wait for n number of flow files to arrive and release n files together then ExecuteScript processor will check if a certain attribute has the same value in all these flow file...
Abhishek Tiwari
1

votes
1

answer
48

Views

I need to get latest data in data ingest template nifi

Hi sir, In data ingest template i need to get this property for ex i have data with date field date data 12-07-2018 a 13-07-2018 b 14-07-2018 c 15-07-2018 d In that , i would like to take latest one i.e, 15-07-2018 if date field got new data 16-07-2018 e then i have...
IMRAN S K
1

votes
1

answer
159

Views

Nifi putHiveStreaming Failed to connect to metastore uri

I'm facing issues with putHiveStreaming Processor as it is not connecting to hive metastore. I am using kylo-cloudera-sandbox-0.9.1, please help me on this as I'm not able to figure out the issue.
Karthik Mannava

View additional questions