Questions tagged [spring-cloud-dataflow]

1

votes
0

answer
78

Views

Can Spring Cloud Task application can run on embedded tomcat

I have a simple webapp with @EnableTask annotation at main class. My Application is running on embedded tomcat. Is it is possible to run SpringBoot WebApp wrapped with @EnableTask annotation in embedded tomcat. Is yes can i have a simple example for that? Thanks
Jagadheeswaran Mohan
1

votes
0

answer
337

Views

How to create complex POJO from multiple lines with Spring Batch

I'm new to Spring Batch and I'm kinda lost with a need I have to fulfill with this batch job and I was wondering if any of you could enlighten me about how can I properly do things right in this case. My need here is basically receive a text file containing multiple lines with fixed-length delimited...
Enrico Bergamo
1

votes
0

answer
38

Views

Spring cloud task “Deadlock found when trying to get lock”

I am facing problems while setting up spring cloud task: It is throwing 'Deadlock found when trying to get lock' exception. Don't know whats wrong in code? Limit number of concurrent task per processor.
Bhuvnesh
1

votes
1

answer
230

Views

Spring Batch/Task Repository Data Mismatch when introducing use of Tasks after historic Spring Batch data already exists

My project has 20+ batch jobs that are built with Spring Batch and have been in Production for a couple of years. We are currently in the process of migrating them to individual Spring Boot applications that are built with Spring Batch and Spring Cloud Task. These will then be deployed as Tasks in S...
lframirez89
1

votes
2

answer
307

Views

Spring cloud data flow shell : Stuck on “The stream is being deployed”

I successfully registered three apps named appSink, appSource and appProcessor as follows dataflow:>app register --name appSource --type source --uri maven://com.example:source:jar:0.0.1-SNAPSHOT --force Successfully registered application 'source:appSource' dataflow:>app register --name appProcesso...
frigocat
1

votes
1

answer
122

Views

Error Deploying Scriptable-Transform in Spring Cloud Data Flow : ParserException

I'm trying to construct a stream in spring cloud data flow with kafka-based scriptable transform. I followed the instruction here: https://docs.spring.io/spring-cloud-stream-app-starters/docs/Celsius.SR2/reference/htmlsingle/#spring-cloud-stream-modules-scriptable-transform and my stream definition...
Kevin Yang
1

votes
1

answer
150

Views

Writing data on a Kafka compacted topic - how to set the record key

Using spring cloud dataflow, I'm trying to replicate data from a database using a Kafka compacted topic. For this reason I need to set the key of the record while producing records into the topic (called foo). kafka-topics --zookeeper localhost:2181 --create --topic foo --replication-factor=1 --part...
user3753094
1

votes
0

answer
51

Views

Spring Cloud Data Flow with S3 as artifact repo

Is it possible to configure spring cloud data flow to download the artifacts from an S3 storage bucket like ECSS3 as against maven repository ? Due to corporate security reasons we can't download artifacts dynamically from internal maven repository in production. We are looking into an approach wher...
Nitty
1

votes
1

answer
108

Views

When deploying a stream with MongoDB Sink I got a MappingException “Couldn't find PersistentEntity for type class [B!”

I'm facing this problem when I'm trying to use the MongoDB Sink app starter with an information read by a JDBC source: MappingException: Couldn't find PersistentEntity for type class [B! Class [B! means that is a byte[]. I printed in a log and this is the JSON of the object. Is this a bug? The other...
Norberto Ritzmann
1

votes
0

answer
247

Views

Spring Batch with Spring cloud data flow

Getting the below error after adding @EnableTask in Spring Batch application The bean 'transactionManager', defined in class path resource [org/springframework/batch/core/configuration/annotation/SimpleBatchConfiguration.class], could not be registered. A bean with that name has already been defined...
Deepak
1

votes
1

answer
44

Views

Spring cloud data flow server cloud foundry config server

I deployed spring cloud data flow server cloud foundry to PCF (with cf push SPRING_CLOUD_DATA_FLOW_SERVER_CLOUD_FOUNDRY JAR).When i run it it try to connect to http://localhost:8888 as a config server. I'm using my own config sever and in manifest file i'am using services: config-service (as a user...
user3548905
1

votes
0

answer
87

Views

Spring Cloud Stream App VS Spring Cloud Task App

I'm exploring the the SCDF for orchestrating the existing batch task for my system. I noticed that the SC Stream App Starters already provides some OOTB ready to use streaming applications, which suits my use case of: Extracting xml file from FTP -> the ftp source Customize transformation Upload t...
mengjiann
1

votes
1

answer
65

Views

How can Spring Cloud Dataflow Server use new tables( with custom prefix ) created for Spring batch and Spring cloud task?

I have created spring cloud task tables i.e. TASK_EXECUTION, TASK_TASK_BATCH with prefix as MYTASK_ and spring batch Tables with prefix MYBATCH_ in oracle database. There are default tables also there in the same schema which got created automatically or by other team mate. I have bound my Oracle d...
DipikaV
1

votes
0

answer
25

Views

PCF reports application crash each 5 minutes

We have spring-boot batch task application that is hosted on pcf and deployed via spring-cloud-dataflow(scdf). After we have connected it to app-dynamics agent it start to report crash every 5 minutes with the following resolution: But there is no any exceprion within application and there is no c...
b1gs
1

votes
2

answer
38

Views

Deployment of sample task fails in PCF

spring-cloud-dataflow-server-2.0.1.RELEASE.jar I am trying to deploy the sample task app on [email protected] Deployment fails with the following Exception : Shell side : No Launcher found for the platform named 'default'. Available platform names are [] org.springframework.cloud.dataflow.rest.client.DataF...
Tajdar Siddiqui
1

votes
1

answer
29

Views

Spring Cloud Data Flow + Kubernetes, asking for the task pod to be deployed on non-default namespaces

I have a setup with scdf-server on kubernetes working fine, it deploys each task in an on-demand pod on the very same default namespace, the one that hosts the scdf-server pod. Now, I need to deploy a pod in another namespace and I can't find the argument/property to use in the scdf server dashboar...
Whimusical
1

votes
0

answer
18

Views

Spring Cloud Data Flow - declare dependency on another job

I want to create job (a composed task, specifically), that declares a dependency on starting based on whether a previous job completed. I've not seen how one could do this with SCDF. Is this possible?
sethcall
1

votes
1

answer
515

Views

Spring Cloud Dataflow ticktock example fails in CloudFoundry

I'm trying to get most basic example for Spring Cloud Dataflow running on CloudFoundry. I've followed the steps here: http://docs.spring.io/spring-cloud-dataflow-admin-cloudfoundry/docs/current-SNAPSHOT/reference/htmlsingle/#getting-started to make the admin app available in my org/space. Then I tri...
Alexander
1

votes
1

answer
233

Views

spring XD - http-client add application/json header

I've installed Spring XD in my PaaS environment and for legacy issue I'm stuck with version 1.0.0.M1. ( see reference doc). My goal is to call a http rest API using the http-client module. My stream definition is: http | httpclient --url='''''' --http-method=POST --mappedRequestHeaders=HTTP_REQUEST_...
Luca
1

votes
1

answer
109

Views

spring cloud data flow component collisions

I just started to use the spring cloud dataflow project. I tried creating some custom components using the spring initialr that was recommended at http://start-scs.cfapps.io I am finding that quick a few of the components selected are colliding For example, I generated one that had the following t...
EvilJinious1
1

votes
1

answer
564

Views

Spring Cloud DataFlow local Server not importing the new custom source for Tailing File

I am trying to write a custom source for 'Tailing' and I used the snippets from spring samples to write that code. Here is the actual code, @ComponentScan @EnableBinding(Source.class) @EnableConfigurationProperties(TailSourceProperties.class) public class TailSourceConfiguration { @Autowired private...
hercules
1

votes
1

answer
1.4k

Views

How to execute spring cloud task using rest-api

I know a cloud task can be scheduled and can be configured using stream also to be executed. As a developer I want to execute my spring cloud task using rest-api so that I can execute the task on demand. Basically i have a work flow management system and we are using control-m agent. So now some of...
Sunil Chauraha
1

votes
2

answer
865

Views

Executing spring cloud task based on event into messaging source (i.e. RabbitMQ, Kafka)

I am new in Learning of Spring cloud Task and SCDF so asking this. I wand to execute my SCT based on an event (say a message is posted into Rabbit MQ), so I am think it can be done in two ways: Create a source which polls message from RabbitMQ and sends the data to stream, now create a sink which re...
Sunil Chauraha
1

votes
2

answer
841

Views

Is there a way to configure the Kafka client timeout settings in Spring Cloud Data Flow?

When starting a Spring Cloud Data Flow stream, quite often the apps fail to deploy on my machine because of various Kafka related errors. For instance: org.springframework.cloud.stream.binder.BinderException: Cannot initialize binder [...] Caused by: kafka.common.KafkaException: fetching topic metad...
Remon Sinnema
1

votes
1

answer
523

Views

Stream CSV file using Spring Cloud Dataflow with Kafka

I am trying to stream csv file with Spring Cloud dataflow and Kafka. The idea is to stream the file line by line with header included for each line. For example, if the content of my csv file is street,city,zip,state 3526 HIGH ST,SACRAMENTO,95838,CA 51 OMAHA CT,SACRAMENTO,95823,CA 2796 BRANCH ST,SA...
krajwade
1

votes
1

answer
2.2k

Views

Advantage of using Spring cloud data flow instead of spring batch

We need to write an application to read a flat file every day and write into a database table. We are planning to use Spring Batch to do this job. The limitation and addition we are looking for are 1.The application itself would run in a single VM. It would never be deployed in more than one VM at...
user3720735
1

votes
1

answer
184

Views

Can slave process be dynamically provisioned based on load using Spring Cloud data flow?

We are currently using Spring batch - remote chunking for scaling batch process . Thinking of using Cloud data flow but would like to know if based on load Slaves can be dynamically provisioned? we are deployed in Google Cloud and hence want to think of using Spring Cloud data flow support for kuber...
Raghavan Narasimhan
1

votes
1

answer
219

Views

Spring Stream router-sink won't start on CDF

I tried to deploy the following stream: http --port=1234 | router --script=D:\my-router.groovy on local-cdf-server but router application won't start: *************************** APPLICATION FAILED TO START *************************** Description: Parameter 1 of method router in org.springframework....
alturkovic
1

votes
1

answer
621

Views

Spring Cloud DataFlow for HTTP request/response exchange

I would like to use streams to handle an HTTP request/response exchange. I didn't see any Spring Cloud Stream App Starters with HTTP sink functionality. Will I need to build a custom sink to handle the response? If so, do I pass the request through my processing pipeline, then use the request in my...
James Hines
1

votes
1

answer
282

Views

Spring-Cloud-flow orchestration for realtime data processing

I need to build a orchestration layer for couple of Rest APIs (microservices), message based services and post the result to a messaging layer (RabbitMQ/Kafka). Our application is realtime similar to loan processing. Customer provide some details via UI and wait for the response. I am looking at the...
Chandru
1

votes
1

answer
177

Views

How dose Kafka offset change when redeploying SCDF stream?

I use SCDF on YARN 1.0.2.RELEASE with Kafka 0.10.0 for stream applications. When I undeploy a stream and then deploy it again, it seems sometimes it continues at the current Kafka topic offset, while sometimes it starts from the oldest offset. Where can I configure to get an predictable behavior(e.g...
JavyZheng
1

votes
1

answer
79

Views

Spring Cloud DataFlow RabbitMQ min-start-interval

I want to start 50 Rabbit consumers as soon as I start the app and start more consumers each second if needed up to 500 consumers. How can I set the startConsumerMinInterval for RabbitMQ in the application.yaml of the deployer? I found startConsumerMinInterval property in the Spring AMQP documentati...
alturkovic
1

votes
1

answer
80

Views

spring-cloud-starter-stream-sink-sftp Sftp Sink , file-expression is not workin

I am using stfp sink and trying to name the remote file based on header. I am providing following property to name the file on remote sftp.file-expression=payload.headers['id'] + payload.headers['file_name'] I also tried sftp.file-expression=headers['id'] + headers['file_name'] but these is not work...
gamepop
1

votes
1

answer
600

Views

How to use a SPEL expression that is applied on a POJO for the routingKey?

My Source APP: @InboundChannelAdapter(value = RequestSource.CHANNEL_NAME) public Event eventMessageSource() throws Exception { final Random r = new Random(); Event event = generateSampleEvent(EventType.values()[r.nextInt(EventType.values().length - 1)]); System.out.println('---- Putting now ==> ' +...
user1189332
0

votes
0

answer
4

Views

Spring Data Flow JDBC Sink JSON

I've created a custom sink off of the JDBC Sink Starter. The input to the sink is JSON content, but when the sink receives the message I receive the following error in the logs: Property or field 'field' cannot be found on object of type 'byte[]' I'm using the property jdbc.columns as follows: jdbc....
dardo
1

votes
1

answer
89

Views

String Cloud Stream httpclient processor expected-response-type as JSON

I am using httpclient processor as one of my steps, as follows (specified in Shell syntax as I'm testing locally, newlines added for readability here): dataflow:>stream create --name test | httpclient --httpclient.urlExpression='''https://host/restUrl''' --httpclient.body-expression=null --heade...
user944849
1

votes
1

answer
377

Views

Spring Cloud Stream Kafka consumer with time based auto commit

I have a Spring Cloud Stream app using the Kafka binder. It appears to be committing offsets automatically based on AckMode=BATCH (had to find that in debug mode as it doesn't appear documented). This is causing significant overhead to the broker in commit frequency. I would like to either switch t...
Kevin Niemann
1

votes
1

answer
208

Views

spring-cloud-stream-binder-kstream-1.3.0: IllegalAccessError cannot access its superinterface

Trying with spring-cloud-stream-binder-kstream 1.3.0.RELEASE version pom.xml: 4.0.0 kafka-kstream-processor kafka-kstream-processor demo application org.springframework.boot spring-boot-starter-parent 1.5.7.RELEASE UTF-8 UTF-8 1.8 Dalston.SR3 org.springframework.boot spring-boot-starter-web org.sp...
Barath
1

votes
1

answer
452

Views

How to scale the spring cloud data flow server

Normally we run the Jar of spring cloud data flow in one of the machine, but what if over the period we create many flows on the machine and the server gets overloaded and becomes a single point of failure, Do we have some thing where we can run the spring cloud data flow server jar on another machi...
shishir ranjan
1

votes
1

answer
765

Views

PCF Scheduling jobs

I have been trying to schedule spring cloud task via PCF scheduler, however I can't create a job from the app/task (following this documentation on the site - http://docs.pivotal.io/pcf-scheduler/1-1/using-jobs.html) $ cf apps name requested state instances memory...
Nitty

View additional questions