Questions tagged [factory]

1

votes
3

answer
229

Views

Data factory lookup (dot) in the item() name

I am having lookup wherein salesforce query is there. I am using elements (item()) in subsequent activities. Till now i had item().name or item().email but now i have item().NVMStatsSF__Related_Lead__r.FirstName which has (dot) in the field name. How should i parse it through body tag so that it rea...
Gagan
1

votes
0

answer
216

Views

Azure Data Factory V2 Trigger Pipeline only when current execution finished

Is there a way I can create a trigger in Azure Data Factory that will only kick my pipeline if it's not already running? I basically want to run a pipeline continuously and so for whatever reason it finished (failure / success) I want it to resume as soon as possible.
Ryan Papke
1

votes
1

answer
339

Views

Query a specific time range data from cosmos db and store it in sql database via azure data factory

I am using Azure Data Factory V2 and want to copy the json data stored as documents from Azure cosmos db to a azure sql table, using a SQL query. Situation: I want to query just the last hour json files from Cosmos db and store it in sql table. To do that, I create a new Trigger in my Data factory w...
MahmoudSwaid
1

votes
2

answer
42

Views

One of my tests fails due to some issues with using let! in rspec

I have no clue what is wrong with the following: let!(:account1) { create(:account) } describe '.active_between' do subject { Account.active_between(Date.today - 10.days, Date.today).to_a } it { is_expected.to eq([account1]) } end Somehow this tests fails like this: expected: [#
bo-oz
1

votes
0

answer
100

Views

factory_boy DjangoModelFactory not creating Model entries for testing

I am trying to test my django app using factory_boy to setup the model instances. I have the following factories setup for certain Django models. class StudyFactory(DjangoModelFactory): class Meta: model = Study group = SubFactory(GroupFactory) @create_attrs(values) class SomeFactory(DjangoModelFact...
Ry John
1

votes
0

answer
48

Views

FactoryGirl - is it possible to define a factory that will create only one instance and always return that one instance?

Say I have this in multiple specs... let(:thing){FactoryGirl.create(:thing)} ...is there a way to set up the factory such that the first time it's used to create a Thing it creates it, but the 2nd and following time it's used, it returns the instance it created on the first call?
blogofsongs
1

votes
1

answer
123

Views

rails minitest is driving me insane with PG::UniqueViolation errors

Half of the time I'll be able to make my tests run. The other half they fail because of a uniqueness violation, the source of which I am unable to locate. Right now I am in the latter half. My error is this: ItemTest#test_valid_setup: ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: dupl...
sabaeus
1

votes
2

answer
88

Views

Is after_update run when calling FactoryBot.create?

On my test I'm seeing that after_update is been called after calling FactoryBot.create(:object). Is it normal? As far as I know, it should be called only when a record gets updated, no?
sekmo
1

votes
0

answer
207

Views

Execute a PySpark Job with Dependencies using Azure Data Factory V2

I'd like to execute a PySpark Job with dependencies (either egg or zip files) using Data Factory V2. When running the command directly on the Head Node cluster (HD Insight) in the form of a spark-submit method it goes as following (and works): spark-submit --py-files 0.3-py3.6.egg main.py 1 In Data...
gilibi
1

votes
1

answer
292

Views

execute scala jar file in azure data factory

Here is the code I am trying to execute : SimpleApp.scala package test import java.sql.DriverManager import org.apache.spark.SparkContext import org.apache.spark.SparkConf object SimpleApp { def main(args: Array[String]) { val conf = new SparkConf().setAppName('bouh').setMaster('yarn') val sc = new...
Steven
1

votes
0

answer
163

Views

Using Azure Data Factory 2, I can not copy containers that have a large number of blobs

I'm having a problem with Azure Data Factory V2, where containers with a large number of blobs are not copied. One pipeline run went almost 75 hours before it finally failed with an error. Activity Copy1 failed: ErrorCode=FailedStorageOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliv...
ASalvo
1

votes
0

answer
132

Views

Azure Data Factory v2 - OData to Cosmos DB

I have set up an oData to Cosmos DB Pipeline in Azure Data Factory v2. I was using this service to test out the functionality: http://services.odata.org/V3/OData/OData.svc/ I am mapping one string field to another but I keep receiving the following error: { 'errorCode': '1000', 'message': 'ErrorCod...
robboc
1

votes
0

answer
214

Views

Error Id: VertexFailedFast, Error Message: Vertex failed with a fail-fast error

when I run the Below U SQL I am getting 'Activity U-SQL1 failed: Error Id: VertexFailedFast, Error Message: Vertex failed with a fail-fast error.' The input schema has 7 columns. But still i am getting this error.I'm also skipping the first row because it contains headers. DECLARE @file_set_path st...
vignesh asokan
1

votes
1

answer
119

Views

M requires a parameter Validation Error

I have a bunch of Copy Pipelines in Azure Data Factory (v2). They've been working fine up until today. Now whenever I try to publish changes, I am getting the following validation error: M requires a parameter And of course with a validation error I can't save my changes. Any dieas as to what I migh...
Steve Kaschimer
1

votes
1

answer
344

Views

Dataset error message - PdwManagedToNativeInteropException

Currently have a pipeline running in our production environment that has an activity that copies data from an on prem sql database to sql azure database. This pipeline is replicated among the dev and QA environments but don't fail in those environments. Wanted to get a bit more insight as to what th...
Saj
1

votes
1

answer
182

Views

“Value cannot be null.\r\nParameter name: endpoint” in Azure Data Factory V2

I am getting the following error when I execute Azure ML Batch Execution Activity in ADF V2. I have written following JSON query in ML Activity { 'name': 'MLBatchExecution1', 'description': '', 'type': 'AzureMLBatchExecution', 'linkedServiceName': { 'name': 'AzureMLLinkedservice2', 'type': 'AzureML'...
DEEPAK RAJ
1

votes
2

answer
771

Views

Error in running Azure Data Factory Pipeline. Linked Service Reference not found

I am facing the below issue in creating an Azure Machine Learning Batch Execution activity to execute a scoring ML experiment. Please help: Please let me know if any other relevant information is needed. I am new to this so, please help Created an AzureML Linked Service as below: { 'name': 'Predicti...
Utsav Datta
1

votes
1

answer
85

Views

Cannot access files on FTP server from Azure Data Factory

I currently have access to a third party's FTP server which, upon login, automatically redirects me to a directory that does not contain the files I am trying to download. ftp://ftp.fakehost.com -> ftp://ftp.fakehost.com/uselessDir My files are in ftp://ftp.fakehost/usefulDir. This ftp server does n...
Thomas Pouget
1

votes
1

answer
151

Views

Django Generic Factory

I have two questions that bother me. I'm facing an implementation where some document are related to different level of geodata and would like a factory to generate them. Let's see an example how I thought it may work: from django.contrib.gis.db import models class Country(models.Model): name = mode...
Merichbier
1

votes
3

answer
134

Views

Why are ADF datasets important?

In Azure Data Factory v2 I've created a number of pipelines. I noticed that each pipeline I create there is a source and destination dataset created. According to the ADF documentation: A dataset is a named view of data that simply points or references the data you want to use in your activities as...
Randy Minder
1

votes
1

answer
219

Views

Data Factory copy activity and ZipDeflate throughput low

I currently have a pipeline comprised of a simple Copy activity which fetches zipped files from a third party's FTP server, unzips the files and copies them to a blob storage container. The throughput is incredibly low (7 KB/s) and simply copying the files without unzipping does not pose any issue (...
Thomas Pouget
1

votes
1

answer
17

Views

Destroying created objects from factory (dangling pointer issue)

Suppose I have a factory class with a creation method that creates objects and also has a shutdown option that invalidates all the created objects (so they cannot be used anymore). something like that (C#): public class MyFactory { public MyObject CreateObject() { ... } public void DestroyAllObjects...
iradicatoratfb
1

votes
1

answer
559

Views

Error while trying to read file in Data Lake storage

In my Azure Data Lake Store I seek to read a file that I imported using a pipeline in Azure Data Factory 2. Although I am logged in with the same credentials that I used to create the Data Factory, the App Registration for the Data Factory, and the Data Lake itself, I get the following error message...
Michelle Turner
1

votes
1

answer
28

Views

How do you create factories for classes with dependencies

If I have a class that requires a dependency through its constructor: public MyClass( IDependencyInterface dependency ) and the dependency uses a factory/builder (it is still confusing to me the distinction between the two) to set its parameters: IDependencyInterface dependency = dependencyFactory ....
WhatIsASyntax
1

votes
1

answer
255

Views

Can't run Azure data factory from Azure function

This is my c# Azure Function script: #r 'System.Runtime' #r 'System.Threading.Tasks' using System; using System.Net; using System.Threading.Tasks; using Newtonsoft.Json; using Microsoft.Azure; using Microsoft.Azure.Management.DataFactory; using Microsoft.Azure.Management.DataFactory.Models; using Mi...
das
1

votes
1

answer
204

Views

Loading with ADF to DW - Error happened when loading data into SQL Data Warehouse

I created a pipeline with Azure Data Factory that copies tables from Azure sql to Azure DW (with staging blob storage) and one of the copy operations gave me this nice error: { 'errorCode': '2200', 'message': 'ErrorCode=FailedDbOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryExcep...
alvipeo
1

votes
1

answer
82

Views

Refreshing cache in Dependency Injection

I am new to development in C# so still figuring a few things out. I am trying to stick to good design principals so as to keep my code maintainable. So doing I am building a Line of Business application using the MVVM and Factory patterns. I am also using Dependency Injection (Unity). My question is...
Tony
1

votes
0

answer
38

Views

Idioms receiving by reference and private constructor are incompatible?

I have Some class and Factory for this class. Factory is friend to Some, and constructor of Some is private. I do it because I want that everyone can create new new instance of Some only via Factory. Something like that: class Some { friend class Factory; Some(..) {} public: Some& operator=(const So...
Stepan Loginov
1

votes
1

answer
326

Views

how to run a shell script from Azure data factory.

how to run a shell script from Azure data factory. Inside the shell script i am trying to execute a hql file like below /usr/bin/hive -f 'wasbs://[email protected]/hivetest/extracttemp.hql' > wasbs://[email protected]/hivetest/ext...
user4127818
1

votes
1

answer
127

Views

Multi-character column delimiter not supported in ADF connection

My text file has a delimiter with 2 characters (~*). However ADF supports column delimiters with only one chararacter. Any help or workaround? It is a text file present in Azure blob storage which needs to be imported to SQL database.
Vidya Sagar
1

votes
1

answer
21

Views

Can the waiting time in Wait Acitivity be variable?

I am asking this, because I need to check whether or not a file has been already copied by some pipe line in a different suscription. Thus,I need to check on time basis this event, I want to increase the value of the waiting activity each time the copy activity fails, because this last one did not f...
FelipePerezR
1

votes
1

answer
790

Views

Azure Data Factory mapping 2 columns in one column

Can somebody help me solving the error am getting in concatenating the two columns i.e first name and last name from my text file and merging the two columns into one name column in my Azure SQL database as a sink in Azure Data Factory and another question is that I want to choose the first letter o...
sxb1649
1

votes
0

answer
72

Views

Azure Data Factory truncating files in a copy pipeline from an S3 bucket to ADLS

I have a copy pipeline set up to copy a handful of files from a daily folder in an S3 bucket into a data lake in Azure using data factory. I run into this really weird issue. Suppose there are three files in the S3 bucket. One is 30MB, another is 50MB, and the last is 70MB. If I put the 30M file 'f...
ceilingfan999
1

votes
1

answer
942

Views

Unable to publish Azure Data factory Pipeline changes

I have created a simple data factory pipeline for copying files from azure blob storage to azure data lake. For this i have used one event based trigger. Trigger will automatically run pipeline if new blob will come to the blob storage location. If i am publishing my pipeline with my trigger in stop...
TechGuru
1

votes
1

answer
386

Views

Azure Data factory copy activity failed mapping strings (from csv) to Azure SQL table sink uniqueidentifier field

I have an Azure data factory (DF) pipeline that consists a Copy activity. The Copy activity uses HTTP connector as source to invoke a REST end-point and returns csv stream that sinks with Azure SQL Database table. The Copy fails when CSV contains strings (such as 40f52caf-e616-4321-8ea3-12ea3cbc54e...
Lee
1

votes
2

answer
202

Views

Azure Data Factory V2: MDX Query on SAP BW exception Microsoft.DataTransfer.Common.Shared.HybridDeliveryException

I am trying to fetch data from SAP BW system into Azure Data Lake using MDX Query in SAP BW connector. But I am getting exception message in Azure is following : { 'errorCode': '2200', 'message': 'Failure happened on 'Source' side. ErrorCode=UserErrorInvalidDataValue,'Type=Microsoft.DataTransfer.Com...
PSM
1

votes
1

answer
707

Views

Azure ADF v2- How to add/append timestamp to file name getting deployed to Azure Data Lake

I am deploying a file to azure datalake but want to append datetime stamp to filename . I created a dataset and created a parameter with value as utcnow(). But file is not getting deployed with timestamp appended. { 'name': 'tst', 'properties': { 'linkedServiceName': { 'referenceName': 'LS_ADLS_Azu...
cheers519
1

votes
0

answer
57

Views

Azure datafactory v2 -dataset copy activty -zipdeflate

I am trying to deploy Azure dataset using PowerShell. While doing this activity, I was uploading json file to ADF. Issue: I am uploading json file with below format. 'compression': { 'type': 'ZipDeflate', 'level': 'Optimal' } but when I m checking at Azure environment ,it is changed like below , 'co...
ARAVIND eie
1

votes
1

answer
141

Views

Has anyone managed to successfully pass parameters to SSIS via a Data Factory ExecuteSSISPackage task?

I am trying to pass parameters to SSIS via a Data Factory ExecuteSSISPackage task. I have added project parameters, package parameters etc to the JSON code but they are always null in SSIS. An excerpt of the JSON code is shown below. 'packageParameters ': { 'X': { 'value': '@pipeline().parameters...
W Jenkins
1

votes
0

answer
132

Views

Correct way to use Autofac with Factory Pattern?

IN a Winforms project, I have set up Autofac and Factory Pattern and it looks like it's working as expected. However, I'm still not sure whether the following is best practice. The factory class is: public static class Client { public static readonly IRequestFactory RequestFactory = new RequestFacto...

View additional questions