Questions tagged [aws-data-pipeline]

0

votes
0

answer
2

Views

Has anyone used AWS system manager parameter in data pipeline, to allocate value to a parameter in pipeline?

'id': 'myS3Bucket', 'type': 'String', 'default': '\'aws ssm get-parameters --names variable --query \'Parameters[*].{myS3Bucket:Value}\'\'' I tried this , Where I created a variable in AWS parameter and was able to retrieve the value using this command in AWS CLI, but not able to retrieve the value...
user11204973
0

votes
2

answer
15

Views

AWS to GCP data migration

I'm planning a Data Migration from AWS MySQL instances to GCP BigQuery. I don't want to migrate every MySQL Database because finally I want to create a Data Warehouse using BigQuery. Would exporting AWS MySQL DB to S3 buckets as csv/json/avro, then transfer to GCP buckets be a good option? What woul...
Buddhika Sameera
0

votes
0

answer
3

Views

AWS data pipeline execute redshift query for every line in s3 file

I have partitioned file (few files with same name template) in s3. In every file there are few lines of data. I would like to execute redshift for each line from file while substituting paramters from file line. I need to do it in AWS data pipelint For example file val1,val2,val3 val4,val5,val6 val...
Evgeny Makarov