Questions tagged [boto3]

1

votes
0

answer
394

Views

Upload a file asynchronously to S3 in Flask request

I have a Flask API that uploads files. Currently I am working on scaling the application. I have a scenario where a user uploads a large file - in that case, I save the file locally in the flask request and then kick off a celery task that then uploads it to S3 (and does other things). My problem n...
George B
1

votes
0

answer
319

Views

boto3 attached device name

I am trying to get the device name where an EBS volume is attached. I can pull the attach data, but would just like to isolate the device name. import boto3 from botocore.client import ClientError ec2 = boto3.resource('ec2') filters = [{ 'Name': 'tag:app', 'Values': [ 'splunklogidx' ], 'Name': 'tag...
outOfTune
1

votes
0

answer
221

Views

pd.read_csv killed when Reading large files

I have 30 each has 2 gb of data compressed files in s3 location and I'm trying to decompress the files, convert them to data frames and then want to subset the data based on column names. When I'm running the code it is getting Killed(Please refer the screenshot for error). Please help me out how to...
DPs
1

votes
1

answer
1.3k

Views

Downloading multiple S3 objects in parallel in Python

Is there a way to concurrently download S3 files using boto3 in Python3? I am aware of the aiobotocore library, but I would like to know if there is a way to do it using the standard boto3 library.
Tomer
1

votes
1

answer
195

Views

boto3 support for role_arn

I want to insert data from AWS Lambda to AWS DynamoDB and upload a file from Lambda to AWS S3. Currently, I'm using Access Key and Secret Key to do this from boto3. But, due to security issues, I had to change this approach to role-based approach. (i.e) To insert data in DDB or uploading a file to...
Droid
1

votes
1

answer
242

Views

dynamodb scan not working with IN condition

I'm failing to get the IN condition in my dynamodb scan call to work correctly. Background I'm trying to collect all permissions a user have based on the the roles associated with him, based on this data structure: # Users | user | roles | admin | ['system-admin', 'basic-user'] # Roles | role...
Martin Schulze
1

votes
1

answer
450

Views

AWS boto3 — Difference between `batch_writer` and `batch_write_item`

I'm currently applying boto3 with dynamodb, and I noticed that there are two types of batch write batch_writer is used in tutorial, and it seems like you can just iterate through different JSON objects to do insert (this is just one example, of course) batch_write_items seems to me is a dynamo-speci...
JChao
1

votes
0

answer
211

Views

How to upload private files to s3 using boto3 and serializer

I am using S3Boto3Storage to manage private file uploads to s3. When I upload via model, the upload works as expected. When I upload via serializer, I get this error: botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found Here are what my files look...
user3015876
1

votes
0

answer
1.3k

Views

python code gets error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

I got a error like this botocore.exceptions.NoCredentialsError: Unable to locate credentials when I was running the following code: # setup AWS Connection details awsSession = boto3.Session(profile_name='opsdev') def getInfoFromDynamoDB(service): client = awsSession.client('kms') dynamodb = awsSessi...
Besides Penguin
1

votes
1

answer
557

Views

PySpark and Pandas - Read partitioned csv files from S3 skipping empty ones

Using PySpark, I have some code that runs through a bunch of queries. for index, query in enumerate(query_map): spark_dataframe.filter( query).write.csv('s3://OutputBucket/Csvs/Query_{}'.format(index) I'm new to spark, but I understand that each partition is writing individual csv files to a direc...
jwillis0720
1

votes
2

answer
421

Views

Attaching role while creating boto3 ec2 Instance Error

I just started to read about AWS and am trying to create an ec2 instance and attach a role to it. As far as i have read i need to create an instance profile, attach a role to it and then attach it to EC2 instance but i am unable to do so because of the errors described below. I dont have enough time...
Suman Sourav Singh
1

votes
2

answer
790

Views

How do I correctly use the CloudWatch boto3 API to retrieve data from metrics?

I'm using Python3's boto3 to try to pull data from SQS's metrics using the get_metric_statistics function documented here: https://boto3.readthedocs.io/en/latest/reference/services/cloudwatch.html#CloudWatch.Client.get_metric_statistics This is the code I made to try to pull it: import boto3 import...
Helios
1

votes
1

answer
3.8k

Views

import boto3 error on Mac OS

I'm trying to install boto3 on my mac (high sierra 10.13.3) and tried to follow : https://github.com/boto/boto3. I had already installed python 3 using homebrew before, but when I tried to see pip --version, I get error. So, I did 1) modify .bash_profile to add alias pip=pip3 2) verify $ pip --versi...
Suvro Choudhury
1

votes
0

answer
465

Views

How can I wait for a dynamodb update table or ACTIVE state using boto3

I am changing the table, for example the capacity settings using boto3 then I need to wait for its completation I would prefer a solution using boto3.resource('dynamodb').Table('MyTable') instead of dynamodb client.
1

votes
1

answer
318

Views

Find aws ecs latest task and image used

I trying to find out the latest task and image used for ECS container service. I tried using AWS ECS cli, able to list services/tasks running but not able to find out running latest task ID and image used for that task. Can someone help with AWS CLI or Boto script to find out latest task and image u...
beginnertopython
1

votes
0

answer
1.3k

Views

boto3 s3.Object- - expected string or bytes-like object

I tried searching for a solution to this problem but still couldn't figure out the issue. So, I am trying to check if a key exists in a bucket. My code works perfectly fine if I use the following: s3.Object('bucket_name', 'key_name').load() But when I pass the reference variable rather than the actu...
1

votes
0

answer
257

Views

How to get profile data using AWS Cognito with Boto3 and OAuth?

Okay, so after a user logs in, I want to get their profile data. So the way I tried doing that was via boto3.client('cognito-idp').get_user but it keeps returning: NotAuthorizedException: An error occurred (NotAuthorizedException) when calling the GetUser operation: Access Token does not have requir...
Joseph Manley
1

votes
1

answer
656

Views

How do I use python to download an S3 file from a link with a signature and expiration?

I have a s3 link provided to me by a third-party with the following structure: http://s3.amazonaws.com/bucket_name_possibly/path/to/file_possibly/filename?AWSAccessKeyId=SomeKey&Expires=888888&Signature=SomeCharactersPossiblyHTMLencoded Clicking on the link downloads the file for me. However, in pyt...
user3776598
1

votes
1

answer
933

Views

boto s3 - write a csv file using array

I want to upload a file to s3 without writing the file on my local system, so I am using set_contents_from_string of boto library. I am able to upload this file back to s3. But how to write an array to a csv file using boto library. arrlist = [[2, 'jack'], [3, 'john'], [4, 'robert']] file_name = 'te...
usr_11
1

votes
1

answer
172

Views

Is there a boto3 funciton to convert authorization_code into authorization_token

My project is python and using boto3 lib. I'm using aws cognito Authorization code grant flow with return_type=code instead of return_type=token (implicit flow). Once my user is authorized my redirect url is injected with the queryStringParameter code=4d55a121-8ffc-4058-844b-xxxx. outlined here I n...
knittledan
1

votes
1

answer
140

Views

Updating item in dynamo with multiple expression, can't get delete working with SET

Trying to use multiple expression SET and DELETE. everything works fine for SET, But when i add DELETE, not able to figure out right syntax. status = 'Previously Deployed version' message = 'New version deployment started' NewVersion = 'PipelineTestAPI_1.5.0' json_ = {':val1' : status, ':val2' : me...
Meet101
1

votes
0

answer
53

Views

Get user from identityId

I saved one element in dynamodb using the id with AWS.config.credentials.identityId and restricting with ${cognito-identity.amazonaws.com:sub} This generates a us-east-1:14b37fe3-xxxx-xxxx-xxxx-xxxxxxxxxxxx record in dynamodb for a user with sub in cognito user pool: 20a3902b-xxxx-xxxx-xxxx-xxxxxxxx...
Lucas Paim
1

votes
1

answer
1.1k

Views

Can we fetch AWS Cloudtrail logs using boto3 in python?

Is a direct way to get the logs / events from cloudtrail by using boto3 in python. How can this be done ?
FCoding
1

votes
1

answer
236

Views

a presigned post in boto3 is the same as a browser based post with rest api signature calulucation server side?

for the past 3 days I have been attempting to manually sign restapi post calls from the users browser. I have failed. I continue to get a signature mismatch error: So now I am going to try boto3. Boto3 it seems is used when you upload a file to your server then send it to amazon. I would rather do a...
1

votes
4

answer
637

Views

How to import my AWS credentials for below mentioned python script to work?

import boto3 s3 = boto3.resource('s3') for bucket in s3.buckets.all(): print(bucket.name) but this code does not work, how to import my AWS credentials so that this will return correct output? error : botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: 'https://s3.aws...
Ganesh Shinde
1

votes
0

answer
159

Views

While downloading a folder contents from Amazon S3 using boto3, only some files get downloaded and for others, i get the following exception :

File 'C:\Program Files\Python36\lib\site-packages\boto3\s3\inject.py', line 130, in download_file extra_args=ExtraArgs, callback=Callback) File 'C:\Program Files\Python36\lib\site-packages\boto3\s3\transfer.py', line 307, in download_file future.result() File 'C:\Program Files\Python36\lib\site-pack...
Sruthi
1

votes
1

answer
155

Views

Boto3 - python script to view all directories and files

I tried to follow the Boto3 examples, but can literally only manage to get the very basic listing of all my S3 buckets via the example they give: import boto3 s3 = boto3.resource('s3') for bucket in s3.buckets.all(): print(bucket.name) I cannot find documentation that explains how I would be able to...
Monty
1

votes
1

answer
289

Views

How to authenticate session in application running on Beanstalk with boto3

My app is deployed via Elastic Beanstalk. It needs to access S3. I can do it locally with my own access key, but I don't want to store that anywhere when I deploy. Given that the instance is on Beanstalk, there must be an easier way to auth , perhaps with roles? I have given full S3 permissions to t...
turnip
1

votes
1

answer
183

Views

aioboto3 speedup not as expected

I'm trying the aioboto3 lib, which looks extremely promising for speeding certain tasks up. For example, I need to find tags for all S3 objects within a particular bucket and prefix. But unfortunately the speed increase is not what I had hoped for. With 1000 objects, it's about half the time. With...
dude102438
1

votes
2

answer
251

Views

Botocore fails to read credentials when run as daemon.service

I had my script running smoothly from command line, however, when I start it as a systemd.service, I get the following error: iot_local.service - My iot_local Service Loaded: loaded (/lib/systemd/system/iot_local.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun...
Nazar
1

votes
1

answer
844

Views

S3 Select with boto3 - internalerror

Has anyone got 'S3 Select' (https://aws.amazon.com/blogs/aws/s3-glacier-select/ , https://aws.amazon.com/about-aws/whats-new/2018/04/amazon-s3-select-is-now-generally-available/) with boto3 (or even cli or another sdk) working? I am getting cryptic InternalError below: Running this on EC2 that has...
tooptoop4
1

votes
1

answer
636

Views

AWS Lambda Python/Boto3/psycopg2 Redshift temporary credentials

I'm pretty new to AWS so please let me know if what I'm trying to do is not a good idea, but the basic gist of it is that I have a Redshift cluster that I want to be able to query from Lambda (Python) using a combination of psycopg2 and boto3. I have assigned the Lambda function a role that allows i...
GSkurski
1

votes
3

answer
1.4k

Views

Why does installing boto3 break my installation of awscli?

I am trying to install boto3 so I can use it in my python script that I want to run on an EC2 instance running Ubuntu 14.04. Anyway I install boto3 it breaks my installation of awscli on the non root user. I install awscli using apt get with sudo apt-get install awscli. After I install it, it works...
Shawn Volpe
1

votes
2

answer
519

Views

Saving boto3 output to JSON

Need to save boto3 output (as a backup) to JSON file #!/bin/python import boto3 import json client = boto3.client('iam') response = client.get_account_authorization_details( ) sys.stdout = open('output.json','wt') print response i got output: {u'RoleDetailList': [{u'AssumeRolePolicyDocument': {u'Ve...
Milister
1

votes
1

answer
105

Views

how to handle internet not avaialble while uploading the file to s3?

I have written scheduler program in python to uploading file to AWS S3 at a particular time(using scheduler). if internet connection is not available at that time , I'm getting socket.gaierror: [Errno -3] Temporary failure in name resolution exception. This breaks the total flow, so that i need to r...
muthu
1

votes
1

answer
828

Views

Having trouble while fetching data from PYTHON script using AWS Athena-Boto3

I am trying to query the dataset present in s3 bucket, using Athena query via python script with help of boto3 functions. I am using start_query_execution() to run my query. this is being executed perfectly, next to get results in my python script, so that I get access to the result of the query I...
mindaJalaj
1

votes
1

answer
245

Views

How can I use python (BOTO3) to cycle through all AWS EC2 instances and remove a security group from them? [duplicate]

This question already has an answer here: BOTO3 — Attach / Detach Security Group from EC2 instance 1 answer I am looking for a script that will cycle through all of my AWS EC2 instances, check for a specific security group, and then delete that group from the instance. I've been reading through t...
RaptorPete
1

votes
0

answer
459

Views

BOTO3 DynamoDB error: delete an item from a LIST attribute in a dynamodb table

we have a dynamoDB table with 'item_id' as primary/hash key. One of the attributes 'my_list' is a list of strings. I am trying to remove index '0' from the list. When I hard code the index, it works: item_key = {'item_id': {'S': item_id}} update_expression = 'remove mylist[0]' response = client.upda...
Bas
1

votes
1

answer
225

Views

Setting boto3 dynamodb endpoint_url globaly

I want to use dynamodb local for local (unit) testing of my python/boto3 based application. Now I can do something like this if test_mode: client = boto3.client('dynamodb', endpoint_url='localhost:8000') resource = boto3.resource('dynamodb', endpoint_url='localhost:8000') else: client = boto3.client...
Nathan
1

votes
1

answer
193

Views

Using Boto3 to send S3 Put Requests to SNS

I'm trying to get S3 put requests sent to an SNS topic. My code looks like this: s3.put_bucket_notification_configuration( Bucket=event_name, NotificationConfiguration={ 'TopicConfigurations': [ { 'TopicArn': sns_arn, 'Events': [ 's3:ObjectCreated:*', ], }, ] } ) Unfortunately, I'm getting an except...
user3816764

View additional questions