Questions tagged [boto3]
1075 questions
1
votes
2
answer
1.3k
Views
AWS Lambda boto3: Filter Instances by Launch Time = today
I'm trying to automatically backup my EC2 instances through AWS Lambda.
I used these 2 scripts and they work just fine.
https://gist.github.com/bkozora/724e01903a9ad481d21e
https://gist.github.com/bkozora/d4f1cf0e5cf26acdd377
But I would like to add the functionality that the scripts only delete or...
1
votes
2
answer
334
Views
get the Alarm object of CloudWatch using boto 2
I created an alarm and want to delete it afterward...
The documentation for boto 2 doesn't show how to do that.
Any help ?
Thanks
1
votes
1
answer
524
Views
AWS CLI command for querying S3 bucket
I have a bucket which I will call some-images for showing here. I am trying to get my command to return all objects that have a key value which starts_with a number. Here is the code I am using:
cmd = 'aws s3api list-objects --bucket some-images --query \'Contents[?starts_with(Key, '10000'\''
push =...
0
votes
0
answer
5
Views
how to create endpoints and send messages in amazon pinpoint
im trying to use boto3 to send messages with pinpoint to, but the functions for update_endpoint and send_messages keeps return me Resource not found
client = boto3.client('pinpoint', region_name='us-east-1', aws_access_key_id='xxx', aws_secret_access_key='xxx')
client.update_endpoint(ApplicationId='...
0
votes
0
answer
5
Views
Generate Code (java, python, etc.) from CloudFormation Template
Is there a mechanism/framework available to generate code from a CloudFormation template?
1
votes
1
answer
988
Views
S3: ExpiredToken error for S3 pre-signed url within expiry period
This is how I am generating pre-signed url for an S3 object from my python script.
s3client = boto3.client('s3')
url = s3client.generate_presigned_url('get_object', Params={'Bucket': args.bucket, 'Key': dated_filename}, ExpiresIn=86400)
where I am giving an expiry of 24 hours.
When I try to downloa...
1
votes
1
answer
95
Views
Creating default VPC using boto3?
Is there a way to do an equivalent of aws ec2 create-default-vpc using boto3? (more generally, I'm wondering if there is a way to figure out boto3/botocore equivalent of aws CLI directive)
1
votes
2
answer
43
Views
Amazon SQS for browser polling?
I would like to know if AWS SQS is the right service for doing browser polling.
For example :
1) User acesses application through browser, and requests a large PDF to be generated
2) API responds back with 'OK' to user and forwards the request to SQS
3) SQS queue is being read by a lambda which gene...
1
votes
3
answer
82
Views
How to get top-level folders in an S3 bucket using boto3?
I have an S3 bucket with a few top level folders, and hundreds of files in each of these folders. How do I get the names of these top level folders?
I have tried the following:
s3 = boto3.resource('s3', region_name='us-west-2', endpoint_url='https://s3.us-west-2.amazonaws.com')
bucket = s3.Bucket('X...
0
votes
0
answer
4
Views
Downloading files from AWS S3 Bucket with boto3 results in ClientError: An error occurred (403): Forbidden
I am trying to download files from a s3 bucket by using the Access Key ID and Secret Access Key provided by https://db.humanconnectome.org. However, even though I am able to navigate the database and find the files (as I have configured my credentials via aws cli), attempting to download them result...
0
votes
0
answer
4
Views
Using Python Boto3 AWS Lambda, S3 Key LastModified returns in two types for two different ways
The LastModified of a given key returns in two different formats.
If I store in a variable or directly print, it gives me this format:
2019-04-17 11:14:11+00:00
And
If I try to store it as a value in a dict, and then print it, it shows in this format:
{'redshift_data_source/grid/load.csv': datetim...
1
votes
0
answer
394
Views
Upload a file asynchronously to S3 in Flask request
I have a Flask API that uploads files. Currently I am working on scaling the application.
I have a scenario where a user uploads a large file - in that case, I save the file locally in the flask request and then kick off a celery task that then uploads it to S3 (and does other things).
My problem n...
1
votes
0
answer
319
Views
boto3 attached device name
I am trying to get the device name where an EBS volume is attached. I can pull the attach data, but would just like to isolate the device name.
import boto3
from botocore.client import ClientError
ec2 = boto3.resource('ec2')
filters = [{
'Name': 'tag:app',
'Values': [ 'splunklogidx' ],
'Name': 'tag...
1
votes
0
answer
221
Views
pd.read_csv killed when Reading large files
I have 30 each has 2 gb of data compressed files in s3 location and I'm trying to decompress the files, convert them to data frames and then want to subset the data based on column names. When I'm running the code it is getting Killed(Please refer the screenshot for error). Please help me out how to...
1
votes
1
answer
1.3k
Views
Downloading multiple S3 objects in parallel in Python
Is there a way to concurrently download S3 files using boto3 in Python3?
I am aware of the aiobotocore library, but I would like to know if there is a way to do it using the standard boto3 library.
1
votes
1
answer
195
Views
boto3 support for role_arn
I want to insert data from AWS Lambda to AWS DynamoDB and upload a file from Lambda to AWS S3. Currently, I'm using Access Key and Secret Key to do this from boto3.
But, due to security issues, I had to change this approach to role-based approach. (i.e) To insert data in DDB or uploading a file to...
1
votes
1
answer
242
Views
dynamodb scan not working with IN condition
I'm failing to get the IN condition in my dynamodb scan call to work correctly.
Background
I'm trying to collect all permissions a user have based on the the roles associated with him, based on this data structure:
# Users
| user | roles
| admin | ['system-admin', 'basic-user']
# Roles
| role...
1
votes
1
answer
450
Views
AWS boto3 — Difference between `batch_writer` and `batch_write_item`
I'm currently applying boto3 with dynamodb, and I noticed that there are two types of batch write
batch_writer is used in tutorial, and it seems like you can just iterate through different JSON objects to do insert (this is just one example, of course)
batch_write_items seems to me is a dynamo-speci...
1
votes
0
answer
211
Views
How to upload private files to s3 using boto3 and serializer
I am using S3Boto3Storage to manage private file uploads to s3. When I upload via model, the upload works as expected.
When I upload via serializer, I get this error:
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
Here are what my files look...
1
votes
0
answer
1.3k
Views
python code gets error: botocore.exceptions.NoCredentialsError: Unable to locate credentials
I got a error like this botocore.exceptions.NoCredentialsError: Unable to locate credentials when I was running the following code:
# setup AWS Connection details
awsSession = boto3.Session(profile_name='opsdev')
def getInfoFromDynamoDB(service):
client = awsSession.client('kms')
dynamodb = awsSessi...
1
votes
1
answer
557
Views
PySpark and Pandas - Read partitioned csv files from S3 skipping empty ones
Using PySpark, I have some code that runs through a bunch of queries.
for index, query in enumerate(query_map):
spark_dataframe.filter(
query).write.csv('s3://OutputBucket/Csvs/Query_{}'.format(index)
I'm new to spark, but I understand that each partition is writing individual csv files to a direc...
1
votes
2
answer
421
Views
Attaching role while creating boto3 ec2 Instance Error
I just started to read about AWS and am trying to create an ec2 instance and attach a role to it. As far as i have read i need to create an instance profile, attach a role to it and then attach it to EC2 instance but i am unable to do so because of the errors described below.
I dont have enough time...
1
votes
2
answer
790
Views
How do I correctly use the CloudWatch boto3 API to retrieve data from metrics?
I'm using Python3's boto3 to try to pull data from SQS's metrics using the get_metric_statistics function documented here:
https://boto3.readthedocs.io/en/latest/reference/services/cloudwatch.html#CloudWatch.Client.get_metric_statistics
This is the code I made to try to pull it:
import boto3
import...
1
votes
1
answer
3.8k
Views
import boto3 error on Mac OS
I'm trying to install boto3 on my mac (high sierra 10.13.3) and tried to follow : https://github.com/boto/boto3. I had already installed python 3 using homebrew before, but when I tried to see pip --version, I get error.
So, I did
1) modify .bash_profile to add
alias pip=pip3
2) verify
$ pip --versi...
1
votes
0
answer
465
Views
How can I wait for a dynamodb update table or ACTIVE state using boto3
I am changing the table, for example the capacity settings using boto3
then I need to wait for its completation
I would prefer a solution using boto3.resource('dynamodb').Table('MyTable') instead of dynamodb client.
1
votes
1
answer
318
Views
Find aws ecs latest task and image used
I trying to find out the latest task and image used for ECS container service. I tried using AWS ECS cli, able to list services/tasks running but not able to find out running latest task ID and image used for that task. Can someone help with AWS CLI or Boto script to find out latest task and image u...
1
votes
0
answer
1.3k
Views
boto3 s3.Object- - expected string or bytes-like object
I tried searching for a solution to this problem but still couldn't figure out the issue. So, I am trying to check if a key exists in a bucket. My code works perfectly fine if I use the following:
s3.Object('bucket_name', 'key_name').load()
But when I pass the reference variable rather than the actu...
1
votes
0
answer
257
Views
How to get profile data using AWS Cognito with Boto3 and OAuth?
Okay, so after a user logs in, I want to get their profile data. So the way I tried doing that was via boto3.client('cognito-idp').get_user but it keeps returning:
NotAuthorizedException: An error occurred (NotAuthorizedException) when calling the GetUser operation: Access Token does not have requir...
1
votes
1
answer
656
Views
How do I use python to download an S3 file from a link with a signature and expiration?
I have a s3 link provided to me by a third-party with the following structure: http://s3.amazonaws.com/bucket_name_possibly/path/to/file_possibly/filename?AWSAccessKeyId=SomeKey&Expires=888888&Signature=SomeCharactersPossiblyHTMLencoded
Clicking on the link downloads the file for me. However, in pyt...
1
votes
1
answer
933
Views
boto s3 - write a csv file using array
I want to upload a file to s3 without writing the file on my local system, so I am using set_contents_from_string of boto library. I am able to upload this file back to s3.
But how to write an array to a csv file using boto library.
arrlist = [[2, 'jack'], [3, 'john'], [4, 'robert']]
file_name = 'te...
1
votes
1
answer
172
Views
Is there a boto3 funciton to convert authorization_code into authorization_token
My project is python and using boto3 lib.
I'm using aws cognito Authorization code grant flow with return_type=code instead of return_type=token (implicit flow). Once my user is authorized my redirect url is injected with the queryStringParameter code=4d55a121-8ffc-4058-844b-xxxx.
outlined here
I n...
1
votes
1
answer
140
Views
Updating item in dynamo with multiple expression, can't get delete working with SET
Trying to use multiple expression SET and DELETE. everything works fine for SET, But when i add DELETE, not able to figure out right syntax.
status = 'Previously Deployed version'
message = 'New version deployment started'
NewVersion = 'PipelineTestAPI_1.5.0'
json_ = {':val1' : status,
':val2' : me...
1
votes
0
answer
53
Views
Get user from identityId
I saved one element in dynamodb using the id with AWS.config.credentials.identityId and restricting with ${cognito-identity.amazonaws.com:sub}
This generates a us-east-1:14b37fe3-xxxx-xxxx-xxxx-xxxxxxxxxxxx record in dynamodb for a user with sub in cognito user pool: 20a3902b-xxxx-xxxx-xxxx-xxxxxxxx...
1
votes
1
answer
1.1k
Views
Can we fetch AWS Cloudtrail logs using boto3 in python?
Is a direct way to get the logs / events from cloudtrail by using boto3 in python. How can this be done ?
1
votes
1
answer
236
Views
a presigned post in boto3 is the same as a browser based post with rest api signature calulucation server side?
for the past 3 days I have been attempting to manually sign restapi post calls from the users browser. I have failed. I continue to get a signature mismatch error: So now I am going to try boto3. Boto3 it seems is used when you upload a file to your server then send it to amazon. I would rather do a...
1
votes
4
answer
637
Views
How to import my AWS credentials for below mentioned python script to work?
import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print(bucket.name)
but this code does not work, how to import my AWS credentials so that this will return correct output?
error : botocore.exceptions.EndpointConnectionError: Could not connect
to the endpoint URL: 'https://s3.aws...
1
votes
0
answer
159
Views
While downloading a folder contents from Amazon S3 using boto3, only some files get downloaded and for others, i get the following exception :
File 'C:\Program Files\Python36\lib\site-packages\boto3\s3\inject.py', line 130, in download_file
extra_args=ExtraArgs, callback=Callback)
File 'C:\Program Files\Python36\lib\site-packages\boto3\s3\transfer.py', line 307, in download_file
future.result()
File 'C:\Program Files\Python36\lib\site-pack...
1
votes
1
answer
155
Views
Boto3 - python script to view all directories and files
I tried to follow the Boto3 examples, but can literally only manage to get the very basic listing of all my S3 buckets via the example they give:
import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print(bucket.name)
I cannot find documentation that explains how I would be able to...
1
votes
1
answer
289
Views
How to authenticate session in application running on Beanstalk with boto3
My app is deployed via Elastic Beanstalk. It needs to access S3. I can do it locally with my own access key, but I don't want to store that anywhere when I deploy. Given that the instance is on Beanstalk, there must be an easier way to auth , perhaps with roles?
I have given full S3 permissions to t...
1
votes
1
answer
183
Views
aioboto3 speedup not as expected
I'm trying the aioboto3 lib, which looks extremely promising for speeding certain tasks up. For example, I need to find tags for all S3 objects within a particular bucket and prefix. But unfortunately the speed increase is not what I had hoped for.
With 1000 objects, it's about half the time.
With...