Questions tagged [boto3]

1

votes
2

answer
1.3k

Views

AWS Lambda boto3: Filter Instances by Launch Time = today

I'm trying to automatically backup my EC2 instances through AWS Lambda. I used these 2 scripts and they work just fine. https://gist.github.com/bkozora/724e01903a9ad481d21e https://gist.github.com/bkozora/d4f1cf0e5cf26acdd377 But I would like to add the functionality that the scripts only delete or...
T. Moser
1

votes
2

answer
334

Views

get the Alarm object of CloudWatch using boto 2

I created an alarm and want to delete it afterward... The documentation for boto 2 doesn't show how to do that. Any help ? Thanks
Sam Gomari
1

votes
1

answer
524

Views

AWS CLI command for querying S3 bucket

I have a bucket which I will call some-images for showing here. I am trying to get my command to return all objects that have a key value which starts_with a number. Here is the code I am using: cmd = 'aws s3api list-objects --bucket some-images --query \'Contents[?starts_with(Key, '10000'\'' push =...
5pence
0

votes
0

answer
5

Views

how to create endpoints and send messages in amazon pinpoint

im trying to use boto3 to send messages with pinpoint to, but the functions for update_endpoint and send_messages keeps return me Resource not found client = boto3.client('pinpoint', region_name='us-east-1', aws_access_key_id='xxx', aws_secret_access_key='xxx') client.update_endpoint(ApplicationId='...
beren5000
0

votes
0

answer
5

Views
1

votes
1

answer
988

Views

S3: ExpiredToken error for S3 pre-signed url within expiry period

This is how I am generating pre-signed url for an S3 object from my python script. s3client = boto3.client('s3') url = s3client.generate_presigned_url('get_object', Params={'Bucket': args.bucket, 'Key': dated_filename}, ExpiresIn=86400) where I am giving an expiry of 24 hours. When I try to downloa...
pratpor
1

votes
1

answer
95

Views

Creating default VPC using boto3?

Is there a way to do an equivalent of aws ec2 create-default-vpc using boto3? (more generally, I'm wondering if there is a way to figure out boto3/botocore equivalent of aws CLI directive)
Yaroslav Bulatov
1

votes
2

answer
43

Views

Amazon SQS for browser polling?

I would like to know if AWS SQS is the right service for doing browser polling. For example : 1) User acesses application through browser, and requests a large PDF to be generated 2) API responds back with 'OK' to user and forwards the request to SQS 3) SQS queue is being read by a lambda which gene...
Mojimi
1

votes
3

answer
82

Views

How to get top-level folders in an S3 bucket using boto3?

I have an S3 bucket with a few top level folders, and hundreds of files in each of these folders. How do I get the names of these top level folders? I have tried the following: s3 = boto3.resource('s3', region_name='us-west-2', endpoint_url='https://s3.us-west-2.amazonaws.com') bucket = s3.Bucket('X...
Derek Zhang
0

votes
0

answer
4

Views

Downloading files from AWS S3 Bucket with boto3 results in ClientError: An error occurred (403): Forbidden

I am trying to download files from a s3 bucket by using the Access Key ID and Secret Access Key provided by https://db.humanconnectome.org. However, even though I am able to navigate the database and find the files (as I have configured my credentials via aws cli), attempting to download them result...
Semih Cantürk
0

votes
0

answer
4

Views

Using Python Boto3 AWS Lambda, S3 Key LastModified returns in two types for two different ways

The LastModified of a given key returns in two different formats. If I store in a variable or directly print, it gives me this format: 2019-04-17 11:14:11+00:00 And If I try to store it as a value in a dict, and then print it, it shows in this format: {'redshift_data_source/grid/load.csv': datetim...
Aakash Basu
1

votes
0

answer
394

Views

Upload a file asynchronously to S3 in Flask request

I have a Flask API that uploads files. Currently I am working on scaling the application. I have a scenario where a user uploads a large file - in that case, I save the file locally in the flask request and then kick off a celery task that then uploads it to S3 (and does other things). My problem n...
George B
1

votes
0

answer
319

Views

boto3 attached device name

I am trying to get the device name where an EBS volume is attached. I can pull the attach data, but would just like to isolate the device name. import boto3 from botocore.client import ClientError ec2 = boto3.resource('ec2') filters = [{ 'Name': 'tag:app', 'Values': [ 'splunklogidx' ], 'Name': 'tag...
outOfTune
1

votes
0

answer
221

Views

pd.read_csv killed when Reading large files

I have 30 each has 2 gb of data compressed files in s3 location and I'm trying to decompress the files, convert them to data frames and then want to subset the data based on column names. When I'm running the code it is getting Killed(Please refer the screenshot for error). Please help me out how to...
DPs
1

votes
1

answer
1.3k

Views

Downloading multiple S3 objects in parallel in Python

Is there a way to concurrently download S3 files using boto3 in Python3? I am aware of the aiobotocore library, but I would like to know if there is a way to do it using the standard boto3 library.
Tomer
1

votes
1

answer
195

Views

boto3 support for role_arn

I want to insert data from AWS Lambda to AWS DynamoDB and upload a file from Lambda to AWS S3. Currently, I'm using Access Key and Secret Key to do this from boto3. But, due to security issues, I had to change this approach to role-based approach. (i.e) To insert data in DDB or uploading a file to...
Droid
1

votes
1

answer
242

Views

dynamodb scan not working with IN condition

I'm failing to get the IN condition in my dynamodb scan call to work correctly. Background I'm trying to collect all permissions a user have based on the the roles associated with him, based on this data structure: # Users | user | roles | admin | ['system-admin', 'basic-user'] # Roles | role...
Martin Schulze
1

votes
1

answer
450

Views

AWS boto3 — Difference between `batch_writer` and `batch_write_item`

I'm currently applying boto3 with dynamodb, and I noticed that there are two types of batch write batch_writer is used in tutorial, and it seems like you can just iterate through different JSON objects to do insert (this is just one example, of course) batch_write_items seems to me is a dynamo-speci...
JChao
1

votes
0

answer
211

Views

How to upload private files to s3 using boto3 and serializer

I am using S3Boto3Storage to manage private file uploads to s3. When I upload via model, the upload works as expected. When I upload via serializer, I get this error: botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found Here are what my files look...
user3015876
1

votes
0

answer
1.3k

Views

python code gets error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

I got a error like this botocore.exceptions.NoCredentialsError: Unable to locate credentials when I was running the following code: # setup AWS Connection details awsSession = boto3.Session(profile_name='opsdev') def getInfoFromDynamoDB(service): client = awsSession.client('kms') dynamodb = awsSessi...
Besides Penguin
1

votes
1

answer
557

Views

PySpark and Pandas - Read partitioned csv files from S3 skipping empty ones

Using PySpark, I have some code that runs through a bunch of queries. for index, query in enumerate(query_map): spark_dataframe.filter( query).write.csv('s3://OutputBucket/Csvs/Query_{}'.format(index) I'm new to spark, but I understand that each partition is writing individual csv files to a direc...
jwillis0720
1

votes
2

answer
421

Views

Attaching role while creating boto3 ec2 Instance Error

I just started to read about AWS and am trying to create an ec2 instance and attach a role to it. As far as i have read i need to create an instance profile, attach a role to it and then attach it to EC2 instance but i am unable to do so because of the errors described below. I dont have enough time...
Suman Sourav Singh
1

votes
2

answer
790

Views

How do I correctly use the CloudWatch boto3 API to retrieve data from metrics?

I'm using Python3's boto3 to try to pull data from SQS's metrics using the get_metric_statistics function documented here: https://boto3.readthedocs.io/en/latest/reference/services/cloudwatch.html#CloudWatch.Client.get_metric_statistics This is the code I made to try to pull it: import boto3 import...
Helios
1

votes
1

answer
3.8k

Views

import boto3 error on Mac OS

I'm trying to install boto3 on my mac (high sierra 10.13.3) and tried to follow : https://github.com/boto/boto3. I had already installed python 3 using homebrew before, but when I tried to see pip --version, I get error. So, I did 1) modify .bash_profile to add alias pip=pip3 2) verify $ pip --versi...
Suvro Choudhury
1

votes
0

answer
465

Views

How can I wait for a dynamodb update table or ACTIVE state using boto3

I am changing the table, for example the capacity settings using boto3 then I need to wait for its completation I would prefer a solution using boto3.resource('dynamodb').Table('MyTable') instead of dynamodb client.
1

votes
1

answer
318

Views

Find aws ecs latest task and image used

I trying to find out the latest task and image used for ECS container service. I tried using AWS ECS cli, able to list services/tasks running but not able to find out running latest task ID and image used for that task. Can someone help with AWS CLI or Boto script to find out latest task and image u...
beginnertopython
1

votes
0

answer
1.3k

Views

boto3 s3.Object- - expected string or bytes-like object

I tried searching for a solution to this problem but still couldn't figure out the issue. So, I am trying to check if a key exists in a bucket. My code works perfectly fine if I use the following: s3.Object('bucket_name', 'key_name').load() But when I pass the reference variable rather than the actu...
1

votes
0

answer
257

Views

How to get profile data using AWS Cognito with Boto3 and OAuth?

Okay, so after a user logs in, I want to get their profile data. So the way I tried doing that was via boto3.client('cognito-idp').get_user but it keeps returning: NotAuthorizedException: An error occurred (NotAuthorizedException) when calling the GetUser operation: Access Token does not have requir...
Joseph Manley
1

votes
1

answer
656

Views

How do I use python to download an S3 file from a link with a signature and expiration?

I have a s3 link provided to me by a third-party with the following structure: http://s3.amazonaws.com/bucket_name_possibly/path/to/file_possibly/filename?AWSAccessKeyId=SomeKey&Expires=888888&Signature=SomeCharactersPossiblyHTMLencoded Clicking on the link downloads the file for me. However, in pyt...
user3776598
1

votes
1

answer
933

Views

boto s3 - write a csv file using array

I want to upload a file to s3 without writing the file on my local system, so I am using set_contents_from_string of boto library. I am able to upload this file back to s3. But how to write an array to a csv file using boto library. arrlist = [[2, 'jack'], [3, 'john'], [4, 'robert']] file_name = 'te...
usr_11
1

votes
1

answer
172

Views

Is there a boto3 funciton to convert authorization_code into authorization_token

My project is python and using boto3 lib. I'm using aws cognito Authorization code grant flow with return_type=code instead of return_type=token (implicit flow). Once my user is authorized my redirect url is injected with the queryStringParameter code=4d55a121-8ffc-4058-844b-xxxx. outlined here I n...
knittledan
1

votes
1

answer
140

Views

Updating item in dynamo with multiple expression, can't get delete working with SET

Trying to use multiple expression SET and DELETE. everything works fine for SET, But when i add DELETE, not able to figure out right syntax. status = 'Previously Deployed version' message = 'New version deployment started' NewVersion = 'PipelineTestAPI_1.5.0' json_ = {':val1' : status, ':val2' : me...
Meet101
1

votes
0

answer
53

Views

Get user from identityId

I saved one element in dynamodb using the id with AWS.config.credentials.identityId and restricting with ${cognito-identity.amazonaws.com:sub} This generates a us-east-1:14b37fe3-xxxx-xxxx-xxxx-xxxxxxxxxxxx record in dynamodb for a user with sub in cognito user pool: 20a3902b-xxxx-xxxx-xxxx-xxxxxxxx...
Lucas Paim
1

votes
1

answer
1.1k

Views

Can we fetch AWS Cloudtrail logs using boto3 in python?

Is a direct way to get the logs / events from cloudtrail by using boto3 in python. How can this be done ?
FCoding
1

votes
1

answer
236

Views

a presigned post in boto3 is the same as a browser based post with rest api signature calulucation server side?

for the past 3 days I have been attempting to manually sign restapi post calls from the users browser. I have failed. I continue to get a signature mismatch error: So now I am going to try boto3. Boto3 it seems is used when you upload a file to your server then send it to amazon. I would rather do a...
1

votes
4

answer
637

Views

How to import my AWS credentials for below mentioned python script to work?

import boto3 s3 = boto3.resource('s3') for bucket in s3.buckets.all(): print(bucket.name) but this code does not work, how to import my AWS credentials so that this will return correct output? error : botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: 'https://s3.aws...
Ganesh Shinde
1

votes
0

answer
159

Views

While downloading a folder contents from Amazon S3 using boto3, only some files get downloaded and for others, i get the following exception :

File 'C:\Program Files\Python36\lib\site-packages\boto3\s3\inject.py', line 130, in download_file extra_args=ExtraArgs, callback=Callback) File 'C:\Program Files\Python36\lib\site-packages\boto3\s3\transfer.py', line 307, in download_file future.result() File 'C:\Program Files\Python36\lib\site-pack...
Sruthi
1

votes
1

answer
155

Views

Boto3 - python script to view all directories and files

I tried to follow the Boto3 examples, but can literally only manage to get the very basic listing of all my S3 buckets via the example they give: import boto3 s3 = boto3.resource('s3') for bucket in s3.buckets.all(): print(bucket.name) I cannot find documentation that explains how I would be able to...
Monty
1

votes
1

answer
289

Views

How to authenticate session in application running on Beanstalk with boto3

My app is deployed via Elastic Beanstalk. It needs to access S3. I can do it locally with my own access key, but I don't want to store that anywhere when I deploy. Given that the instance is on Beanstalk, there must be an easier way to auth , perhaps with roles? I have given full S3 permissions to t...
turnip
1

votes
1

answer
183

Views

aioboto3 speedup not as expected

I'm trying the aioboto3 lib, which looks extremely promising for speeding certain tasks up. For example, I need to find tags for all S3 objects within a particular bucket and prefix. But unfortunately the speed increase is not what I had hoped for. With 1000 objects, it's about half the time. With...
dude102438

View additional questions