Questions tagged [boto3]
1064 questions
1
votes
0
answer
394
Views
Upload a file asynchronously to S3 in Flask request
I have a Flask API that uploads files. Currently I am working on scaling the application.
I have a scenario where a user uploads a large file - in that case, I save the file locally in the flask request and then kick off a celery task that then uploads it to S3 (and does other things).
My problem n...
1
votes
0
answer
319
Views
boto3 attached device name
I am trying to get the device name where an EBS volume is attached. I can pull the attach data, but would just like to isolate the device name.
import boto3
from botocore.client import ClientError
ec2 = boto3.resource('ec2')
filters = [{
'Name': 'tag:app',
'Values': [ 'splunklogidx' ],
'Name': 'tag...
1
votes
0
answer
221
Views
pd.read_csv killed when Reading large files
I have 30 each has 2 gb of data compressed files in s3 location and I'm trying to decompress the files, convert them to data frames and then want to subset the data based on column names. When I'm running the code it is getting Killed(Please refer the screenshot for error). Please help me out how to...
1
votes
1
answer
1.3k
Views
Downloading multiple S3 objects in parallel in Python
Is there a way to concurrently download S3 files using boto3 in Python3?
I am aware of the aiobotocore library, but I would like to know if there is a way to do it using the standard boto3 library.
1
votes
1
answer
195
Views
boto3 support for role_arn
I want to insert data from AWS Lambda to AWS DynamoDB and upload a file from Lambda to AWS S3. Currently, I'm using Access Key and Secret Key to do this from boto3.
But, due to security issues, I had to change this approach to role-based approach. (i.e) To insert data in DDB or uploading a file to...
1
votes
1
answer
242
Views
dynamodb scan not working with IN condition
I'm failing to get the IN condition in my dynamodb scan call to work correctly.
Background
I'm trying to collect all permissions a user have based on the the roles associated with him, based on this data structure:
# Users
| user | roles
| admin | ['system-admin', 'basic-user']
# Roles
| role...
1
votes
1
answer
450
Views
AWS boto3 — Difference between `batch_writer` and `batch_write_item`
I'm currently applying boto3 with dynamodb, and I noticed that there are two types of batch write
batch_writer is used in tutorial, and it seems like you can just iterate through different JSON objects to do insert (this is just one example, of course)
batch_write_items seems to me is a dynamo-speci...
1
votes
0
answer
211
Views
How to upload private files to s3 using boto3 and serializer
I am using S3Boto3Storage to manage private file uploads to s3. When I upload via model, the upload works as expected.
When I upload via serializer, I get this error:
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
Here are what my files look...
1
votes
0
answer
1.3k
Views
python code gets error: botocore.exceptions.NoCredentialsError: Unable to locate credentials
I got a error like this botocore.exceptions.NoCredentialsError: Unable to locate credentials when I was running the following code:
# setup AWS Connection details
awsSession = boto3.Session(profile_name='opsdev')
def getInfoFromDynamoDB(service):
client = awsSession.client('kms')
dynamodb = awsSessi...
1
votes
1
answer
557
Views
PySpark and Pandas - Read partitioned csv files from S3 skipping empty ones
Using PySpark, I have some code that runs through a bunch of queries.
for index, query in enumerate(query_map):
spark_dataframe.filter(
query).write.csv('s3://OutputBucket/Csvs/Query_{}'.format(index)
I'm new to spark, but I understand that each partition is writing individual csv files to a direc...
1
votes
2
answer
421
Views
Attaching role while creating boto3 ec2 Instance Error
I just started to read about AWS and am trying to create an ec2 instance and attach a role to it. As far as i have read i need to create an instance profile, attach a role to it and then attach it to EC2 instance but i am unable to do so because of the errors described below.
I dont have enough time...
1
votes
2
answer
790
Views
How do I correctly use the CloudWatch boto3 API to retrieve data from metrics?
I'm using Python3's boto3 to try to pull data from SQS's metrics using the get_metric_statistics function documented here:
https://boto3.readthedocs.io/en/latest/reference/services/cloudwatch.html#CloudWatch.Client.get_metric_statistics
This is the code I made to try to pull it:
import boto3
import...
1
votes
1
answer
3.8k
Views
import boto3 error on Mac OS
I'm trying to install boto3 on my mac (high sierra 10.13.3) and tried to follow : https://github.com/boto/boto3. I had already installed python 3 using homebrew before, but when I tried to see pip --version, I get error.
So, I did
1) modify .bash_profile to add
alias pip=pip3
2) verify
$ pip --versi...
1
votes
0
answer
465
Views
How can I wait for a dynamodb update table or ACTIVE state using boto3
I am changing the table, for example the capacity settings using boto3
then I need to wait for its completation
I would prefer a solution using boto3.resource('dynamodb').Table('MyTable') instead of dynamodb client.
1
votes
1
answer
318
Views
Find aws ecs latest task and image used
I trying to find out the latest task and image used for ECS container service. I tried using AWS ECS cli, able to list services/tasks running but not able to find out running latest task ID and image used for that task. Can someone help with AWS CLI or Boto script to find out latest task and image u...
1
votes
0
answer
1.3k
Views
boto3 s3.Object- - expected string or bytes-like object
I tried searching for a solution to this problem but still couldn't figure out the issue. So, I am trying to check if a key exists in a bucket. My code works perfectly fine if I use the following:
s3.Object('bucket_name', 'key_name').load()
But when I pass the reference variable rather than the actu...
1
votes
0
answer
257
Views
How to get profile data using AWS Cognito with Boto3 and OAuth?
Okay, so after a user logs in, I want to get their profile data. So the way I tried doing that was via boto3.client('cognito-idp').get_user but it keeps returning:
NotAuthorizedException: An error occurred (NotAuthorizedException) when calling the GetUser operation: Access Token does not have requir...
1
votes
1
answer
656
Views
How do I use python to download an S3 file from a link with a signature and expiration?
I have a s3 link provided to me by a third-party with the following structure: http://s3.amazonaws.com/bucket_name_possibly/path/to/file_possibly/filename?AWSAccessKeyId=SomeKey&Expires=888888&Signature=SomeCharactersPossiblyHTMLencoded
Clicking on the link downloads the file for me. However, in pyt...
1
votes
1
answer
933
Views
boto s3 - write a csv file using array
I want to upload a file to s3 without writing the file on my local system, so I am using set_contents_from_string of boto library. I am able to upload this file back to s3.
But how to write an array to a csv file using boto library.
arrlist = [[2, 'jack'], [3, 'john'], [4, 'robert']]
file_name = 'te...
1
votes
1
answer
172
Views
Is there a boto3 funciton to convert authorization_code into authorization_token
My project is python and using boto3 lib.
I'm using aws cognito Authorization code grant flow with return_type=code instead of return_type=token (implicit flow). Once my user is authorized my redirect url is injected with the queryStringParameter code=4d55a121-8ffc-4058-844b-xxxx.
outlined here
I n...
1
votes
1
answer
140
Views
Updating item in dynamo with multiple expression, can't get delete working with SET
Trying to use multiple expression SET and DELETE. everything works fine for SET, But when i add DELETE, not able to figure out right syntax.
status = 'Previously Deployed version'
message = 'New version deployment started'
NewVersion = 'PipelineTestAPI_1.5.0'
json_ = {':val1' : status,
':val2' : me...
1
votes
0
answer
53
Views
Get user from identityId
I saved one element in dynamodb using the id with AWS.config.credentials.identityId and restricting with ${cognito-identity.amazonaws.com:sub}
This generates a us-east-1:14b37fe3-xxxx-xxxx-xxxx-xxxxxxxxxxxx record in dynamodb for a user with sub in cognito user pool: 20a3902b-xxxx-xxxx-xxxx-xxxxxxxx...
1
votes
1
answer
1.1k
Views
Can we fetch AWS Cloudtrail logs using boto3 in python?
Is a direct way to get the logs / events from cloudtrail by using boto3 in python. How can this be done ?
1
votes
1
answer
236
Views
a presigned post in boto3 is the same as a browser based post with rest api signature calulucation server side?
for the past 3 days I have been attempting to manually sign restapi post calls from the users browser. I have failed. I continue to get a signature mismatch error: So now I am going to try boto3. Boto3 it seems is used when you upload a file to your server then send it to amazon. I would rather do a...
1
votes
4
answer
637
Views
How to import my AWS credentials for below mentioned python script to work?
import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print(bucket.name)
but this code does not work, how to import my AWS credentials so that this will return correct output?
error : botocore.exceptions.EndpointConnectionError: Could not connect
to the endpoint URL: 'https://s3.aws...
1
votes
0
answer
159
Views
While downloading a folder contents from Amazon S3 using boto3, only some files get downloaded and for others, i get the following exception :
File 'C:\Program Files\Python36\lib\site-packages\boto3\s3\inject.py', line 130, in download_file
extra_args=ExtraArgs, callback=Callback)
File 'C:\Program Files\Python36\lib\site-packages\boto3\s3\transfer.py', line 307, in download_file
future.result()
File 'C:\Program Files\Python36\lib\site-pack...
1
votes
1
answer
155
Views
Boto3 - python script to view all directories and files
I tried to follow the Boto3 examples, but can literally only manage to get the very basic listing of all my S3 buckets via the example they give:
import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print(bucket.name)
I cannot find documentation that explains how I would be able to...
1
votes
1
answer
289
Views
How to authenticate session in application running on Beanstalk with boto3
My app is deployed via Elastic Beanstalk. It needs to access S3. I can do it locally with my own access key, but I don't want to store that anywhere when I deploy. Given that the instance is on Beanstalk, there must be an easier way to auth , perhaps with roles?
I have given full S3 permissions to t...
1
votes
1
answer
183
Views
aioboto3 speedup not as expected
I'm trying the aioboto3 lib, which looks extremely promising for speeding certain tasks up. For example, I need to find tags for all S3 objects within a particular bucket and prefix. But unfortunately the speed increase is not what I had hoped for.
With 1000 objects, it's about half the time.
With...
1
votes
2
answer
251
Views
Botocore fails to read credentials when run as daemon.service
I had my script running smoothly from command line, however, when I start it as a systemd.service, I get the following error:
iot_local.service - My iot_local Service
Loaded: loaded (/lib/systemd/system/iot_local.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun...
1
votes
1
answer
844
Views
S3 Select with boto3 - internalerror
Has anyone got 'S3 Select' (https://aws.amazon.com/blogs/aws/s3-glacier-select/ ,
https://aws.amazon.com/about-aws/whats-new/2018/04/amazon-s3-select-is-now-generally-available/) with boto3 (or even cli or another sdk) working? I am getting cryptic InternalError below:
Running this on EC2 that has...
1
votes
1
answer
636
Views
AWS Lambda Python/Boto3/psycopg2 Redshift temporary credentials
I'm pretty new to AWS so please let me know if what I'm trying to do is not a good idea, but the basic gist of it is that I have a Redshift cluster that I want to be able to query from Lambda (Python) using a combination of psycopg2 and boto3. I have assigned the Lambda function a role that allows i...
1
votes
3
answer
1.4k
Views
Why does installing boto3 break my installation of awscli?
I am trying to install boto3 so I can use it in my python script that I want to run on an EC2 instance running Ubuntu 14.04. Anyway I install boto3 it breaks my installation of awscli on the non root user. I install awscli using apt get with sudo apt-get install awscli. After I install it, it works...
1
votes
2
answer
519
Views
Saving boto3 output to JSON
Need to save boto3 output (as a backup) to JSON file
#!/bin/python
import boto3
import json
client = boto3.client('iam')
response = client.get_account_authorization_details(
)
sys.stdout = open('output.json','wt')
print response
i got output:
{u'RoleDetailList': [{u'AssumeRolePolicyDocument': {u'Ve...
1
votes
1
answer
105
Views
how to handle internet not avaialble while uploading the file to s3?
I have written scheduler program in python to uploading file to AWS S3 at a particular time(using scheduler).
if internet connection is not available at that time , I'm getting socket.gaierror: [Errno -3] Temporary failure in name resolution exception.
This breaks the total flow, so that i need to r...
1
votes
1
answer
828
Views
Having trouble while fetching data from PYTHON script using AWS Athena-Boto3
I am trying to query the dataset present in s3 bucket, using Athena query via python script with help of boto3 functions.
I am using start_query_execution() to run my query. this is being executed perfectly, next to get results in my python script, so that I get access to the result of the query I...
1
votes
1
answer
245
Views
How can I use python (BOTO3) to cycle through all AWS EC2 instances and remove a security group from them? [duplicate]
This question already has an answer here:
BOTO3 — Attach / Detach Security Group from EC2 instance
1 answer
I am looking for a script that will cycle through all of my AWS EC2 instances, check for a specific security group, and then delete that group from the instance.
I've been reading through t...
1
votes
0
answer
459
Views
BOTO3 DynamoDB error: delete an item from a LIST attribute in a dynamodb table
we have a dynamoDB table with 'item_id' as primary/hash key. One of the attributes 'my_list' is a list of strings. I am trying to remove index '0' from the list. When I hard code the index, it works:
item_key = {'item_id': {'S': item_id}}
update_expression = 'remove mylist[0]'
response = client.upda...
1
votes
1
answer
225
Views
Setting boto3 dynamodb endpoint_url globaly
I want to use dynamodb local for local (unit) testing of my python/boto3 based application.
Now I can do something like this
if test_mode:
client = boto3.client('dynamodb', endpoint_url='localhost:8000')
resource = boto3.resource('dynamodb', endpoint_url='localhost:8000')
else:
client = boto3.client...
1
votes
1
answer
193
Views
Using Boto3 to send S3 Put Requests to SNS
I'm trying to get S3 put requests sent to an SNS topic. My code looks like this:
s3.put_bucket_notification_configuration(
Bucket=event_name,
NotificationConfiguration={
'TopicConfigurations': [
{
'TopicArn': sns_arn,
'Events': [
's3:ObjectCreated:*',
],
},
]
}
)
Unfortunately, I'm getting an except...