Questions tagged [boto3]
672 questions
1
votes
1
answer
2.1k
Views
DynamoDB Primary Key strategy
I'm dabbling with DynamoDB (using boto3) for the first time, and I'm not sure how to define my Partition Key. I'm used to SQL, where you can use AUTO_INCREMENT to ensure that the Key will always increase.
I haven't seen such an option in DynamoDB - instead, when using put_item, the "primary key attr...
1
votes
1
answer
3.1k
Views
Get the Tag “Name” of an EBS volume with Boto 3 AWS Lambda
I am using the script https://github.com/xombiemp/ec2-take-snapshots-lambda/blob/master/ec2-take-snapshots-lambda.py for AWS lambda.
I would like create snapshot and set tag "Name" with the same tag "Name" of the EBS volume, but in the documentation of boto 3 not find how get tag value of EBS volume...
1
votes
1
answer
1.9k
Views
aws boto3 client Stubber help stubbing unit tests
I'm trying to write some unit tests for aws RDS. Currently, the start stop rds api calls have not yet been implemented in moto. I tried just mocking out boto3 but ran into all sorts of weird issues. I did some googling and found http://botocore.readthedocs.io/en/latest/reference/stubber.html
So I ha...
1
votes
1
answer
233
Views
Incorrect padding on a Celery SQS consumer
I'm trying to set up a Celery consumer for a SQS queue.
I'm using Celery 4.1.0 with Python3 and I sent a message with the library boto3 (1.5.28) this way:
response = queue.send_message(MessageBody='Hello World')
and everything it's OK so far.
Then I tried to setup a Celery consumer this way:
@app.ta...
1
votes
1
answer
464
Views
Boto3 get only S3 buckets of specific region
The following code sadly lists all buckets of all regions and not only from "eu-west-1" as specified. How can I change that?
import boto3
s3 = boto3.client("s3", region_name="eu-west-1")
for bucket in s3.list_buckets()["Buckets"]:
bucket_name = bucket["Name"]
print(bucket["Name"])
1
votes
1
answer
23
Views
Python boto3: receive versionId after uploading a file to S3
I want to upload a file to a version-enabled S3 bucket and need its version number. Ideally, without a separate API call to avoid any possibility of a race condition. I'm using the following code snippet for upload (which is working fine):
s3 = boto3.client("s3")
s3.upload_fileobj(file_handle, bucke...
0
votes
1
answer
16
Views
AWS Python Lambda with multiple functions
I have an AWS Lambda function in python3.7. The way its set up im running the lambda_handler(event, context) function and passing data to a separate function that calls itself multiple times depending on what is passed into it. How do I then return data from the second function?
import json
import b...
2
votes
2
answer
57
Views
When to use dynamodb.client, dynamodb.resource and dynamodb.Table?
I am working on Dyanamodb using Boto3 and Python. One of the issues I am finding is when should we use the dynamodb.client, dynamodb.resource and dynamodb.Table?
1
votes
1
answer
1.1k
Views
AWS boto3 source code
How can we get a look at the AWS boto library code. Not all the code is present in the python package.
For example if I wanted to understand how the wait_until_running() method of Waiter class is implemented i.e. if it is implemented serialized fashion and other features of the code. Although the do...
1
votes
1
answer
3.9k
Views
Upload file to AWS S3 using Boto3 with full access permissions
Hi i need help in setting permissions to an object when uploading to S3. I want to make it public read/write.
I have a function as follows.
def UploadFile(self, destFilePath, data):
self._bucket.put_object(Key=destFilePath, Body=data, ACL='public-read-write')
When it runs, i have a permissions error...
1
votes
1
answer
546
Views
upload file to amazon cloud subfolder using python boto3
I'm writing some python script and I am trying to upload a file to the amazon cloud using boto3. The problem is I want to upload the file to a particular subfolder...in some cases I will need to upload the file to a subfolder of a subfolder.
I'm trying to do this:
s3.meta.client.upload_file( "C:\\...
0
votes
0
answer
3
Views
Search file within folders of s3 bucket
I am able to find a file which is directly placed inside s3 bucket but if file is placed inside folders like A/B/test.csv I am NOT able to find it.
# This gives 0 result
key='A/B/test.csv'
bucket = s3.Bucket(bucket_name)
objs = list(bucket.objects.filter(Prefix=key))
print len(objs)
1
votes
2
answer
5.1k
Views
boto3 file_upload does it check if file exists
I was looking through the boto3 documentation and could not find if it natively supports a check to see if the file already exists in s3 and if not do not try and re-upload.
Here's what I have working:
import boto3
s3_client = boto3.client('s3')
s3_bucket = 'bucketName'
s3_folder = 'folder1234/'
tem...
1
votes
1
answer
740
Views
Python 3 Boto 3, AWS S3: Get object URL
I need to retrieve an public object URL directly after uploading a file, this to be able to store it in a database.
This is my upload code:
s3 = boto3.resource('s3')
s3bucket.upload_file(filepath, objectname, ExtraArgs={'StorageClass': 'STANDARD_IA'})
I am not looking for a presigned URL, just the U...
1
votes
1
answer
206
Views
AWS Cognito - Resetting User Password Documentation Seems to Contradict Itself
The Problem
I have worked with both AdminResetPassword and ForgotPassword and I found a contradiction in the API documentation that is confusing and seems to indicate there is no current solution for full admin initialized reset password flow.
The AdminResetPassword documentation says:
When a devel...
1
votes
1
answer
33
Views
'S3' object has no attribute 'put_object_retention'
I'm trying to implement the object lock function introduced recently by AWS s3. The code works perfectly when i run it locally. But as soon as i deploy it on amazon it breaks and gives me the error. I have tried deploying it using zappa, ECS as well as plain lambda. It appears that amazon python pac...
0
votes
0
answer
10
Views
Python AWS Boto3 DynamoDB Paginator hangs when used
I am trying to get a paginator working on a DynamoDB query running in Python using Boto3. When I remove the pagination, the query runs fine. When I introduce the pagination the query finishes, but the pagination loop hangs.
paginator = client.get_paginator('query')
response_iterator = paginator.pa...
0
votes
0
answer
6
Views
Use S3 metadata or a database to search and filter objects in amazon buckets?
I have a bucket with json files, I want to expose those files along with a basic description of what they contains.
So far I have a list of files that could be like that:
{'key' : 'value', 'description' : 'text'}
and the key object that match each file, it is the filename.
So, for fetching a list o...
0
votes
0
answer
3
Views
Is there an alternative way to know when DynamoDb table descreased?
I have a weird problem with one of my tables in DynamoDb. When I make a petition to describe it I find that it was decreased three times today, while in the AWS console I can see only one scale down, that coincides with the one returned by LastDecreaseDateTime when performing a describe_table(TableN...
1
votes
2
answer
924
Views
Fetching ElastiCache Tags
I am trying to fetch ElastiCache Tags using Boto3 and Python. In boto3, there is a function called- list_tags_for_resource(). But, the problem I am facing is, how to find the resource name. I am using the following code:
from boto3.session import Session
sess = Session(aws_access_key_id=id,aws_secr...
1
votes
2
answer
458
Views
get number of used resources in AWS
AWS sets the limit on number of resources. I need to figure out the number of used resources for my account using boto3 and python in a script
Is there any way to do this ?
I am beginner in both boto3 and python
For example:
EBS limits which has multiple resources and its limits defined under it.
1
votes
1
answer
2.7k
Views
Using AWS ECS with Boto3
I would like to lunch a Task on ECS cluster and wait for the task termination.
import boto3
client = boto3.client('ecs')
response = client.run_task(
cluster='default',
taskDefinition='RGB',
overrides={
'containerOverrides': [
{
'name': 'RGB',
'command': [
'python',
'-u',
'rgb.py'
]
}
]
}...
1
votes
1
answer
1.2k
Views
copy file from gcs to s3 in boto3
I am looking to copy files from gcs to my s3 bucket. In boto2, easy as a button.
conn = connect_gs(user_id, password)
gs_bucket = conn.get_bucket(gs_bucket_name)
for obj in bucket:
s3_key = key.Key(s3_bucket)
s3_key.key = obj
s3_key.set_contents_from_filename(obj)
However in boto3, I am lost trying...
1
votes
3
answer
1.5k
Views
Amazon SES send_email Text body with spacing/newlines
I am facing an issue where all of my text e-mails are scrunched together and do not have new lines persisting through the sending process.
Here is the code:
def send_ses_message(email_to, subject, body):
ses = init_ses_internal()
if ses:
ses.send_email(
Source='[email protected]',
Destination={
'T...
1
votes
2
answer
725
Views
find OS and licensed software for EC2 instances
I am trying to figure out the OS and licensed software of an AWS ec2 instance by using API (specifically python's boto3 but if the solution comes from another SDK, that's fine by me).
I naturally tried using describe instances:
client = session.client('ec2')
ec2.describe_instances()
But the data is...
1
votes
1
answer
4.5k
Views
boto : ResourceNotFoundException Dynamodb
I have a table - user_details that has 500 read/write capacity + 500 on indexes as well with over 5k items, currently. I am querying the table based on username and it seems to be giving me "ResourceNotFoundException" all the time -
ClientError: An error occurred (ResourceNotFoundException) when ca...
1
votes
2
answer
1.9k
Views
Get the names and count of all buckets in AWS S3
I'm new to Boto3 and AWS API, and I want to get the list of buckets' names and the count of the available buckets in S3.
Any help is appreciated.
1
votes
1
answer
6.4k
Views
How to download files from s3 given the file path using boto3 in python
Pretty basic but I am not able to download files given s3 path.
for eg, I have this s3://name1/name2/file_name.txt
import boto3
locations = ['s3://name1/name2/file_name.txt']
s3_client = boto3.client('s3')
bucket = 'name1'
prefix = 'name2'
for file in locations:
s3_client.download_file(bucket, 'file...
0
votes
0
answer
3
Views
How can we list all the parameters in the aws parameter store using Boto3? There is no ssm.list_parameters in boto3 documentation?
enter link description here
get_parameters doesn't list all parameters?
2
votes
0
answer
13
Views
AWS : Running Rekognition on DeepLens device
I am creating a DeepLens project to recognise people, when one of select group of people are scanned by the camera.
The project uses a lambda, which processes the images and triggers the 'rekognition' aws api.
On AWS lambda console ( which has 1.8.9 boto version ), I get following issue when I try t...
2
votes
1
answer
148
Views
Boto3 Client Session Failing in Kivy App
I'm working on a Kivy app which needs to upload a file to S3 using boto3 but fails in the JSON decoder.py when trying to load the endpoint model and establish a client connection. This works fine locally running on 16.04 but fails after compiling with buildozer and running on Lollipop 5.1.1.
Buildo...
2
votes
0
answer
76
Views
Howto put object to s3 with Content-MD5
I have tried to upload an XML File to S3 using boto3. As recommended by Amazon, I would like to send a Base64 Encoded MD5-128 Bit Digest(Content-MD5) of the data.
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/serv...
2
votes
1
answer
38
Views
writer.writerow() doesn't write to the correct column
I have three DynamoDB tables. Two tables have instance IDs that are part of an application and the other is a master table of all instances across all of my accounts and the tag metadata. I have two scans for the two tables to get the instance IDs and then query the master table for the tag metadata...
3
votes
1
answer
803
Views
Using Boto3 and Python How Make a Folder Public
I want to make a folder called img that already exists within my private bucket public. I'm using Boto3. I just want to make this folder public not anything else using a script..
This is how I'm currently connecting to the bucket and how far I have got....
ACCESS_KEY_ID = 'xxxxx'
ACCESS_KEY_SECRET =...
1
votes
3
answer
963
Views
Remote signing of boto request for python client
I want to directly upload/download files to Amazon S3 from python clients, running in some users machines. I have a server, that hosts the access Id and Secret keys, as they cannot be in the users side, that can be used to generate a pre-signed url, and that the clients can connect via API to reques...
-2
votes
0
answer
28
Views
What are the three dots for in an python class object?
For example, I am working with a SDK and I am not understanding why this object has 3 dots.
Example: s3.buckets.list
Shouldn't it just be a attribute and a method in this notation?
1
votes
3
answer
2.8k
Views
To check whether AWS instance is up after reboot using python
Is there a way to check if an AWS instance has finally come up in python either using boto3 or otherwise. The running state doesnt differentiate between rebooting and finally up phase.
1
votes
2
answer
3.4k
Views
How to invoke another lambda async and pass context to it?
I'm trying to get working two basic lambdas using Python2.7 runtime for SQS message processing. One lambda reads from SQS invokes and passes data to another lambda via context. I'm able to invoke the other lambda but the user context is empty in it. This is my code of SQS reader lambda:
import boto3...
1
votes
1
answer
7.1k
Views
How to upgrade botocore1.2.6 to 1.4 or above present with boto3?
I have boto3 installed on my linux machine. When I pass below command, I get botocore version as 1.2.6
>>> import boto3
>>> print boto3.__version__
1.2.6
How to upgrade to botocore present within boto3 to 1.4 or latest version? In case I need to downgrade botocore 1.4 to botocore 1.2.6 , what are th...
1
votes
2
answer
99
Views
How can I extract all instances of a key from a list within a python dictionary?
I have a python dictionary named 'available_volumes'.
This dictionary contains a key 'Volumes', and within this key a list of 'VolumeIds.' How can I print each of the 'VolumeId's within this list?
Here is some simplified sample data for my 'available_volumes' data:
{'Volumes':
[
{'VolumeId': 'vol-...