Questions tagged [amazon-s3]

11799 questions
1

votes
1

answer
1.3k

Views

aws: error: argument command: Invalid choice, valid choices are: this is the error showing with some number of commands

aws rekognition index-faces \ --image '{'S3Object':{'Bucket':'mybucketname','Name':'S3ObjectKey'}}' \ --collection-id 'image1' \ --region us-east-1 \ --profile adminuser That is my code which I'm running on terminal, I tried but don't know why is that error coming. What changes should I do??
prithvi Raj
0

votes
1

answer
39

Views

db2RemStgManager analyse error based on error code

I am using a shell script to put a file in object storage. For this I am using; db2RemStgManager S3 put For now I am deliberately providing wrong creds. I want to log a different error message if authentication failed. Currenlty when I log what the above command returns I see: Error code: 251 Error...
aaj
1

votes
1

answer
524

Views

AWS CLI command for querying S3 bucket

I have a bucket which I will call some-images for showing here. I am trying to get my command to return all objects that have a key value which starts_with a number. Here is the code I am using: cmd = 'aws s3api list-objects --bucket some-images --query \'Contents[?starts_with(Key, '10000'\'' push =...
5pence
1

votes
1

answer
1.5k

Views

how to check awscli and compatible botocore package is installed

I am trying to learn python for aws using boto3 so I was trying to execute code given here https://boto3.readthedocs.io/en/latest/guide/s3-example-creating-buckets.html I got errors module boto3 not found so I upgraded to boto3 based on an answer here Unable to install boto3 mentioned to use pip3 in...
ss321c
1

votes
2

answer
3.3k

Views

Terraform aws: Error No configuration files found

I am new to Terraform. I am writing a small script to put a small datafile from my machine to aws S3 bucket.. but I am getting below error. Terraform file code:- provider 'aws' { region = 'us-east-1' version = '~> 1.6' } terraform { backend 's3' { bucket = '${var.bucket_testing}' kms_key_id =...
Arpan
1

votes
1

answer
925

Views

Terraform AWS S3 to Lambda Notification Trigger

I am trying to codify lambda function to be triggered using S3. Creation of Lambda function is successful, but the 'aws_s3_bucket_notification' resource fails at terraform:apply with MethodNotAllowed error with status code 405 : Error: Error applying plan: 20-Sep-2018 15:23:53 1 error(s) occurred...
Samarth
1

votes
3

answer
59

Views

What is the fastest way to save a large pandas DataFrame to S3?

I am trying to figure out what is the fastest way to write a LARGE pandas DataFrame to S3 filesystem. I am currently trying two ways: 1) Through gzip compression (BytesIO) and boto3 gz_buffer = BytesIO() with gzip.GzipFile(mode='w', fileobj=gz_buffer) as gz_file: df.to_csv(TextIOWrapper(gz_file, 'ut...
pmanresa93
0

votes
1

answer
14

Views

AWS S3 - Allow public to view HTML files

I am looking to allow public users to view HTML files located on an AWS S3 bucket on their browser. These HTML files are created and uploaded to my S3 bucket via node.js, and a URL linking to the file is generated. I am using this method to upload the HTML files: s3.upload({ Bucket: bucket, Key: 'H...
Brian
0

votes
0

answer
3

Views

GET files from aws s3 bucket in a react page

How can i get all files hosted in AWS S3 bucket ? I would like you to create a single page application that will load a list of files stored in Amazon S3.I am looking for a single front end page, one RESTful endpoint, and one AWS SDK call on the backend.The frontend should be done in React/Redux - s...
sruthi varghese
0

votes
0

answer
5

Views

Is there any alternate way to access objects from Amazon S3?

I have a file in Amazon S3. I have set it to public access. Is there a way I can access those files by encrypting the URL of S3 objects or any other means?
KillMe
1

votes
1

answer
86

Views

Upload Image into S3 bucket using Api Gateway, Lambda funnction

I'm trying to upload the image (base64) from the postman, I can see when I hit the Serverless API, something has been added in S3 bucket but not image, I'm using nodejs Lambda function, I tried so many solutions but that didn't work out. Please suggest me where I'm wrong: const AWS = require('aws-sd...
Nasreen Ustad
1

votes
0

answer
13

Views

AWS: Access Denied when trying to load presigned url (direct fileupload browser)

I'm attempting to use a presigned URL but I keep getting a 403 Forbidden Access Denied despite setting up everything as I believe that I'm supposed to. I want to upload a file directly from the browser to Amazon S3. I'm first of all enabling the root AWS account to use putObject. I don't have any a...
jones
0

votes
0

answer
4

Views

Which functionality is more efficient/faster in AWS SDK doesObjectExist() or copyObject()?

I have been using both these methods in a utility to copy around 5000+ documents/s3Objects from one bucket to another bucket. While doing so I am checking before copying the s3Object (using copyObject() method) whether that object already exists in the destination bucket or not (using doesObjectExis...
Maddys
1

votes
1

answer
988

Views

S3: ExpiredToken error for S3 pre-signed url within expiry period

This is how I am generating pre-signed url for an S3 object from my python script. s3client = boto3.client('s3') url = s3client.generate_presigned_url('get_object', Params={'Bucket': args.bucket, 'Key': dated_filename}, ExpiresIn=86400) where I am giving an expiry of 24 hours. When I try to downloa...
pratpor
1

votes
2

answer
508

Views

Which AWS services for mobile app backend?

I'm trying to figure out what AWS services I need for the mobile application I'm working on with my startup. The application we're working on should go into the app-/play-store later this year, so we need a 'best-practice' solution for our case. It must be high scaleable so if there are thousands of...
Kai Kuhlmann
1

votes
1

answer
975

Views

Importing data from AWS Athena to RDS instance

Currently I’m listening events from AWS Kinesis and writing them to S3. Then I query them using AWS Glue and Athena. Is there a way to import that data, possibly with some transformation, to an RDS instance?
Bedi Egilmez
1

votes
2

answer
534

Views

how to create multiple folders inside an existing AWS bucket

How to create a multiple folders inside an existing bucket using terraform. example: bucket/folder1/folder2 resource 'aws_s3_bucket_object' 'folder1' { bucket = '${aws_s3_bucket.b.id}' acl = 'private' key = 'Folder1/' source = '/dev/null' }
user_01_02
1

votes
2

answer
344

Views

AWS S3 Bucket Policy Whitelist

I have a bucket policy that whitelists my IP ranges in AWS. I have an EC2 server running a Packer build job, which tries to pull an object from my bucket and I am getting a 403 Forbidden error, even though the IP of my EC2 server running the said job is clearly within the whitelisted range. Even whe...
dmn0972
1

votes
1

answer
330

Views

Pipeline from AWS RDS to S3 using Glue

I was trying AWS Glue to migrate our current data pipeline from python scripts to AWS Glue . I was able to setup a crawler to pull the schema for the different postgres databases . However, I am facing issues in pulling data from Postgres RDS to S3 tables in Athena . Is there a way to directly pull...
Eshank Jain
1

votes
1

answer
163

Views

aws s3 CLI commands are exiting batch file early

This is my batch file: aws s3 sync ./dummy_dist s3://www.mywebsite.com --delete --profile www.mywebsite.com_deploy --debug echo ________________ here 1 __________________ The echo line never runs! This is the debug output from the aws command: 2018-12-23 10:51:52,518 - MainThread - awscli.clidriver...
Force Hero
1

votes
1

answer
99

Views

Create and upload a file in S3 using Lambda function

I am writing a Lambda function to create a text file and upload to a specific S3 directory. import boto3 import os def lambda_handler(event, context): src_bucket = event['Records'][0]['s3']['bucket']['name'] filepath = event['Records'][0]['s3']['object']['key'] head, tail = os.path.split(filepath) n...
Raj
1

votes
1

answer
28

Views

AWS S3 Java SDK not copying file to folder

Java 8 here. I have an S3 bucket (myapp-bucket) with the following folder substructure: /artwork /upload /staging Here's what the myapp-bucket/artwork folder looks like: I have software that uploads image files to the /upload folder, and then another app that I'm writing is supposed to process that...
hotmeatballsoup
1

votes
2

answer
78

Views

AWS S3 authenticated user access using presigned URLs?

I would like to host files on a private AWS S3 bucket which can only be accessed by users who are authenticated to my web application. The links to these file downloads must be static. Simple proxy method: I know this could be done using a proxy service. In this case the static links would point to...
Zugdud
1

votes
1

answer
78

Views

Copy the contents of S3 bucket to another bucket | Node JS

I want to copy all the contents of an S3 bucket to another S3 bucket (in the same account) using AWS Lambda (NodeJs). Right now, what I do is to list the contents of the source bucket and then copy them in the destination bucket. Is there any better approach to copy all the contents? In short, I wan...
Fayza Nawaz
1

votes
2

answer
43

Views

Amazon SQS for browser polling?

I would like to know if AWS SQS is the right service for doing browser polling. For example : 1) User acesses application through browser, and requests a large PDF to be generated 2) API responds back with 'OK' to user and forwards the request to SQS 3) SQS queue is being read by a lambda which gene...
Mojimi
1

votes
3

answer
82

Views

How to get top-level folders in an S3 bucket using boto3?

I have an S3 bucket with a few top level folders, and hundreds of files in each of these folders. How do I get the names of these top level folders? I have tried the following: s3 = boto3.resource('s3', region_name='us-west-2', endpoint_url='https://s3.us-west-2.amazonaws.com') bucket = s3.Bucket('X...
Derek Zhang
0

votes
0

answer
4

Views

Downloading files from AWS S3 Bucket with boto3 results in ClientError: An error occurred (403): Forbidden

I am trying to download files from a s3 bucket by using the Access Key ID and Secret Access Key provided by https://db.humanconnectome.org. However, even though I am able to navigate the database and find the files (as I have configured my credentials via aws cli), attempting to download them result...
Semih Cantürk
0

votes
0

answer
4

Views

Using Python Boto3 AWS Lambda, S3 Key LastModified returns in two types for two different ways

The LastModified of a given key returns in two different formats. If I store in a variable or directly print, it gives me this format: 2019-04-17 11:14:11+00:00 And If I try to store it as a value in a dict, and then print it, it shows in this format: {'redshift_data_source/grid/load.csv': datetim...
Aakash Basu
2

votes
1

answer
90

Views

AWS S3 and Cloudfront CDN for video files

I'm just getting started using AWS S3 and Cloudfront service. So far I've created a S3 instance and a Node.js server which streams the files from S3 to the user reaching the server's endpoint. const params = { Bucket: BUCKET_NAME, Key: KEY }; const obj = s3.getObject(params); const stream = obj.c...
jones
1

votes
1

answer
356

Views

How to get url Cloudfront CDN in Laravel

I am using FileStorage from Laravel to store uploaded files. To get the url of a file I am using: $url = Storage::url('/test/image.jpg'); It does return the url of the file on Amazon S3, however I want it to return the url of the file on CDN CloudFront. How can I achieve this? Thanks all!
Quy Nguyen Vu
1

votes
0

answer
792

Views

Extract email attachment from SES into S3

I want to extract an email attachment and save that attachment into an S3 bucket. I have configured SES to intercept my incoming emails. I am guessing I need to create a SES rule which would have an action to trigger a Lambda function that extracts the email attachment and stores it to S3. I am not...
user1534235
1

votes
0

answer
109

Views

Wagtail can't access images on S3 - I/O operation on closed file

I'm trying to get Wagtail to store and serve images from an AWS S3 bucket. I have followed the instructions and have it working great in a test environment, however when I switch to the Production Database and Bucket, I run into problems using the Admin and occasionally the frontend too. I have: bot...
Brett
1

votes
0

answer
320

Views

Flink 1.4.0 ClassDefNotFoundError … S3ErrorResponseHandler

Working on setting up a local test of Flink 1.4.0 that writes to s3 and I'm getting the following error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.internal.S3ErrorResponseHandler at org.apache.flink.fs.s3presto.shaded.com...
moku
1

votes
0

answer
394

Views

Upload a file asynchronously to S3 in Flask request

I have a Flask API that uploads files. Currently I am working on scaling the application. I have a scenario where a user uploads a large file - in that case, I save the file locally in the flask request and then kick off a celery task that then uploads it to S3 (and does other things). My problem n...
George B
1

votes
1

answer
93

Views

How to get access to AWS S3 bucket

I am trying to run a script from OpenTraffic repository, and it needs access to some AWS S3 buckets. I am unable to figure out how to get access to a particular AWS S3 bucket? FYI: OpenTraffic is a open source platform to obtain and analyse dynamic traffic data : https://github.com/opentraffic The s...
Ravi Kiran
1

votes
1

answer
226

Views

Uploading to S3 a video captured using Cordova Media Capture plugin using Javascript AWS SDK

I am facing a peculiar issue while trying to upload a video file (.mp4 and .mov) to S3, captured using [email protected] or picked from gallery using [email protected] I am using javascript AWS SDK v2.3.11 and calling the .upload function of the SDK. It only copies 15 Byt...
GeeDee
1

votes
0

answer
174

Views

How to access static HTML website from s3 Private bucket in PHP

I have created a new private bucket in AWS S3. In this bucket i have 4 folders like A,B,C and D. In these folders i have uploaded index.html with JS and Images which act as static HTML website. Now i have a website for example (www.test.com) in this website Home Page A,B,C and D's access link are gi...
Daga
1

votes
0

answer
98

Views

Can't pass out 3.5k req/s on AWS Varnish

I'm currently trying to deploy a Varnish on AWS. This varnish is intended to cache 1 file from S3 (only one file, less than 10kb), with a hit rate of 10k+ requests/s I've deployed my server on a m4.large instance (2vCPU / 8Gb Ram). I've launch a performance test from a Jmeter cluster but i can't pas...
Teddy Ferdinand
1

votes
0

answer
198

Views

InvalidToken Error when using Rest API for Downloading file in an S3 AWS bucket in Android

I am trying to use the Rest Api with temporary credentials to download images stored in an S3 bucket in AWS from my Android App. I need to use the rest API instead of the TransferUtility method included in the Android AWS SDK, because I want to use an image downloader for better performance. Accordi...
xyz
1

votes
1

answer
1.3k

Views

Downloading multiple S3 objects in parallel in Python

Is there a way to concurrently download S3 files using boto3 in Python3? I am aware of the aiobotocore library, but I would like to know if there is a way to do it using the standard boto3 library.
Tomer

View additional questions