Questions tagged [google-cloud-storage]

1

votes
1

answer
2.2k

Views

readFileSync from an URL for Twitter media - node.js

I need to use an URL to import a media to Twitter. But I can't figure out how to use an URL without having to download the image, save it temporarily on the server and deleted after... I'm using Google Cloud to store the images that's why I can't access it directly. That's what I'm trying to do : va...
Phil
1

votes
1

answer
37

Views

Duplicate Filename in GCP Storage

I am streaming files to GCP Storage (bucket). Doing so results in a frequent error (roughly 2 million times a day) claiming that my filename policy must generate a unique name. I've tried multiple ways of guaranteeing a unique name such as using currentTimeMillis, currentThread, encrypting the file...
Scicrazed
1

votes
1

answer
38

Views

Where did region europe-west1 go?

I'm unable to create a bucket in region europe-west1. Anyone have the same problem? This is what I see on the dashboard: When the docs region list looks like this:
Carl Engene
1

votes
2

answer
37

Views

Cloud fuction storage trigger on bucket of other project

Not able to create cloud function trigger on bucket which is under other project. Deploying function (may take a while - up to 2 minutes)...failed. ERROR: (gcloud.functions.deploy) OperationError: code=7, message=Insufficient permissions to (re)configure a trigger (permission denied for bucket ing-a...
0

votes
0

answer
3

Views

Anonymous caller does not have storage.objects.create access but my JWT has scope https://www.googleapis.com/auth/devstorage.full_control

I am following the documentation for server to server OAuth2 flow (creating my own JWT as opposed to using the library). I've created a service account with the right permissions to upload to my storage bucket. I successfully get an access_token from https://www.googleapis.com/oauth2/v4/token with...
uniisoverrated
1

votes
2

answer
2.4k

Views

How to read a JSON file data and use it in firebase cloud function

I have a firebase cloud function which will be invoked on HTTP request which is working fine. Now, I want to read data from a JSON file for some business logic. Below are the 2 ways I was trying to read the JSON file: Option #1) Saved the JSON file inside 'public' directory in my nodejs project and...
Naveen
1

votes
1

answer
472

Views

Difference between gcloud auth activate-service-account --key-file and GOOGLE_APPLICATION_CREDENTIALS

I'm creating a shell script to handle automation for some of our workflows, This workflow include accessing Google Buckets via Apache Beam GCP. I'm using a .json file with my service account, in which situations do i need to use: gcloud auth activate-service-account --key-file myfile.json vs export...
spicyramen
1

votes
3

answer
162

Views

Get a massive csv file from GCS to BQ

I have a very large CSV file (let's say 1TB) that I need to get from GCS onto BQ. While BQ does have a CSV-loader, the CSV files that I have are pretty non-standard and don't end up loading properly to BQ without formatting it. Normally I would download the csv file onto a server to 'process it' and...
David542
1

votes
2

answer
47

Views

google ml-engine cloud storage as a file

I am working in Python with Google Cloud ML-Engine. The documentation I have found indicates that data storage should be done with Buckets and Blobs https://cloud.google.com/ml-engine/docs/tensorflow/working-with-cloud-storage However, much of my code, and the libraries it calls works with files....
user1902291
1

votes
3

answer
80

Views

How to read a huge CSV file from Google Cloud Storage line by line using Java?

I'm new to Google Cloud Platform. I'm trying to read a CSV file present in Google Cloud Storage (non-public bucket accessed via Service Account key) line by line which is around 1GB. I couldn't find any option to read the file present in the Google Cloud Storage (GCS) line by line. I only see the...
Tech Guy
1

votes
1

answer
314

Views

Google Cloud Storage Client API fails in Certificate validation if we pass javax ssl params

I have successfully builded and ran a code snippet to upload the document to google cloud storage using the Java API. While integration of Google Sql to our project we have introduced the keystore and truststore params. After the introduction of the store params the upload document API's are failing...
ted
1

votes
0

answer
56

Views

Unable to set notifications for bucket

I am using the golang storage API to add notifications to my bucket. https://godoc.org/cloud.google.com/go/storage#BucketHandle.AddNotification I am getting the following error: Unable to set notifications for bucket: googleapi: Error 403: The service account '[email protected]
Sanket Agarwal
1

votes
2

answer
203

Views

GCP App Engine Access to GCloud Storage without 'sharing publicly'

I would like to know how to grant a Google Cloud Platform App Engine project permissions to serve content from Google Cloud Storage without setting the Google Cloud Storage bucket permissions to ‘share publicly'. My App engine project is running Node JS. Uses Passport-SAML authentication to authen...
Paul Trotter
1

votes
0

answer
240

Views

gsutil parallel composite speed decreases over time

I am trying to upload a 1GB file via gsutil. Here is the command I'm using: $ gsutil -o GSUtil:parallel_composite_upload_threshold=150M cp /Users/david/Desktop/pricing20160112/c_price gs://docs_test Copying file:///Users/david/Desktop/pricing20160112/c_price The first 80% of the upload works just...
David542
1

votes
0

answer
102

Views

Temporary file not found when uploading to a different Google Cloud bucket in a cloud function

In a Firebase cloud function I download an image from one bucket, create a temporary file (to make modifications) and then attempt to upload it to a different bucket. When doing so, the logs show an ApiError: Not found exception when attempting to upload the temporary file. I can upload the temporar...
Jason Parsons
1

votes
1

answer
431

Views

NotFoundError: Expect status [200] from Google Storage. But got status 404 while reading a file

I have created a bucket in google storage and uploaded a sample text file and gave owner permissions. Using the code in google docs, I am trying to read the uploaded text file (test.txt) and create a sample text file. I couldn't do both read/create operations. Connecting to cloud storage: bucket_nam...
Bhavana
1

votes
1

answer
552

Views

streaming write to gcs using apache beam per element

Current beam pipeline is reading files as stream using FileIO.matchAll().continuously(). This returns PCollection . I want to write these files back with the same names to another gcs bucket i.e each PCollection is one file metadata/readableFile which should be written back to another bucket after s...
user179156
1

votes
0

answer
196

Views

Google engine compute image storage vs. snapshot storage - price difference

I am looking at this link and it looks like snapshot storage is only $0.026 GB/month, while image storage is $0.085. I am curious why there is such a significant difference in storage prices? Update: Even more interesting fact: standard provisioned space is only $0.04, cheaper than image storage? Wh...
user1700890
1

votes
1

answer
51

Views

How to integrate Google Cloud Platform into my company's iOS app [closed]

My company currently stores real time manufacturing data (< 10 GigaBytes) locally in Microsoft SQL Server. We would like to push this data to the cloud and serve it to US clients in an iOS app, preferably in real-time. I have experience with Firebase and Cloud Functions, but not Google Cloud. What...
JGuo
1

votes
4

answer
1.4k

Views

AccessDeniedException: 403 when trying to copy file to Google Storage from VM using gsutil

I’m trying to manually do the steps I need to automate the process to understand how it works and to make sure I get all the commands straight. But when I try to do it using the command: gsutil cp file_name gs://bucket_name/ I get the following error: AccessDeniedException: 403 Insufficient OAuth2...
Marcus Vinicius Melo
1

votes
1

answer
51

Views

Cannot Create File in a Google Cloud Bucket

On Google Cloud I have 4 buckets and I have two VMs running, one Ubuntu 14 and the other 16. On both machines, I can mount (gcsfuse) all four buckets. However, I'm having a problem with one (and only one) bucket (project-x-logs). On the machine running Ubuntu 14 I can create a file: [email protected]
Joe
1

votes
1

answer
99

Views

When I destroy a Google Cloud Storage Bucket, how is the data deleted?

Is the data in the bucket securely deleted (ex DoD 5220.22-M) or do I need to do that myself before destroying the bucket?
Noah Horton
1

votes
1

answer
818

Views

Resizing and Serving Images from Google Cloud Storage with Nodejs?

I found this link upload and resize the image with google storage. I look at the google document but seem to be google is not support for nodejs. How can I implement the API on nodejs? or the way to just input google storage link and get output serve_url?
Ihenry
1

votes
1

answer
779

Views

Save python data object to file in google storage from a pyspark job running in dataproc

I'm collecting metrics while running a pyspark job with dataproc and I'm unable to persist them in google storage (using only python functions, not Spark). The point is that I can save them and during the execution, I read and modificate them successfully, but when the job ends, there's nothing in m...
Luis A.G.
1

votes
0

answer
75

Views

Using the gsutil rm command

I finally got around to using the gsutil rm command ... I also noticed that the command shell allowed me to add up to 4 total instances ... I have several buckets that have over 100gig of files that need deletion, so this was a great feature ... However, my concern is with timing out ... it seemed t...
QuickBooksRus
1

votes
2

answer
248

Views

Google Cloud Function - Cloud storage object overwritten event

I have a cloud function which is set to be triggered by a cloud storage bucket. Is there anyway to tell if an event is set off by a newly uploaded object or rather an overwritten object? I tried to console.log out the event object but nothing seems to be indicative of whether or not the object has...
Pepperoni Papaya
1

votes
1

answer
260

Views

NoSuchMethodError while creating Storage object of Google Storage (Java)

I get the following error trying to create a Storage element - java.lang.NoSuchMethodError: com.google.cloud.storage.StorageOptions.newBuilder()Lcom/google/cloud/storage/StorageOptions$Builder; using - AuthCredentials.ServiceAccountAuthCredentials serviceAccountAuthCredentials = AuthCredentials.crea...
Gaurav
1

votes
0

answer
84

Views

Loading Data from Google Cloud Storage to GoogleBigQuery

I am uploading data from json file stored in Google Cloud to BigQUery Table but I am getting below Error Not found: URI gs://encoreteststorage/Myfolder/TestDataJson.json File Location in Cloud-> Buckets/encoreteststorage/Myfolder Is bucket URI Incorrect(Bucket Name,folder and file names are correc...
Mayank
0

votes
1

answer
993

Views

POST request to google cloud storage using Curl

In the following Curl command, content type and content length and access bearer are attached to my bucket URI to upload a file to google cloud storage. C:\softwares\curl>curl -X POST -H 'Content-Type:application/json' \ -H 'Content-Length:100' \ -H 'Authorization: Bearer ' \ 'https://www.googleapi...
sai
1

votes
0

answer
435

Views

Google Cloud Datalab error writing to Cloud Storage

I am using Google Cloud Datalab for the first time to build a classifier for a Kaggle competition. But I am stuck trying to write a csv file containing the pre-processed training data to Cloud Storage using the google.datalab.storage API. The file contains strings with unicode characters which cause...
Kridir
1

votes
0

answer
53

Views

getting Firebase Cloud Storage metadata

After issuing the following Java REST GET request, I recieve a response that does not look like the Resource Representations given on the documentation page at: https://cloud.google.com/storage/docs/json_api/v1/objects#resource Can somebody see what I am doing wrong? HttpGet httpGet = new HttpGet('h...
Martijn Klasing
1

votes
1

answer
119

Views

Storing and “Streaming” from Cloud Storage

I am trying to gasp how to store video files. I know I can store .mp4's on Google Cloud Store. However, I have had a hard time interfacing my application to stream these video files. I have found video URLs like: http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4 Versus what the file on the Cloud Sto...
billybob2
1

votes
2

answer
56

Views

How to create a file with public access

I need to upload a file to Google Cloud Storage and make it public immediately. I use createWriteStream. If I try to write a file and after a finish immediately call makePublic - I get an error about that no such object. For now, I use setTimeout with makePublic callback, but it's a hacky way. the c...
Ruslan Zaytsev
1

votes
1

answer
197

Views

BigQuery load CSV file 'successful' although the table does not update

I am trying to upload a CSV to a preexisting table in Google BigQuery although the table is not updating. It says the job loads 'successfully' and I have no errors. Except when I go into check the table, it doesnt add in the lines of the newly 'added' CSV. I have the setting 'append to table' altho...
fire2018
1

votes
0

answer
67

Views

Google Images API get_serving_url() download

How is it possible to download an image using get_serving_url() with a filename? Example for url and parameter: http://[image-url]=d http://lh3.googleusercontent.com/93uhV8K2yHkRuD63KJxlTi7SxjHS8my2emuHmGLZxEmX99_XAjTN3c_2zmKVb3XQ5d8FEkwtgbGjyYpaDQg=d This downloads the image with unnamed.jpg as fil...
Micha El
1

votes
0

answer
152

Views

ApiError when trying to move Firebase Storage file using Firebase Function

I am trying to move Firebase Storage file using functions. I am on this for hours but whatever I do, I am getting very strange ApiError. Simplest version of the code: const admin = require('firebase-admin') const storageRoot = admin.storage().bucket() exports.moveFile = (oldPath, newPath) => { retur...
Rasto
1

votes
1

answer
366

Views

Laravel laravel-google-cloud-storage NotFoundException

I am using laravel-google-cloud-storage package from github in my Laravel project to upload files to google cloud storage. I've a JSON file with credentials, project id and bucket list in my .env file. .env File GOOGLE_CLOUD_PROJECT_ID='myproject-123456' GOOGLE_CLOUD_KEY_FILE=/var/www/project/credin...
Ariful Haque
1

votes
0

answer
41

Views

What is the format for crc32c hash in Firebase storage rules?

I'm trying to use the request.resource.crc32c from Firebase Storage rules (https://firebase.google.com/docs/reference/security/storage/#properties_2) My goal is to have a rule like: match /blobs/{hash}/{fileName} { allow read; allow write: if request.resource.crc32c == hash; } But I'm getting permis...
Samy Pessé
1

votes
1

answer
339

Views

Google Cloud node install third party libraries Node.js

I am deploying my code on Google Cloud. Everything is working fine when testing, but when I run: gcloud app deploy It shows an error. Basically, I need to install Node.js canvas which requires some dependencies which I already installed on my app, but it still returns the error. Below are the logs:...
iamwebkalakaar
1

votes
1

answer
378

Views

Google Cloud Storage - how to receive specific generation of file in Node.JS

I am using @google-cloud/storage for accessing the GCS API. After enabling versioning, I am trying to fetch a specific generation of a file using the following code: bucket.file(`${name}#${generationId}`).download({ destination }).then(() => { res.set({ 'Content-Type': 'image/jpeg' }); return res.st...
Martin Bories

View additional questions