Questions tagged [google-cloud-datastore]
3142 questions
0
votes
4
answer
428
Views
Explain the functionality of JSON
I think it is better to understand why I have so many problems with JSON that I explain you what my goal is:
I work with Googles App Engine. There I want to store data. The data looks like
user - username
question - question
date1 - date1
date2 - date2
An Android App have the 'simple' function to: S...
1
votes
1
answer
705
Views
How do I find a max value in datastore?
I am using go and datastore and I want to have a query where I get the max value from the datastore. I can't find anything about max values in the docs. How can I accomplish that?
I am using Google App Engine https://cloud.google.com/appengine/docs/go/datastore/
1
votes
2
answer
711
Views
Storing a string from a textbox to the Google App Engine Datastore
Is it possible to create a textbox in HTML, type a string into it, click a 'save' button, store that information onto a GAE datastore model and have the text stay displayed in the textbox and saved in the datastore?
The HTML is on a separate file, just rendered through my main.py file using
class M...
1
votes
0
answer
217
Views
GO Datastore connection reuse
I am new to GO and i need to reuse the datastore client in the entire application instead of creating it again and again for each API call. I have created a db package and in it Init i am storing client in a global db variable to reuse it via a GetDB() function. But i am getting the below error
Per...
1
votes
1
answer
20
Views
Is there any limit in list when applying entity filter with “in” operator
I need to apply 'in' filter. I want to ask is there any limit how much element need to insert in list.
For example
List list; // There is at least 1500 items in this list
ofy().load().type(Entity.class).filter('property in', list);
Can I apply as much element as I want or is there any limit or...
1
votes
1
answer
50
Views
Kindless ancestor query for different entity kinds in golang
According to the documentation, it should be possible to retrieve an ancestor and all of its descendants, regardless of their kind.
In my implementation, I have a different kind of ancestor and descendant. The following codes however always returns the error 'invalid entity type':
q := datastore.New...
1
votes
0
answer
95
Views
Is the schema needed to deserialize protobuf bytestring?
I'm trying to deserialize some Protobuf bytestrings, in Python, outside of GAE(AppEngine) environment. I'm wondering if I need the message schema for this use case.
Approaches I've tried so far includes google.cloud.proto.datastore.v1.entity_pb2.Entity.ParseFromString():
>>> pb = ... ## fetched a...
1
votes
2
answer
86
Views
Google Datastore Transaction API vs. Datastore API when in a transaction
I have been battling Google Cloud Datastore transactions for two days.
I am unsure whether or not I should use datastore.save or transaction.save when within a Google Datastore transaction.
Common code:
const datastore = new Datastore();
const transaction = datastore.transaction();
Scenario 1:
tran...
1
votes
1
answer
208
Views
Memory limit exceeded error when reading a large file with node js and readline
I am getting a memory error when using readline. The section of code is shown below:
var lineReader = require('readline').createInterface({
input: require('fs').createReadStream('/tmp/temp.ttl')
});
let entity;
let tripleKey;
let triple;
console.log('file ready for processing');
lineReader.on('line...
1
votes
0
answer
34
Views
How to generate User specific feeds?
I have 2 tables - Pages and Feeds.
Each feed belong to a specific page.
The user follows the interested page.
My requirement is to provide feeds for the user from the pages he has followed in a random way. That is, if page1 has 5 feeds & page2 has 5 feeds, I would like display these 10 feeds in a...
1
votes
1
answer
80
Views
Google App Engine Datastore - query with StructuredProperty in projection and filter
I have a couple ndb models looks like these:
class Product(ndb.Model):
manufacturer = ndb.StringProperty()
category = ndb.StringProperty()
price = ndb.FloatProperty()
class Customer(ndb.Model):
customerId = ndb.StringProperty()
name = ndb.StringProperty()
products = ndb.StructuredProperty(Produc...
1
votes
1
answer
290
Views
Return entity key from datastore
I would like to know if there's a way to return a key from entity specific using nodejs and datastore.
Example:
I have an entity that has name and age parameters. I would like to retrieve the key by name.
How could I do this?
1
votes
0
answer
352
Views
cannot find module 'google-module'
I'm excited to get started with the Particle Electron logging GPS coordinates and storing them up in the Google Cloud Platform. I have all the integrations connected using this tutorial, https://docs.particle.io/tutorials/integrations/google-cloud-platform/ l and am now getting this error trying t...
1
votes
1
answer
32
Views
Does Objectify have an equivalent of Datastore.add() to insert non-existing entities only?
I would like to insert multiple entities (under the same entity group) into Datastore as a batch and only have the missing ones inserted and the rest unmodified. Datastore.add(Entities...) seems to support it as explained in this client issue and in the docs.
I dont see an alternative on ofy() as t...
1
votes
1
answer
34
Views
Google NDB Model syntax
I have an example line of code for google ndb model (datastore)
V=model.Video.query(model.Video.key==key).get()
Assuming that the 'key' has a value, is the instantiation lines below feasible/valid or syntactically correct ?
+ (Video)key.get()
+ V = model.Video(key)
Not sure if they would work. Thank...
1
votes
0
answer
95
Views
GAE with datanucleus and JDO - fails to test locally - Getting javax.jdo.JDOUserException: Persistent class has no table in the database
I am refactoring a GAE application which I developed few years ago (and works fine) to use Maven and Java 8.
The application uses JDO & datanucleus.
The app is being built with no errors and I can browse to the home page, but I encounter the following error when trying to access the database (for...
1
votes
0
answer
219
Views
Python: google-cloud-datastore in AWS Lambda
Because Google Functions does not support Python, I was hoping to use google-cloud-datastore in AWS Lambda but hit the same error as
AWS Lambda to Firestore error: cannot import name 'cygrpc'
google-cloud-storage works just fine in Lambda so core packaging is not the issue but could pip'ing datasto...
1
votes
0
answer
145
Views
Read flattened entity from cloud datastore in golang
func (db *dataStore) AddAcceptance(ctx context.Context, req *acceptance.PolicyAcceptance) (uint64, error) {
accpKey := datastore.IncompleteKey('Acceptance', nil)
key, err := db.Put(context.Background(), accpKey, req);
if err != nil {
log.Fatalf('Failed to save Acceptance: %v', err)
}
accpKey = key...
1
votes
0
answer
152
Views
no matching index found. recommended index is
I am really confused by how to solve this issue.
I tried all the answers I could find from googling this issue.
I got this error, as many other people got.
no matching index found. recommended index is:\n- kind: Task\n ancestor: yes\n properties:\n - name: appName\n - name: RequestTime\n dir...
1
votes
4
answer
1.4k
Views
How to make http call on DialogFlow v2 using Javascript ajax call
I found this example on the official site of DialogFlow using Node.js and it is working fine, but I dont know how do I integrate this into my web application.
Is it possible that I can integrate this into my other javascript jquery code? and here I need to run node index.js but do I still need to do...
1
votes
1
answer
79
Views
Google Datastore Order with ancestors
Language: Python.
I'm using datastore python lib, everything works fine. When using datastore query to sort the query result, I can add query.order = ['name']; to sort the result. But when query a table with ancestors, like:
ancestor = client.key('user', name, namespace = NAMESPACE)
query = client.q...
1
votes
1
answer
76
Views
How to convert Datastore Key to URL-safe format using PHP client library?
I'm trying to obtain a Google Cloud Datastore URL-safe key using the PHP Google Cloud Client Library.
The java client library provides the functions toUrlSafe and fromUrlSafe to serialize a Key object from/to an URL-safe string. What are the php equivalents?
1
votes
2
answer
93
Views
How to periodically update a moderate amount of data (~2.5m entries) in Google Datastore?
I'm trying to do the following periodically (lets say once a week):
download a couple of public datasets
merge them together, resulting in a dictionary (I'm using Python) of ~2.5m entries
upload/synchronize the result to Cloud Datastore so that I have it as 'reference data' for other things running...
1
votes
2
answer
37
Views
Trouble retreiving an entitiy from Google Cloud Datastore by Key in Google Cloud Functions
I have an entity set up with a name identifier:
Namespace: [default]
Kind: Cocktail
Name: martini
I have a Google Cloud Function:
// Imports the Google Cloud client library
const Datastore = require('@google-cloud/datastore');
const projectId = 'my-project-name';
const datastore = Datastore({
projec...
1
votes
2
answer
54
Views
Optimize Cloud Datastore NDB query to reduce read operation counts
After running my GAE app for 1 week, I got 30 millions count and I have to pay 23.31 usd :(
Cloud Datastore Read Ops Japan 30,245,982.00 Count $23.31
Please help me to optimize queries.
In my app, I have a lot of queries, mainly to check if entity does not exist, then do something
tweet_db_qry = mo...
1
votes
0
answer
192
Views
How to get entity id from entity data with Google Cloud Datastore?
Here is the output I am getting from Google Cloud Datastore query in NodeJS:
{
photo: null,
login: '[email protected]',
[Symbol(KEY)]:
Key {
namespace: undefined,
id: '5637017136267264',
kind: 'Users',
path: [Getter] }
}
I want to get [Symbol(KEY)].Key.id value in NodeJS. I am using gstore-node
Plea...
1
votes
1
answer
50
Views
Querying datastore with NDB
I'm trying to use Python and NDB to access the datastore, which contains one entity:
I've defined by NDB model with the following code:
class Test(ndb.Model):
name = ndb.StringProperty()
val = ndb.IntegerProperty()
Then I run a query for the entity:
query = Test.get_by_id('Testing')
This returns a N...
1
votes
0
answer
417
Views
cloud functions python to access Datastore
I am looking for a tutorial or document on how to access datastore using cloud functions (python).
However, it seems there is only tutorial for nodejs.
https://github.com/GoogleCloudPlatform/nodejs-docs-samples/tree/master/functions/datastore
Can anybody help me out?
Thanks
1
votes
1
answer
48
Views
How to fetch the indexes/Columns data from Google Cloud Platform using GQL(Google Query Language)
Kind name: invoice_header
Name/ID :id=4829628648652800
amount :2900
booking_ids :1,2
created_at :2018-09-04 10:20:30
discount_amount :23
due_amount :9999
indicator...
1
votes
0
answer
70
Views
How to query multiple namespaces using beam ReadFromDatastore
My pipeline produces a Pcollection with unique_namespaces that I want to pass as a query parameter to ReadFromDatastore and then Flatten all the reads.
It looks like it is not possible to pass namespaces iterative list as a side input
coll = (p
| 'Reading from Datastore' >> ReadFromDatastore(projec...
1
votes
1
answer
73
Views
How can I get total records count from Google Datastore using Gstore-Node?
I am using gstore-node package for querying data from Google Datastore.
I did not find anywhere how to get the total record count of the query using gstore-node.
Please help me out if anyone knows.
Thanks in advance.
1
votes
1
answer
91
Views
Datastore client query.fetch() throwing up a error
i am getting this exception when trying to create a new entry into Users tables. Currently Users table is empty.Basically i am querying the datastore kind if the record exist.For some reason it is throwing exception at the below code line
result = list(query.fetch())
Complete Code
class User(object)...
1
votes
1
answer
40
Views
Database dump and restore from AWS postgres-9.6.6 database to Google Cloud SQL (also postgres-9.6)
I've dumped my database as described in Exporting data from an externally-managed database server:
pg_dump -U [USERNAME] --format=plain --no-owner \
--no-acl [DATABASE_NAME] \
| sed -E 's/(DROP|CREATE|COMMENT ON) EXTENSION/-- \1 EXTENSION/g' > [SQL_FILE].sql
The database I'm dumping from is running...
1
votes
1
answer
211
Views
403 Missing or insufficient permissions
I am running datastore emulator. When I run 'dev_appserver.py app.yaml' command, I get this error
403 Missing or insufficient permissions.
This is the warning I get. I know there is a problem with authentication. I have gone through this. But couldn't end up finding a solution.
Some details:
I am us...
1
votes
0
answer
26
Views
CLoud Datastore: Transactions implementation details and monitoring
I've recently started to use transactions. However I'm missing some pieces of documentation.
Up till now I had a regular save and get functions which were calling the datastore. I was measuring these ones for latency.
When working with transactions when do I actually hit the network in the followi...
1
votes
0
answer
31
Views
Bulk get using objectify?
I am using the Objectify Loader ids method to batch load the ids from the google datastore. But from the Google Cloud Trace logs, I can see that this requests are batched in ids of size 10.
Is there any way to increase this size?
However, if I change the consistency level of objectify to Eventual an...
1
votes
0
answer
31
Views
Why is there a 25 entity group limit for transactions in Cloud Datastore?
I understand why there is a write throughput limit of 1 second per entity group (strong consistency requirements for replication, etc). I don't understand why you are limited to 25 different entity groups in a transaction. Presumably this is for some other consistency/performance reason but I am hav...
1
votes
0
answer
27
Views
Google Datastore client query similar to SQL union or In-clause
I'm exploring ways (Since I'm new to google datastore) to achieve bulk request similar to what we do using SQL IN-Clause or Union.
What I currently have (below) is very inefficient way of executing query per item in for loop. Any help is much appreciated.
I need to do this in Flex-Environment, means...
1
votes
1
answer
50
Views
Django app works locally but deployment on GCP say 500 Server Error, “ImportError: Could not import settings”
On GCP dashboard I get this error but on my machine all works fine:
ImportError: Could not import settings 'site.settings' (Is it on sys.path?): No module named settings
at __init__ (/base/alloc/tmpfs/dynamic_runtimes/python27g/3b44e98ed7fbb86b/python27/python27_lib/versions/third_party/django-1.4/d...
1
votes
1
answer
67
Views
GQL Query Limit
I have created a NDB Model
class Account(ndb.Model):
accId = ndb.StringProperty()
firstName = ndb.StringProperty()
lastName = ndb.StringProperty()
So now I need to query on this model and get account Id's of all people having firstName as Mark. So I wrote the query...
acc_obj = Account.gql('WHERE fi...