Questions tagged [google-cloud-pubsub]

1

votes
1

answer
675

Views

how to add protobuf file for topic in gcloud pubsub?

In google cloud's pubsub, I can see that on creating a new topic, I have to create a new message. Can I store a protobuf file there instead of having to write the whole structure of message in key-value pairs? For the protobuf code that shall be written, I mean this. If the protobuf isnt meant to be...
kneelb4darth
1

votes
1

answer
620

Views

Trigger a PubSub message from Cloud Functions

Firebase docs mention a straightforward way of subscribing to a pubSub topic. Does Cloud Functions or Firebase Admin SDK provide a way to publish pubSub events or we are limited to what is described in Google Cloud Platform docs: https://cloud.google.com/pubsub/docs/publisher#pubsub-publish-message...
Łukasz Patecki
0

votes
0

answer
5

Views

Stop Streaming pipeline when no more messages to consume

I have a streaming dataflow pipeline job which reads messages from a given pub-sub topic. I understand there is an auto-ack once the bundles are committed. How to make the pipeline stop where there are no more messages to consume?
user3483129
1

votes
2

answer
64

Views

Cost of Google Cloud PubSub when there are no messages

I was taking a look Pub/Sub pricing and I couldn't get what would be the price if the subscriber hit the endpoint but doesn't get any message. Because my idea is run a worker 24 hours.
Agustin Castro
1

votes
2

answer
52

Views

Google Healthcare API on GKE with PubSub - INVALID_ARGUMENT

We have been testing out the Google Healthcare API specifically with HL7 and as I've run through the tutorials I've hit a roadblock. I should mention that I have a fair bit of experience with Kubernetes and AWS, but not so much Google Cloud. This step here is what is giving me trouble: https://cloud...
damianesteban
1

votes
2

answer
43

Views

Google PubSub - serviceAccount:[email protected] doesn't exist

I am trying to subscribe to Gmail for mail notifications using Google's Pub/Sub and I've gone through the documentation and tutorials provided by Google. At one point, the docs state that I need to publish the correct rights to my PubSub topic: You need to grant publish privileges to serviceAccount:...
Maxwell
1

votes
0

answer
58

Views

Google PubSub subscription with Python API: how to create a generator out of a subscription that can be processed locally?

I searched for an example in the github repo but to no avail. The application I have in mind is to 'stream' a subscriber -- read, to emulate a Python generator so I can window the stream before processing it. Any hints on how to create a generator out of a pubsub subscriber? Or perhaps suggestions...
user791318
1

votes
0

answer
249

Views

How to determine if a subscription notification come from a test account in Android

When a subscription notification from an inapp product (using real-time PUB/SUB) reach our server, how can we know if that notification came from a test purchase? In the Google documentation said that the PUB/SUB test notifications have an attribute as testNotification but this attribute only shows...
Evelin Ponce
1

votes
1

answer
745

Views

Angular 5 and Message Bus

I have a set of RESTful services that my Angular 5 client uses to perform CRUD and business operations for the application. These are a set of micro services and they use pub/sub message queues to communicate between them, e.g. when a user is created the user server publishes a UserCreated event to...
Neil Stevens
1

votes
2

answer
890

Views

Google Pub/Sub Subscriber not receiving messages after a while

I have a simple python script that uses Google pubsub to detect new files in the google cloud storage. The script simply adds new messages to a queue where another thread processes those messages: subscriber = pubsub.SubscriberClient() subscription_path = subscriber.subscription_path( project, subsc...
Sammael
1

votes
0

answer
274

Views

Google Pub/Sub Python Subscriber gets ALREADY_EXISTS after 10 minutes

I deployed a Python subscriber Sunday morning. The subscriber restarts every day. Starting today (3 days later), it experiences an ALREADY_EXISTS error 10-20 minutes after starting. I've restarted it several times now. Every time it runs fine, retrieves previous messages and processes correctly. The...
Casper Alant
1

votes
1

answer
321

Views

Web API to Google Cloud Pub Sub

I'm new to google cloud technologies. If I have a streaming web API online where I want to access JSON data from and write it into Cloud Pub/Sub, Do I need to create a python app engine or something that reads that Web API and writes the data into Cloud Pub/Sub or is there an easier way? Basically...
Colman
1

votes
1

answer
100

Views

pubsub (gcp iam) using firebase auth

I'm writing an (embedded + mobile) app that uses firestore through REST & GRPC APIs. Each access authenticates using firebase bearer tokens which is identifies a firebase userid. I'd like to use that same token/firebase uid to pull notifications from pubsub, and limit that firebase uid to a particu...
fjc101
1

votes
1

answer
251

Views

Google Pub Sub “Permission Denied” for project owner account

I've been trying to get the Google Pub/Sub Java libraries to work using the Quickstart guides. None of them work as written, at least not for me. I'm working in IntelliJ, Maven framework, OSX, Java 8. Take this example. I followed all the steps: Created a service account as Project Owner, installed...
argybarg
1

votes
1

answer
161

Views

PubSub watermark not advancing

I've written an Apache Beam job using Scio with the purpose of generating session ids for incoming data records and then enriching them in some manner, before outputting them to BigQuery. Here's the code: val measurements = sc.customInput('ReadFromPubsub', PubsubIO .readMessagesWithAttributes() .wit...
simpaj
1

votes
1

answer
138

Views

Google Python cloud-dataflow instances broke without new deployment (failed pubsub import)

I have defined a few different Cloud Dataflow jobs for Python in the Google AppEngine Flex Environment. I have defined my requirements in a requirements.txt file, included my setup.py file, and everything was working just fine. My last deployment was on May 3rd, 2018. Looking through logs, I see tha...
John Allard
1

votes
1

answer
338

Views

Firebase Pub/Sub trigger function is called but event is null

I successfully publish to a topic: gcloud pubsub topics publish my-topic --message '{'value':'data'}' --attribute value=myValue and can succesfuly view the entry using the gcloud command: gcloud beta pubsub subscriptions pull --auto-ack my-subcription I then created a firebase pubsub onPublish trigg...
AfroWizzxx
1

votes
0

answer
89

Views

How to scale thousands of Firebase app instances

I need to initialize potentially thousands of Firebase app instances server-side using firebase-admin with a service account and then listen to specific Realtime DB and Firestore events on these app instances. These instances have nothing to do with one another and the only access I have to them is...
jpunk11
1

votes
1

answer
408

Views

Google Cloud Dataflow - Python Streaming JSON to PubSub - Differences between DirectRunner and DataflowRunner

Trying to do something conceptually simple but banging my head against the wall. I am trying to create a streaming dataflow job in Python which consumes JSON messages from a PubSub topic/subscription, performs some basic manipulation on each message (in this case, converting the temperature from C t...
Gummy
1

votes
1

answer
107

Views

Cloud Pub/Sub and Cloud Dataflow fix worker to topic

I have a Google Cloud Pub/Sub and Cloud Dataflow stream processing architecture, i need guaranteed message ordering. Is it possible to set the subscription message flow so that the dataflow workers are fixed per topic, so the messages are routed through the same dataflow worker and therefore should...
Al Hennessey
1

votes
1

answer
154

Views

Is Google Cloud Function Trigger from PubSub Topic a Subscription

I've got two environments in GCC with similar setups. Cloud functions triggered by PubSub Topics. In both the cloud functions list the PubSub topic as the trigger. In one environment the Topic lists a subscription count of 1 which is listed as a Push to endpoint with a url of https://guid-dot-uid-tp...
Robin Southgate
0

votes
0

answer
7

Views

How to use “Try this API” explorer for Pub/Sub

After entering the relevant values, I am getting 403 error (see screenshot) I then noticed that the API Explorer does not have a way for me to input my API key. So, I copy pasted the provided JS code and tried to run it manually, after having inserted the API key that I created. But still, I was get...
Abhishek Jain
1

votes
0

answer
149

Views

Callback function timeout/disruption in google Pub/Sub asynchronous pull subscriber

I have a subscriber application which pulls from a Google Cloud Pub/Sub asynchronously using the google-cloud-pubsub python library. I am running into intermittent issues where my callback function doesnt finish running/is interrupted. Unfortunately I dont have any errors, I only know this is the ca...
1

votes
2

answer
455

Views

Sending PubSub message manually in Dataflow

How can I send a PubSub message manually (that is to say, without using a PubsubIO) in Dataflow ? Importing (via Maven) google-cloud-dataflow-java-sdk-all 2.5.0 already imports a version of com.google.pubsub.v1 for which I was unable to find an easy way to send messages to a Pubsub topic (this versi...
The hBar Tender
1

votes
1

answer
122

Views

Google pubsub into HTTP triggered cloud function?

Is it possible to trigger an HTTP cloud function in response to a pubsub message? When editing a subscription, google makes it possible to push the message to an HTTPS endpoint, but for abuse reasons one has to be able to prove that you own the domain in order to do this, and of course you can't pro...
FrobberOfBits
1

votes
1

answer
153

Views

can't get google auth to work inside docker to publish to pubsub

I'm trying to get my small go app (pub/sub) to work inside of docker so I an put it in GKE but I can't get the auth to work for some reason. docker run --rm -it gcr.io/snappy-premise-118915/sensorgen:v1 {'pressure':24.10712641247902,'temperature':70.24302653595491,'dewpoint':41.3666446148299,'timeco...
lightweight
1

votes
0

answer
60

Views

Google app engine terrible latency with specific pubsub messages

I'm working on a project in the Google Cloud App Engine (node.js flexible runtime) and while I've had a pretty good experience with it thus far, I've recently ran into a problem where the engine will sometimes not respond a specific pubsub notification. Sometimes, the results will appear after a few...
dropTableUsers
1

votes
0

answer
83

Views

Pubsub latency reaching minutes

I've been working on a project utilizing the Gcloud pubsub platform on the node.js flexible runtime and for some reason have had some pretty crazy latency that has been increasing in severity over time. At first, only messages of a certain kind would sometimes experience heavy latency. However, as I...
dropTableUsers
1

votes
1

answer
82

Views

Is it possible to deploy a background Function “myBgFunctionInProjectB” in “project-b” and triggered by my topic “my-topic-project-a” from “project-a”

It's possible to create a topic 'my-topic-project-a' in project 'project-a' so that it can be publicly visible (this is done by setting the role 'pub/sub subscriber' to 'allUsers' on it). Then from project 'project-b' I can create a subscription to 'my-topic-project-a' and read the events from 'my-...
Jean-Baptiste
1

votes
1

answer
298

Views

Is there a dataflow trigger that fires after the window session is complete?

I use Google Cloud PubSub and Dataflow to handle my data. I would like to detect when my daily process is done, in other word when the window session is complete/gap duration is reached. Is there a single trigger that could be fired for this case? If not, is there a workaround I could use? Pipeline...
Gwendal Le Cren
1

votes
1

answer
109

Views

Google Cloud PubSub messages not processed by callback

I'm trying to use Google PubSub to pass and receive messages between two services. However, some of the messages sent seem to be randomly dropped, and are not processed by the subscriber's callback method. When sending messages, about half the messages are processed by the callback method. For the o...
Capeo
1

votes
0

answer
46

Views

How does PubSubIO determine reading speed when using DataflowRunner

Question about how PubSubIO works in DataflowRunner using AutoscalingAlgorithmType.THROUGHPUT_BASED: How does PubSubIO determine how fast should it pull the subscription(i.e. pull frequncy and number of messages in each pull)? I wonder if that has to do with the time between pulling the message and...
xiu shi
1

votes
2

answer
85

Views

DataFlow SDK 2.x: PubSubIO authorization error

Reading from Pubsub using Dataflow Java SDK 2 I'm trying like below: public class App2 { public static void main(String[] args) { Pipeline pipeline = Pipeline.create(PipelineOptionsFactory.fromArgs(args).create()); pipeline .apply('ReadStrinsFromPubsub', PubsubIO.readStrings().fromTopic('projects/de...
Matthew
1

votes
1

answer
69

Views

PubSub Google cloud

Hi I am getting this error when trying to run the Google Dataflow template example Pub/Sub to BigQuery using Eclipse. Exception in thread 'main' java.lang.Error: Unresolved compilation problem: AutoValue_JavascriptTextTransformer_FailsafeJavascriptUdf cannot be resolved to a type at com.google.clou...
Nayyar
1

votes
0

answer
78

Views

How do I limit the number of threads on Publisher in Google Pub/Sub?

I am currently using Google pub/sub and using the Google pubsub python library. I can't find a way to limit the number of threads spawned on my machine which uses most of my resources. Here's my code: def publish(data): def callback(message_future): if message_future.exception(timeout=3): self.logge...
Dinesh
1

votes
0

answer
80

Views

Can we receive gmail push notifications for multiple users by creating different subscriptions to same topic using cloud pub/sub

I am developing an application where users register their Gmail accounts. I want to receive new mail notifications for all the users who are registered with my application. I have followed below steps - 1. Created a Google console project and a topic for my application. 2. Authorized google service...
jak2018
1

votes
1

answer
37

Views

How to call a global variable from a pubsub callback function in google-cloud-platform vminstance?

I'm running a neural network model that supposed to run in callback function every time asynchronously when a pubsub event is triggered. I loaded the model as global variable, during the initialization of main program, so that it can access in any function and to avoid model loading every time in ca...
PRNV JB
1

votes
1

answer
31

Views

Autoscaling GCE Instance groups based on multiple Cloud pub/sub queues

I found in the docs to autoscale based on number of undelivered messages in a topic. (Autoscaling GCE Instance groups based on Cloud pub/sub queue). I want to know if the following scenario is possible: I have two pub/sub queues and I want to write an autoscaling condition that if any one of the qu...
amulya349
1

votes
0

answer
54

Views

Gmail API with PubSub PUSH notification, time issue

I am developing a ticket system, where customer can ask their queries or questions about services, When customer send mail to SUPPORT email id, then an automatic mail will be sent to customer with an unique #TicketId, For this scenario, I am using Gmail API with PubSub PUSH type notification, When I...
harish sharma
1

votes
1

answer
69

Views

Problem publishing JSON output to PubSub. A 'u' is being added to every field

I have a script that pulls from the city if Chicago and grabs a json file and then published to Pub Sub. Once the data gets into pub sub I have a dataflow template that pulls the data into Google Big Query. The final data move to BQ is failing and when I print the output in the script. I am getting...
john williams

View additional questions