Questions tagged [google-cloud-pubsub]

0

votes
0

answer
4

Views

Kafka to google Pub/Sub using Sink Connector

Sorry for the basic question. Requirement: Need to send json payload from spring boot application to google pub sub. I have started implementing kafka producer and publishing message to topics and from there to kafka consumer is receiving the json. Now i need to send the json payload from kafka cons...
krthk
0

votes
0

answer
3

Views

Unable to publish Pubsub message in Airflow Python 3

I am unable to publish using the PubSubHook in Airflow with Python 3. Everything works perfectly with Python 2, but with Python 3 I get this error {models.py:1760} ERROR - Object of type 'bytes' is not JSON serializable. It seems that encoding the message in Python 3 results in a byte that then can'...
W Khattak
0

votes
0

answer
3

Views

Dataflow TextIO.write issues with scaling

I created a simple dataflow pipeline that reads byte arrays from pubsub, windows them, and writes to a text file in GCS. I found that with lower traffic topics this worked perfectly, however I ran it on a topic that does about 2.4GB per minute and some problems started to arise. When kicking off the...
Tuubeee
1

votes
1

answer
318

Views

Efficient Google PubSub Publishing

The docs for PubSub state that the max payload after decoding is 10MB. My question is whether or not it is advantageous to compress the payload at the publisher before publishing to increase data throughput? This especially can be helpful if the payload has a high compression ratio like a json forma...
Paul Mazzuca
1

votes
1

answer
869

Views

Accessing Google Cloud Pubsub over HTTP 2 using grpc and protobuf (generated client)

I am attempting to use the google cloud pubsub service by using the service definitions at https://github.com/google/googleapis/blob/master/google/pubsub/v1/pubsub.proto I am able to generate the client code and have the following code to get a topic: ManagedChannelImpl channelImpl = NettyChannelBui...
Ankur Chauhan
1

votes
1

answer
71

Views

How to enable scopes at gmail console

I am working on sending notification from gmail to webserver through pub/sub, I have gone through documentation in that they have mentioned to enable the scopes https://cloud.google.com/pubsub/configure I am specifying the following scopes through code https://www.googleapis.com/auth/cloud-platform...
testing account
1

votes
3

answer
579

Views

Autoscaling GCE Instance groups based on Cloud pub/sub queue

Can GCE Instance groups be scaled up/down bases on Google Cloud PubSub queue counts or other asynchronous task queues such as PSQ?
TimK
1

votes
1

answer
119

Views

Transform data to pubsub events

I have a dataflow pipeline that collects user data like navigation, purchases, crud actions etc. I have this requirement to be able to identify patterns real time and then dispatch pubsub events that other services can listen to in order to provide the user real time tips, offers or promotions. I'm...
chchrist
1

votes
1

answer
723

Views

Request had invalid authentication credentials. on the compute engine vm

I have a default n1 instance on the GCloud with preinstalled java and latest google sdk. When I run code locally within appengine:devserver which connects to the real pubsub subscription, it works correct. If I do the same on the Instance I get an authentication error: [INFO] SEVERE: 401 OK [INFO] {...
Thomas Anderson
1

votes
1

answer
80

Views

Google Dataflow with PubSubIO source constantly available?

Assume we have some data coming in through a Google PubSub topic and its traffic pattern is spiky in nature, with potentially long quiet time before a burst of data coming in at fast rate for minutes. For processing that data, if we are going to use streaming mode Dataflow with subscription based P...
M Song
1

votes
1

answer
1.1k

Views

Error in pubsub subscriber: Max message size exceeded

I am using Google Cloud Pubsub for my application. The subscriber to the pubsub topic is written in Javascript and runs on Nodejs. I am using the official pubsub client provided by google. The code looks like this: var topic = gcloud.pubsub({projectId: 'myProjectId'}).topic('topicName'); var pubsub...
Max
1

votes
1

answer
497

Views

Gmail API: Watch Inbox label Only

I have the code below to watch a mailbox. As seen, in the lableIds, I only have one label, INBOX because I want to listen to only new messages. However, when I run this, it receives notifications every 30s or so with different messageid. Yet no change occurred in the INBOX, no new item was added/rem...
jpo
1

votes
1

answer
429

Views

Can we push/pull message from angular4 using google cloud pubsub?

In Angular4 can we push/pull message directly to google cloud pubsub without calling any service/code in backend ? If it is possible how can we do? Thanks in advance.
ust3000
1

votes
2

answer
380

Views

Google Cloud dataflow : Read from a file with dynamic filename

I am trying to build a pipeline on Google Cloud Dataflow that would do the following: Listen to events on Pubsub subscription Extract the filename from event text Read the file (from Google Cloud Storage bucket) Store the records in BigQuery Following is the code: Pipeline pipeline = //create pipeli...
Darshan Mehta
1

votes
1

answer
507

Views

Stackdriver Metrics through Pub/Sub?

I was curious if Stackdriver metrics are only available via the API or is there a way to send them through Pub/Sub? I'm currently not seeing any of the metrics listed here for Compute Engine in my Pub/Sub output. I did create a sink for all gce vm instances to export from Stackdriver logging in Pub/...
cphelps
1

votes
1

answer
388

Views

Logstash with Google PubSub Input Plugin “Error: certificate verify failed”

I'm struggling to make the Google PubSub Plugin work, I followed all the setup steps (creating a topic, creating a dedicated service account etc. pp.) My configuration looks like this: input { google_pubsub { project_id => 'secret' topic => 'fw-services-logs' subscription => 'fw-logstash-sub' json_k...
Livioso
1

votes
1

answer
274

Views

Apache beam : Read from multiple subscriptions

I want to create a dataflow that listens to multiple subscriptions and writes to BigQuery. As per Google's documentation, I can read multiple PCollection objects and combine them together. However, looking at PubsubIO.Read's javadoc here, it seems subscription method accepts only one String. So, do...
Darshan Mehta
1

votes
1

answer
134

Views

How to use existing PubSub Subscription with Google-Provided PubSub to BigQuery Dataflow Template

I am trying to setup a Dataflow job using the google provided template PubSub to BigQuery. I see an option to specify the Cloud Pub/Sub input topic but I don't see any option to specify Pub/Sub input subscription in GCP console UI. If I provide the topic, job would automatically create a subscript...
Code Geass
1

votes
2

answer
830

Views

Reading from Pubsub using Dataflow Java SDK 2

A lot of the documentation for the Google Cloud Platform for Java SDK 2.x tell you to reference Beam documentation. When reading from PubSub using Dataflow, should I still be doing PubsubIO.Read.named('name').topic(''); Or should I be doing something else? Also building off of that, is there a way...
Panda
1

votes
1

answer
217

Views

Hangouts Chat Pub/Sub Bot only receiving Mentions in Room

We are building our first bot for Hangouts Chat to support JIRA Server (https://github.com/seibert-media/hangouts-jira-bot) and I had to notice that Events in Google PubSub only get sent when the bot receives a direct message or get's mentioned in a room, but not for every room message. Is this inte...
kwiesmueller
1

votes
1

answer
217

Views

Delete data from BigQuery while streaming from Dataflow

Is it possible to delete data from a BigQuery table while loading data into it from an Apache Beam pipeline. Our use case is such that we need to delete 3 days prior data from the table on the basis of a timestamp field (time when Dataflow pulls message from Pubsub topic). Is it recommended to do so...
ShubhamR
1

votes
2

answer
253

Views

gcloud pubsub subscriptions pull often report empty message list

Why does 'gcloud pubsub subscriptions pull' often report an empty message list while there are messages to be acknowleged in the subscription? See below the effect. The message 118870127432164 is available in the subscription, it wasn't ever acknowleged, yet it is often not reported. [email protected]:~$ whi...
Everton
1

votes
2

answer
252

Views

Run a Google Cloud Function for each file in a bucket

I have a Google Cloud Function triggered by a Google Cloud Storage object.finalize event. When I deploy a new version of this function, I would like to run it for every existing file in the bucket (which have already been processed by the previous version of the function). Processing all the existin...
1

votes
1

answer
120

Views

Loadbalancing subscribers - How is it done?

GCP pubsub docs mention load balancing for pull mode, it's not clear how to use it. The Subsciption nor The Subscriber builder api, doesn't seem to have a method to turn this on. Question: How to configure load balancing accross multiple pubsub subscribers? Background: We use multiple subscriber...
1

votes
1

answer
113

Views

Inserting rows on BigQuery: InsertAllRequest Vs BigQueryIO.writeTableRows()

When I'm inserting rows on BigQuery using writeTableRows, performance is REALLY bad compared to InsertAllRequest. Clearly, something is not setup correctly. Need help. Use case 1: Wrote a Java program to process 'sample' Twitter stream using Twitter4j. When a tweet comes in I write it to BigQuery us...
DilTeam
1

votes
1

answer
66

Views

Messages stuck in GBP when streaming from multiple PubSub topics to BigQuery using DataFlow?

I have a Java DataFlow pipeline with the following parts: PubSub subscriber reading from several topics Flatten.pCollections operation Transform from PubsubMessage to TableRow BigQuery writer to write all to a dynamic table When there's more than one PubSub-topic in the list of subscriptions to conn...
Egbert
1

votes
1

answer
46

Views

What happen to PubSub messages when not consumed?

According to the Cloud PubSub documentation messages cease to be stored if not consumed within 7 days. Retains unacknowledged messages in persistent storage for 7 days from the moment of publication. What happens to PubSub messages when the retention delay is over ? Are they simply deleted ? Is ther...
vdolez
1

votes
1

answer
86

Views

Google cloud deployment manager couldn't create deployment using “logging.v2.sink” by Permission Denied

I deployed logging.v2.sink using Google Cloud Deployment Manager, however the deployment was failed by permission denied. The problem was also happened in this logsink example of Google Cloud Deployment Manager. The result is following: - code: RESOURCE_ERROR location: /deployments/my-project-id/res...
inoutch
1

votes
1

answer
31

Views

priotizing a message on Google Pubsub

I have a pubsub topic with a number of pull subscriptions. I would like some mechanism where I can publish a message with a 'priority' label that causes the message to jump as near to the front of the queue as possible. I don't need any guaranteed ordering semantics, just a 'best effort' prioritiza...
Alex Flint
1

votes
1

answer
91

Views

Can we receive gmail push notifications for multiple users to same push end point using cloud pub/sub

I have implemented code in java for receiving gmail push notifications. It is working perfectly fine for the mail id, in which I have created the project, topic and subscription. The notifications are published to my mentioned URL and I can fetch the data of the notifications. But I want to receive...
Imtiaz Hussain
1

votes
1

answer
263

Views

How can I use IoT telemetry events on Pub/Sub topic?

I had read documentation about Google cloud IoT API. And I had wrote simple application for Android of things. Based on this google's library. My application is successfully connected to IoT platform and I had sent test data. My application code. ConnectionParams connectionParams = new ConnectionPar...
Dev.Hrant
1

votes
1

answer
33

Views

Which casing of property names is considered the “most correct” in a Google Cloud Pub/Sub Push Message?

If you use a 'Push' subscription to a Google Cloud Pub/Sub, you'll be registering an HTTPS endpoint that receives messages from Google's managed service. This is great if you wish to avoid dependencies on Google Cloud's SDKs and instead trigger your asynchronous services via a traditional web reques...
Technetium
1

votes
2

answer
730

Views

Replay events with Google Pub/Sub

I'm looking into Google Cloud, it is very appealing, specially for data intensive applications. I'm looking into Pub/Sub + Dataflow and I'm trying to figure out the best way to replay events that were send via Pub/Sub in case the processing logic changes. As far as I can tell, Pub/Sub retention has...
luisobo
1

votes
1

answer
482

Views

Job Queue using Google PubSub

I want to have a simple task queue. There will be multiple consumers running on different machines, but I only want each task to be consumed once. If I have multiple subscribers taking messages from a topic using the same subscription ID is there a chance that the message will be read twice? I've t...
David Tong
1

votes
1

answer
838

Views

GCP PubSub: Publish message via CURL type of request

Does anyone have a working example of how to publish a message to a GCP PubSub topic via CURL type of commands, directly from shell? I'm trying to not use the CLI and not use the client libraries, and I'm getting hung up on the OAUTH stuff. I'd be great to have a bullet list of the things a bash scr...
Rich Murnane
1

votes
1

answer
255

Views

Google Pubsub Python Client library subscriber crashes randomly

Please could someone help me with the Google Pubsub Python Client Library? I am following the tutorial at https://cloud.google.com/pubsub/docs/pull#pubsub-pull-messages-async-python closely and seem to get unprompted errors. I have a simple script called 'sendmessage.py' that sends a text message...
Paul
1

votes
1

answer
177

Views

Streaming from google pub/sub using apache flink

I spent about couple of hours to find a way to stream from google pub/sub using apache-flink. Does apache-flink have support to stream from google pub/sub ? if yes, how can i do this ?
Md Shihab Uddin
1

votes
1

answer
35

Views

Can we increase number of topics and subscriptions in a google console project?

I am using google cloud pub/sub for receiving the push notifications of any gmail user by setting a watch on the concerned users mailbox. I am able to do that. I am creating a one topic per user in my google console project and also creating one subscription per topic. As per my knowledge we cannot...
Imtiaz Hussain
1

votes
1

answer
44

Views

GCS Object Change Notifications for sub-folders

Following up to this thread since I can't comment. I've successfully created object change notifications as instructed by @Brandon Yarbrough (thanks!). But I would like to narrow down the monitoring from the entire bucket to a sub-folder. For example, from 'gs://mainbucket/' to 'gs://mainbucket/subf...
Larry Cai
1

votes
1

answer
53

Views

environment variables in Docker images in Kubernetes Cluster

I'm working on some GCP apps which are dockerized in a Kubernetes cluster in GCP (I'm new to Docker and Kubernetes). In order to access some of the GCP services, the environment variable GOOGLE_APPLICATION_CREDENTIALS needs to point to a credentials file. Should the environment variable be set and t...
CCC

View additional questions