Stream connectivity metadata directly to Google Cloud Pub/Sub in real-time

Before each device is able to send data to the application it needs to communicate with the mobile network to ensure that the data is allowed to be sent over the network. The EMnify DataStreamer makes connectivity metadata available in real-time to your cloud platform for detailed insights on the event and usage data of your devices. 

Prerequisites

• A device with a cellular modem and EMnify SIM card
• A Google Cloud account

Benefits 

• connectivity metadata integrated into operational dashboards for service teams
• allows faster triaging between device, connectivity and application issues
• insights about network attach, data and roaming issues or when device is above service limit
• view on service usage and cost per device 
• directly delivered to Google Cloud without need for managing application servers

Integration steps

Hint: users who wish to use the gcloud SDK instead of the Google Cloud web console may skip to Configuring via gcloud SDK.

To create a Pub/Sub topic and subscription, users need to have an associated project. We will create a project from scratch in this guide for illustrative purposes.
In the Google Cloud console,

• Create a new project:

• Give the project the name EMnify Data. The project ID emnify-data is required in a later step
  
• From the sidebar menu, navigate to the Pub/Sub service
  
• Create a Pub/Sub topic called emnify-device-data
• To allow for the EMnify data stream to be written to the topic, click + Add Member under the Permissions tab
• Add the following member with a Pub/Sub Publisher role:
  

  pubsub@emnify-datastreamer.iam.gserviceaccount.com
  

  Copy
  

Adding Stream in the EMnify Portal

Log in to the EMnify portal and navigate to the Integrations page:

Click Add New Data Stream and under Google Pub/Sub click ADD 

• • Enter the Cloud Pub/Sub Topic, in this example emnify-device-data
  • Enter the Project ID  into the "Cloud Pub/Sub Project containing Topic" field, in this example emnify-data

Newly-created data streams are immediately active and should show a green Running status indicating a 200 HTTP response code when correctly configured.
If the configuration is invalid, a warning message will be displayed indicating incorrect configuration. 

Upon successful creation, event data arrives directly into Google Pub/Sub for further processing by other services:

Configuring via gcloud SDK

For users who already use the gcloud SDK, setting up the topic, adding EMnify publish permissions, creating a subscription and checking the incoming messages can be done directly from the CLI in the following four steps:

# Create the topic 'emnify-device-data' and allow EMnify publish permissions
gcloud pubsub topics create emnify-device-data
gcloud pubsub topics add-iam-policy-binding emnify-device-data \
  --member='serviceAccount:pubsub@emnify-datastreamer.iam.gserviceaccount.com' \
  --role='roles/pubsub.publisher'

# ! Note: the data stream must be activated via EMnify API at this step !

# Create a subscription for the 'emnify-device-data' topic and pull published messages
gcloud pubsub subscriptions create my_subscription --topic=emnify-device-data
gcloud pubsub subscriptions pull my_subscription

Verifying the integration

When incoming messages are successfully sent to a Pub/Sub topic, clicking on view messages allows to preview incoming data.

• Click View Messages and select Create a Subscription
• Give the subscription a name, ensure it is of Pull type to read incoming messages and click Next
  
• Click View Messages, select the newly-created subscription and click Pull to show latest published messages:
  

More details of the events can be inspected by expanding the columns icon on the right and selecting message attributes to be previewed: