EMnify Data Streamer integration into AWS Kinesis

Stream connectivity metadata directly to AWS Kinesis in real-time

Before each device is able to send data to the application it needs to communicate with the mobile network to ensure that the data is allowed to be send over the network. This communication happens through signaling events which is usually hidden from the application.

Prerequisites

• A device with a cellular modem and EMnify SIM card
• An AWS account 

Benefits 

• connectivity metadata integrated into operational dashboards for service teams
• allows faster triaging between device, connectivity and application issue
• insights about network attach, data and roaming issues or when device is above service limit
• overview of service (data, SMS) usage and cost per device 
• intra-cloud delivered to Kinesis without need for managing public IPs

Kinesis integration steps

Data streams can be directly ingested by AWS Kinesis which then allows for real-time analytics on streaming data.

1. In the AWS console, navigate to the Kinesis service and create a data stream:
2. In IAM -> Policies click Create Policy which allows PutRecord and PutRecords write permissions to the Kinesis stream.
   
3. In IAM -> Roles, click Create Role for Kinesis Analytics and click Next: Permissions
4. Attach the policy created in step 2 and click Next: Tags -> Create Role.
   (Important: Make note of the Role's ARN as it will be needed for adding a new data stream in the EMnify UI)
5. Edit the newly-created role and click Trust Relationships -> Edit Trust Relationships
   
6. Copy the policy document JSON below and replace your organisation ID in after org- to allow write access:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "",
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::884047677700:role/datastreamer"
           ]
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "org-1234"
           }
         }
       }
     ]
   }

   To know your organization ID, send a GET request to the EMnify API, or find it on the EMnify portal under Secure Connections.

Adding Stream in the EMnify Portal

In to the EMnify portal, navigate to Integrations to create, inspect and manage Data Streams:

1. Click Add New Data Stream and under AWS Kinesis click ADD
2. Enter the details for the Kinesis Stream:

• • User Role ARN - the Role ARN created at Step 4 above
  • Region - the region of the stream, this example uses eu-central-1
  • Kinesis stream - the name of the kinesis stream to deliver data to, this example uses emnify-datastreamer
  • Stream Type - Choose the stream type, either usage data or events. It is recommended to have separate integrations for usage data and events.
  • Event Options - If you choose to stream events, select the events you want to stream

Verifying the integration

The data stream should show a 200 HTTP response code in the EMnify portal when correctly configured. In the AWS console, navigate to the 'Monitoring' tab of the kinesis stream. The number of records ingested into the stream via PUT requests should be visible in the lower half of the monitoring console:

Next Steps

As connectivity data is now arriving in real-time, it's simple to implement other services which can react to any events which interest us. One useful integration that can be quickly launched is a simple notification system which delivers Slack alerts based on the contents of platform events.

To get started with this example integration, see the Slack notifications via AWS Lambda article or directly launch the project using a cloudformation template provided in the EMnify GitHub repository.