Webinar: Email / Slack notifications from the EMnify Data Streamer

The EMnify Data Streamer can be used to stream events directly to S3. The access credentials and the destination bucket should be configured in the EMnify User Interface. This has been covered in in detail in the previous EMnify webinar "How to store EMnify events and usage data in AWS S3".

The first part of the example stack is a Python lambda which parses and routes incoming messages whenever a new object is written to our S3 bucket. To create this, open the Lambda service in AWS and create a new Python lambda from scratch called s3tosns.py. Create a new role for the lambda which should have the following permissions:

• ReadOnly Access for the S3 bucket with the event data

• Write permissions to CloudWatch Logs

• SNS Publish permissions

A basic working lambda which simply reads the event CSV file looks like the following:

Click + Add Trigger, select S3 and the bucket used for storing incoming EMnify events and select All object create events as the event type to trigger the lambda:

NOTE: Adding a trigger for this lambda can also be done on the S3 bucket directly at Properties → Events → +Add Notification and point the notification to the Lambda above.

At this point, the first lambda should be running and reporting incoming events to the console each time we have EMnify platform events.

To get started with SNS, navigate to Simple Notification Service (SNS) in AWS. Create a topic by clicking on Topics → Create Topic. Name the topic with something like my-email-topic and in Access Policy, allow everyone to publish and subscribe to the topic (this will be restricted to only our Lambdas later). After the topic is created, make note of the ARN which should resemble the following format:
arn:aws:sns:{region}:my-email-topic

Lambda Subscriber

In the same way that we created an email notification topic, create a Slack topic named my-slack-topic and make note of the ARN which should be something like arn:aws:sns:{region}:my-slack-topic and ensure that it has a relaxed access policy for the setup steps.

Lambda Subscriber

Now that we have two externally-publishing lambdas that post incoming SNS messages to either email or Slack channels, we can clean up our routing lambda and add some basic filtering. The final version of the example read & routing lambda looks like the following:

In this basic example, we are doing some routing to different SNS topics based on the text content of the event_type_description row of our event CSV entry.

Test events

Mock test events when creating Lambdas by clicking Select a test event → Configure test events and adding some dummy data. For testing the email and Slack lambdas from this example, use an sns mock message.

Better Templating

SES offers a service for handlebars-like HTML templates. The downside to using this feature is that the templates may only be created via the CLI. An example HTML template which can be uploaded to SES via a JSON file and can be invoked using the .send_templated_mail function as demonstrated in this example lambda.

Restrict Access Policies

Some other steps for cleaning up would be restricting the access policies for the SNS topics so that only our newly-created Lambdas can publish and subscribe.

Add More Filtering

Filtering by event type can be done in the EMnify Data Stream in the EUI, in our first 'routing' lambda and also in the SNS topic subscriptions. For examples with finer granularity on match criteria on incoming SNS messages, see the AWS documentation on SNS Subscription Filter Policies.