Store events from Twilio Segment

Overview

This solution listens to events from Twilio Segment and writes data to a Teradata Vantage instance. The example uses Google Cloud but it can be translated into any cloud platform.

Architecture

In this solution, Twilio Segment writes raw event data to Google Cloud Pub/Sub. Pub/Sub forwards events to a Cloud Run application. The Cloud Run app writes data to a Teradata Vantage database. It's a serverless solution that doesn't require allocation or management of any VM's.

Deployment

Prerequisites

1. A Google Cloud account. If you don't have an account, you can create one at https://console.cloud.google.com/.
2. gcloud installed. See https://cloud.google.com/sdk/docs/install.
3. A Teradata Vantage instance that Google Cloud Run can talk to.

note

If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com

Build and deploy

1. Clone the sample repository:

2. The repo contains segment.sql file that sets up the database. the script on your Vantage db using your favorite SQL IDE, Teradata Studio or command line tool called bteq (download for Windows, Linux, macOS. The SQL script will create a new database called Segment and a set of tables to store Segment events.

3. Set the default project and region:

4. Retrieve the project id and the number. We will need it in subsequent steps:

5. Enable required Google Cloud services:

6. Build the application:

7. Define an API key that you will share with Segment. Store the API key in Google Cloud Secret Manager:

8. The application that write Segment data to Vantage will use Cloud Run. We first need to allow Cloud Run to access secrets:

9. Deploy the app to Cloud Run (replace <VANTAGE_HOST> with the hostname or IP of your Teradata Vantage database). The second export statement saves the service url as we need it for subsequent commands:

10. Create a Pub/Sub topic that will receive events from Segment:

11. Create a service account that will be used by Pub/Sub to invoke the Cloud Run app:

12. Give the service account permission to invoke Cloud Run:

13. Allow Pub/Sub to create authentication tokens in your project:

14. Create a Pub/Sub subscription with the service account:

15. Allow Segment to publish to your topic. To do that, assign pubsub@segment-integrations.iam.gserviceaccount.com role Pub/Sub Publisher in your project at https://console.cloud.google.com/cloudpubsub/topic/list. See Segment manual for details.

16. Configure your Google Cloud Pub/Sub a destination in Segment. Use the full topic projects/<PROJECT_ID>/topics/segment-events and map all Segment event types (using * character) to the topic.

Try it out

1. Use Segment's Event Tester functionality to send a sample payload to the topic. Verify that the sample data has been stored in Vantage.

Limitations

• The example shows how to deploy the app in a single region. In many cases, this setup doesn't guarantee enough uptime. The Cloud Run app should be deployed in more than one region behind a Global Load Balancer.

Summary

This how-to demonstrates how to send Segment events to Teradata Vantage. The configuration forwards events from Segment to Google Cloud Pub/Sub and then on to a Cloud Run application. The application writes data to Teradata Vantage.

Further reading

• Segment Pub/Sub destination documentation

note

If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members.