Stream Events from Redis to Teradata

This guide demonstrates how to ingest event data from a local Redis stream into Teradata using Python.

Architecture

Prerequisites

• Teradata Vantage instance with network access
• Python 3.8+
• uv (Python package manager - install from uv.astral.sh)
• Redis (Docker or local installation)
• Docker (for running Redis container)

Step 1: Create the event data table in Teradata

Connect to your Teradata instance and create the events table:

This table will store all events streamed from Redis.

Step 2: Start Redis locally

We'll use Docker for simplicity. Run the following command:

This starts a Redis container on localhost:6379.

If you prefer to use an existing Redis installation, ensure it's running and accessible on port 6379.

Verify Redis is running:

You should see PONG in response.

Step 3: Set up Python environment with uv

Create pyproject.toml

Create a pyproject.toml file in your project directory:

Sync dependencies

Install all dependencies and create a lockfile:

This creates uv.lock which ensures reproducible environments across machines.

Step 4: Create the event producer script

Create a file named producer.py:

Step 5: Create the event consumer script

Create a file named consumer.py:

Step 6: Set up environment variables

Create a .env file in the same directory as consumer.py:

Replace the placeholders with your actual Teradata credentials.

Step 7: Run the streaming pipeline

Open two terminal windows:

Terminal 1 — Start the consumer (runs continuously, waiting for events):

The consumer will wait for events, showing:

Terminal 2 — Run the producer (publishes events once):

You'll see events appear in Terminal 1 as they're ingested into Teradata:

To stop the consumer, press Ctrl+C in Terminal 1.

Step 8: Verify the data in Teradata

Query the table to confirm all events were ingested:

Expected output: 5

See event details:

Expected output:

Step 9: Clean up

To stop and remove the Redis container:

To drop the Teradata table:

Extending the guide

Run the producer in a loop

To continuously generate events, modify producer.py:

Add multiple consumers

Start multiple consumer instances with different CONSUMER_NAME values to scale horizontally:

Filter events by type

In the consumer, add filtering:

Monitor Redis stream

Check stream size and lag:

Troubleshooting

Redis connection refused:

→ Ensure Redis is running. Check with redis-cli ping

Teradata connection failed:

→ Verify credentials and network access to Teradata instance

Consumer group error:

→ The consumer group already exists from a previous run. You can either:

• Use a different CONSUMER_GROUP name
• Reset with: redis-cli XGROUP DESTROY events:stream teradata-consumer

Missing environment variables:

→ Ensure your .env file exists in the same directory as consumer.py and contains all required variables.

Summary

We now have a working local event streaming pipeline:

• Redis manages event queues with built-in consumer groups
• Producer simulates event sources (easily replaced with real data)
• Consumer durably ingests events into Teradata with transaction management
• Teradata stores the event history for analytics

This architecture scales from development to production by simply pointing to managed Redis and Teradata services.