Use Apache Airflow with Teradata Vantage

Overview

This quickstart demonstrates how to use airflow with Teradata Vantage. Airflow will be installed on a Ubuntu System.

Prerequisites

• Ubuntu 22.x
• Access to a Teradata Vantage instance.
  노트

  If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com

• Python 3.8, 3.9, 3.10 or 3.11 installed.
• pip

Install Apache Airflow

1. Set the AIRFLOW_HOME environment variable. Airflow requires a home directory and uses ~/airflow by default, but you can set a different location if you prefer. The AIRFLOW_HOME environment variable is used to inform Airflow of the desired location.

2. Install apache-airflow stable version 2.8.2 from PyPI repository:

3. Install the Airflow Teradata provider stable version from PyPI repository:

   노트

   For security reasons, the test connection functionality is disabled by default across Airflow UI, API and CLI. The availability of the functionality can be controlled by the test_connection flag in the core section of the Airflow configuration ($AIRFLOW_HOME/airflow.cfg) or define below environment variable before starting airflow server:

Start Airflow Standalone

1. Run Airflow Standalone
2. Access the Airflow UI. Visit https://localhost:8080 in the browser and log in with the admin account details shown in the terminal.

Teradata Connections may be defined in Airflow in the following ways:

1. Using Airflow Web UI
2. Using Environment Variable

Define a Teradata connection in Airflow Web UI

1. Open the Admin -> Connections section of the UI. Click the Create link to create a new connection.

2. Fill in input details in New Connection Page.

   • Connection Id: Unique ID of Teradata Connection.
   • Connection Type: Type of the system. Select Teradata.
   • Database Server URL (required): Teradata instance hostname to connect to.
   • Database (optional): Specify the name of the database to connect to
   • Login (required): Specify the user name to connect.
   • Password (required): Specify the password to connect.
   • Click on Test and Save.

Define a Teradata connection in Environment Variable

Airflow connections may be defined in environment variables in either of one below formats.

1. JSON format

2. URI format

   노트

   The naming convention is AIRFLOW_CONN_CONN_ID, all uppercase (note the single underscores surrounding CONN). So if your connection id is teradata_conn_id then the variable name should be AIRFLOW_CONN_TERADATA_CONN_ID

JSON format example

URI format example

Refer Teradata Hook for detailed information on Teradata Connection in Airflow.

Define a DAG in Airflow

Create a DAG as a python file like sample.py under DAG_FOLDER - $AIRFLOW_HOME/files/dags directory.

Load DAG

Airflow loads DAGs from Python source files, which it looks for inside its configured DAG_FOLDER - $AIRFLOW_HOME/files/dags directory.

Run DAG

DAGs will run in one of two ways:

1. When they are triggered either manually or via the API
2. On a defined schedule, which is defined as part of the DAG example_teradata_operator is defined to trigger as manually. To define a schedule, any valid crontab schedule value can be passed to the schedule argument.

Summary

This quickstart demonstrated how to use Airflow and the Airflow Teradata provider with a Teradata Vantage instance. The example DAG provided creates my_users table in the Teradata Vantage instance defined in Connection UI.

Further reading

• airflow documentation
• airflow DAGs

노트

If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members.