Skip to main content
Join our research panel and help shape the future of Teradata.Sign up.

Use Apache Airflow with Teradata Database

Overview​

This quickstart demonstrates how to use airflow with Teradata Database. Airflow will be installed on a Ubuntu System.

Prerequisites​

  • Ubuntu 22.x (Or Windows System for Linux WSL)
  • Access to a Teradata Database instance.
    Note

    If you need a test instance of Teradata, you can provision one for free at https://www.teradata.com/try

  • Python 3.9, 3.10, 3.11 or 3.12 installed.
  • uv

Install Apache Airflow​

  1. Set the AIRFLOW_HOME environment variable. Airflow requires a home directory and uses ~/airflow by default, but you can set a different location if you prefer. The AIRFLOW_HOME environment variable is used to inform Airflow of the desired location.

  2. Install apache-airflow stable version 3.2.2 from PyPI repository:

  3. Install the Airflow Teradata provider stable version from PyPI repository:

    Note

    For security reasons, the test connection functionality is disabled by default across Airflow UI, API and CLI. The availability of the functionality can be controlled by the test_connection flag in the core section of the Airflow configuration ($AIRFLOW_HOME/airflow.cfg) or define below environment variable before starting airflow server:

Start Airflow Standalone​

Note

If you have a previous Airflow installation, run airflow db migrate before starting airflow standalone to ensure your configuration file is up to date for the installed version.

  1. Run Airflow Standalone
  2. Access the Airflow UI. Visit http://localhost:8080 in the browser and log in with the admin account details shown in the terminal.

Teradata connections may be defined in Airflow in the following ways:

  1. Using the Airflow CLI
  2. Using an Environment Variable

Define a Teradata connection using the Airflow CLI​

Use the airflow connections add command to register a connection directly from the terminal. Replace the placeholder values with your actual Teradata host, login, and password.

Verify the connection was created:

Refer Teradata Hook for the full list of supported connection parameters.

Define a Teradata connection in Environment Variable​

Airflow connections may be defined in environment variables in either of one below formats.

  1. JSON format

  2. URI format

    Note

    The naming convention is AIRFLOW_CONN_CONN_ID, all uppercase (note the single underscores surrounding CONN). So if your connection id is teradata_conn_id then the variable name should be AIRFLOW_CONN_TERADATA_CONN_ID

JSON format example​

URI format example​

Refer Teradata Hook for detailed information on Teradata Connection in Airflow.

Define a DAG in Airflow​

Create a DAG as a python file like sample.py under DAG_FOLDER - $AIRFLOW_HOME/dags directory.

Load DAG​

Airflow loads DAGs from Python source files, which it looks for inside its configured DAG_FOLDER - $AIRFLOW_HOME/dags directory.

Run DAG​

DAGs will run in one of two ways:

  1. When they are triggered either manually or via the API
  2. On a defined schedule, which is defined as part of the DAG example_teradata_operator is defined to trigger as manually. To define a schedule, any valid crontab schedule value can be passed to the schedule argument.

Summary​

This quickstart demonstrated how to use Airflow and the Airflow Teradata provider with a Teradata Database instance. The example DAG provided creates my_users table in the Teradata Database instance defined via the Airflow CLI connection.

Further reading​

Also of interest