What is DBT

DBT (data build tool) is a command-line tool that enables data analysts and engineers to transform data in their warehouses simply by writing SELECT statements.

DBT does the T in ELT (Extract, Load, Transform) process — it doesn’t extract or load data, but it’s extremely good at transforming data that’s already loaded into your warehouse.

Pre-requisite

Make sure you have dbt & python installed in your machine. If not, then follow this guide https://docs.getdbt.com/docs/get-started/installation

Installation

  1. Create a new directory & cd into it
mkdir dbt_bytehouse_demo
cd dbt_bytehouse_demo
  1. Reach out to the ByteHouse team for the links to download the latest DBT plugin
pip install ~/Downloads/dbt-bytehouse-1.3.1.tar.gz
  1. Check whether bytehouse plugin is successfully installed on your machine or not
dbt --version
602
  1. Create a new dbt project using dbt init
dbt init
778
  1. We have to update the profiles.yml to setup connection to our data warehouse (bytehouse). For mac machines, profiles.yml can be found here ~/.dbt/profiles.yml
dbt_demo:
  target: dev
  outputs:
    dev:
      type: bytehouse
      schema: dbt_demo

      driver: native
      port: 19000
      user: bytehouse
      host: gateway-rc.volc-boe.offline.bytehouse.cn
      password: 8YwpHMkXtRLI6OSW3iiU
      secure: True
      connect_timeout: 10
      custom_settings: {}
      database_engine: ''
dbt_demo (top level)Your profile name which needs to be referred to in your project
typeDatasource name
schemaDatabase name (There is no schema concept in ByteHouse / ClickHouse)
driverNative (tcp-driver), We don't have support for http-driver yet
userIf using API_KEY as authentication, then "bytehouse", else "AccountID::UserID"
hostFQDN for bytehouse server
passwordIf using API_KEY as authentication, then api_key, else password
secureTrue (Secured connection)
database_engineNo custom database engine
  1. The previous dbt init command will create a sample repository for us to work with. cd into the dbt_bytehouse_demo directory
cd dbt_bytehouse_demo
494
  1. Update dbt_project.yml file with the profile name defined in ~/.dbt/profiles.yml
794
  1. Run "dbt debug" to verify that you have a working connection to bytehouse