Spark runtime version 2.0 has become the default version as of Jan 2023. Now you may be getting an error like below if you do not set version in your gcloud command; java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be instantiated at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:586)…

In this blog post you can find how you can create parametric/dynamic pipelines to handle SCD2 updates automatically for any table in BigQuery using Composer and Airflow. Slowly Changing Dimension (SCD) Type 2 is a common technique used within data warehouses to handle changes in the dimension tables. If you…

Typical data replication job consists of two phases; the initial load and replicating changes. It may be time consuming to build a separate pipeline for each table for initial and incremental loads. It is better to build a dynamic pipeline driven by metadata. In this post you can find how…

How would you load 100s of tables from GCS to BigQuery? Probably, you would use some python script, an ETL tool like Dataflow or Cloud DataFusion. In this post, you will find; how you can load any BQ table from GCS with Composer and one dynamic DAG: Before getting into…

Background: I searched for an easy, end-to-end, working guide to configure Oracle Database Gateway with use of Simba BigQuery ODBC driver. However I could not find a complete guide so I decided to write this one. Oracle has a cool feature called heterogenous services (or Oracle Database Gateway) to connect…

Do you want to deploy Trino on Dataproc easily? Are you searching for Trino initialization script? Here it is; Download “trino.sh” from github and upload it to your GCS bucket. Here is my github link https://github.com/sametkaradag/initialization-actions/blob/master/trino/trino.sh which is fork of https://github.com/GoogleCloudDataproc/initialization-actions (waiting for the pull as of writing this post)

Do you need to fill a BigQuery table with random set of numbers, dates…? cool go on.. BigQuery has a cool Generate_Array function which gets the range bounds as inputs and generates an array (nested field). Generate_date_array is the equivalent for generating date arrays. Unnest function can be used to…

Prerequisites: You need terraform Refer to https://learn.hashicorp.com/tutorials/terraform/install-cli for installing terraform. 1- Create tfvar file: #cat terraform.tfvars project = "project-id" region = "europe-west4" 2- Create tf file and put mysql database related information in it, such as instance, database name and root password: #cat main.tf variable "project" {} variable "region" {} …

Duplicate data sometimes can cause wrong aggregates or results. You probably need to remove those duplicate rows before doing any aggregation, join or calculation. There are various ways to deal with duplicate data and you can find one of these methods to deal with the duplicate keys/columns/rows in this post…