Typical data replication job consists of two phases; the initial load and replicating changes. It may be time consuming to build a separate pipeline for each table for initial and incremental loads. It is better to build a dynamic pipeline driven by metadata. In this post you can find how…

How would you load 100s of tables from GCS to BigQuery? Probably, you would use some python script, an ETL tool like Dataflow or Cloud DataFusion. In this post, you will find; how you can load any BQ table from GCS with Composer and one dynamic DAG: Before getting…

Background: I searched for an easy, end-to-end, working guide to configure Oracle Database Gateway with use of Simba BigQuery ODBC driver. However I could not find a complete guide so I decided to write this one. Oracle has a cool feature called heterogenous services (or Oracle Database Gateway) to connect…

Do you want to deploy Trino on Dataproc easily? Are you searching for Trino initialization script? Here it is; Download “trino.sh” from github and upload it to your GCS bucket. Here is my github link https://github.com/sametkaradag/initialization-actions/blob/master/trino/trino.sh which is fork of https://github.com/GoogleCloudDataproc/initialization-actions (waiting for the pull as of writing this post)

Do you need to fill a BigQuery table with random set of numbers, dates…? cool go on.. BigQuery has a cool Generate_Array function which gets the range bounds as inputs and generates an array (nested field). Generate_date_array is the equivalent for generating date arrays. Unnest function can be used to…

Prerequisites: You need terraform Refer to https://learn.hashicorp.com/tutorials/terraform/install-cli for installing terraform. 1- Create tfvar file: #cat terraform.tfvars project = "project-id" region = "europe-west4" 2- Create tf file and put mysql database related information in it, such as instance, database name and root password: #cat main.tf variable "project" {} variable "region" {} …

Duplicate data sometimes can cause wrong aggregates or results. You probably need to remove those duplicate rows before doing any aggregation, join or calculation. There are various ways to deal with duplicate data and you can find one of these methods to deal with the duplicate keys/columns/rows in this post…

Want to migrate Oracle Database to Big Query? Don’t know how to move data from Oracle to Big Query? Want to do this with GCP native tools? Keep reading, you are on the right post! Here are some advantages using Dataproc and Sqoop for your Oracle migrations; Dataproc will be…