In today’s data-driven world, the ability to efficiently ingest, transform, and analyze data in real-time is crucial for organizations to gain insights and make data-driven decisions. …

Migrating data from legacy systems to BigQuery is a common need in today’s data-driven world. This process involves several challenges, such as dealing with char columns containing empty spaces, removing leading zeros, handling precision in floating-point data, and managing time zone issues. …

In this blog post, we will explore how to effectively manage access to your Analytics Hub and authorized views in Google BigQuery. We will use a sample project structure and authorization flow to demonstrate best practices. Let’s dive in. Project Structure Imagine an organization, “Organization A,” with the following project…

Technical indicators are essential tools used by traders and analysts to gain insights into price trends, momentum, and potential reversal points in financial markets. These indicators are often calculated using historical price data and can provide valuable information for predicting stock or cryptocurrency prices and identifying overbought or oversold situations…

Analytics Lakehouse is a unified data platform that combines the best of data warehouses and data lakes. It provides a single, scalable, and secure repository for all your data, and makes it easy to analyze and visualize your data, regardless of its format or structure. Data lakehouses offer the scalability…

There are instances where batch data jobs need to run for extended periods. Some of the reasons may be API rate limits or interactions with multiple systems. While serverless options like Cloud Run and Cloud Functions have duration limitations, and Spark serverless may not be the best fit for sequential…

You need to follow a process similar to a migration. — There is no direct option to change dataset region / location or name as of writing this post. But there is BQ Data Transfer Service Dataset Copy feature which will tackle the hardest part which is moving tables and the data in an automated and fastest way. To change your…

Today Dataproc Serverless is the modernest way to run your spark jobs in GCP. …

In this blog post you can find how you can create parametric/dynamic pipelines to handle SCD2 updates automatically for any table in BigQuery using Composer and Airflow. Slowly Changing Dimension (SCD) Type 2 is a common technique used within data warehouses to handle changes in the dimension tables. If you…