Course Overview
TOPGet hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hands-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning. This course covers structured, unstructured, and streaming data.
Scheduled Classes
TOP| 12/01/26 - TDV - Virtual-Instructor Led - Virtual-Instructor Led | (click to enroll)
 |
What You'll Learn
TOPDesign scalable data processing systems in Google Cloud.
- Differentiate data architectures and implement data lakehouse and pipeline concepts.
- Build and manage robust streaming and batch data pipelines.
- Utilize AI/ML tools to optimize performance and gain process and data insights.
Outline
TOPImplement robust data governance and security practices across the unified data platform, including sensitive data protection and metadata management.
- Explore advanced analytics and machine learning directly on lakehouse data
- Reinforce the core principles of Google Cloud's data platform
- Lab: Getting Started with BigQuery ML
- Lab: Vector Search with BigQuery
- Explain the critical role of a data engineer in developing and maintaining batch data pipelines.
- Describe the core components and typical lifecycle of batch data pipelines from ingestion to downstream consumption.
- Analyze common challenges in batch data processing, such as data volume, quality, complexity, and reliability, and identify key Google Cloud services that can address them.
- Design scalable batch data pipelines for high-volume data ingestion and transformation.
- Optimize batch jobs for high throughput and cost-efficiency using various resource management and performance tuning techniques.
- Lab: Build a Simple Batch Data Pipeline with Serverless for Apache Spark (optional)
- Lab: Build a Simple Batch Data Pipeline with Dataflow Job Builder UI (optional)
- Develop data validation rules and cleansing logic to ensure data quality within batch pipelines.
- Implement strategies for managing schema evolution and performing data deduplication in large datasets.
- Lab: Validate Data Quality in a Batch Pipeline with Serverless for Apache Spark (optional)
- Orchestrate complex batch data pipeline workflows for efficient scheduling and lineage tracking.
- Implement robust error handling, monitoring, and observability for batch data pipelines.
- Lab: Building Batch Pipelines in Cloud Data Fusion
- Introduce the the scenario that will be used to bring hands on learning to building streaming data pipelines
- Describe the concept of streaming data pipelines, challenges associated with it, and the role of these pipelines within the data engineering process.
- Understand various streaming use cases and their applications, including Streaming ETL, Streaming AI/ML, Streaming Application, and Reverse ETL
- Identify and describe common sample architectures for streaming data, including Streaming ETL, Streaming AI/ML, Streaming Application, and Reverse ETL.
- Define messaging concepts for Apache Kafka
- Establish a streaming pipeline from Dataflow to Bigtable
- Analyze the Bigtable continuous data stream for trends using BigQuery
- Synchronize the trends analysis back into the user-facing application
- Lab: Stream data with pipelines - Esports use case (optional)
- Lab: Use Apache Beam and Bigtable to enrich esports downloadable content (DLC) data
- Lab: Stream e-sports data with Pub/Sub and BigQuery
- Lab: Monitor e-sports chat with Streamlit
- Know when to use Pub/Sub or Managed Service for Apache Kafka
- Describe the service and challenges with streaming data
- Build and deploy a streaming pipeline
- Explore various data ingestion methods
- Use BigQuery continuous queries, BigQuery ETL, and reverse ETL
- Configure Pub/Sub to BigQuery streaming
- Architecting BigQuery streaming pipelines
- Describe the big picture of data movement and interaction
- Next steps
- Explain the role of a data engineer.
- Understand the differences between a data source and a data sink.
- Explain the different types of data formats.
- Explain the storage solution options on Google Cloud.
- Learn about the metadata management options on Google Cloud.
- Understand how to share datasets with ease using Analytics Hub.
- Understand how to load data into BigQuery using the Google Cloud console or the gcloud CLI.
- Lab: Loading Data into BigQuery
- Explain the baseline Google Cloud data replication and migration architecture.
- Understand the options and use cases for the gcloud command-line tool.
- Explain the functionality and use cases for Storage Transfer Service.
- Explain the functionality and use cases for Transfer Appliance.
- Understand the features and deployment of Datastream.
- Explain the baseline extract and load architecture diagram.
- Understand the options of the bq command-line tool.
- Explain the functionality and use cases for BigQuery Data Transfer Service.
- Explain the functionality and use cases for BigLake as a non-extract-load pattern
- Lab: BigLake: Qwik Star
- Explain the baseline extract, load, and transform architecture diagram.
- Understand a common ELT pipeline on Google Cloud.
- Explain the functionality and use cases for Dataform.
- Lab: Create and Execute a SQL Workflow in Dataform
- Explain the baseline extract, transform, and load architecture diagram.
- Learn about the GUI tools on Google Cloud used for ETL data pipelines.
- Explain batch data processing using Dataproc.
- Learn how to use Dataproc Serverless for Spark for ETL.
- Explain streaming data processing options.
- Explain the role Bigtable plays in data pipelines.
- Lab: Use Dataproc Serverless for Spark to Load BigQuery (optional)
- Lab: Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow
- Explain the automation patterns and options available for pipelines.
- Learn about Cloud Scheduler and Workflows.
- Learn about Cloud Composer.
- Learn about Cloud Run functions.
- Explain the functionality and automation use cases for Eventarc.
- Lab: Use Cloud Run Functions to Load BigQuery (optional)
- Compare and contrast data lake, data warehouse, and data lakehouse architectures
- Evaluate the benefits of the lakehouse approach
- Discuss data storage options, including Cloud Storage for files, open table formats like Apache Iceberg, BigQuery for analytic data, and AlloyDB for operational data.
- Understand the role of AlloyDB for operational data use cases.
- Explain why BigQuery is a scalable data warehousing solution on Google Cloud.
- Discuss the core concepts of BigQuery.
- Understand BigLake's role in creating a unified lakehouse architecture and its integration with BigQuery for external data.
- Learn how BigQuery natively interacts with Apache Iceberg tables via BigLake.
Prerequisites
TOPUnderstanding of data engineering principles, including ETL/ELT processes, data modeling, and common data formats (Avro, Parquet, JSON).
- Familiarity with data architecture concepts, specifically Data Warehouses and Data Lakes.
- Proficiency in SQL for data querying.
- Proficiency in a common programming language (Python recommended).
- Familiarity with using Command Line Interfaces (CLI).
- Familiarity with core Google Cloud concepts and services (Compute, Storage, and Identity management).
Who Should Attend
TOPData Engineers, Data Analysts, and Data Architects
