DP 203 学习笔记

Learning Objects:

工具

Designing and implementing data storage

1. Storage

Azure Synapse Analytics

Azure Databricks

Azure Data Lake Storage Gen2(ADLS2，可代替Hadoop Distributed File System也就是HDFS)

2. Shard

Partition data store in to multiple shards。分块技术提高效率

3. Serving Layer

Designing and developing data processing

1. Investing and transforming data

Transform data: Apache Spark和SQL

Transform feature:

Azure Data Factory(ADF) 可以从storage A导入数据，transform，存入storage B；

Azure Synapse Pipelines是ADF的子集，集成在Azure Synapse Analytics里；

Stream Analytics老，用SQL语句

2. Batch Processing

Azure Data Factory(ADF)

Azure Synapse Pipelines

Azure Databricks

Azure Data Lake Storage Gen2(ADLS2)

3. Stream Processing

Stream Analytics

Azure Databricks

Azure Event Hubs

4. Manage batches and Pipelines

Azure Data Factory(ADF)

Azure Synapse Pipelines

Designing and implementing data security

加密等等

Monitoring and optimizing data storage and data processing

1. Monitor

Azure Monitor

2. Optimize

Azure Data Factory(ADF)

Azure Databricks

Azure Synapse Analytics