什么是Aurora

• Amazon Aurora是一种基于云且完全托管关系型数据库服务，与MySQL 和 PostgreSQL 数据库兼容，完全托管意味着自动对数据库进行管理，包括管理数据备份、硬件配置和软件更新等操作

• Amazon Aurora提供了企业级性能

• Amazon Aurora提供了多种实例类型，不同的实例类型有不同的CPU，内存，带宽等配置，用户可以根据需求选择合适的实例类型来支持自己的业务

• Amazon Aurora 提供了可扩展性，可以根据需要自动扩大或缩小存储规模。Amazon Aurora 将自动从最小 10 GB 存储扩展到 64 TB 存储，每次增量为 10 GB，且扩展不会影响数据库的性能

什么是Redshift

• Amazon Redshift 是一个基于云的完全托管的数据仓库服务。开发人员可以使用 redshift 来构建、查询和维护多达 PB 或更多的数据。

• 每个 Redshift 数据仓库都是完全托管的，这意味着备份创建、安全和配置等管理任务都是自动化的。

• Redshift 中的每个数据仓库都由一组节点组成。这些节点被组织成集群，使得对大量数据运行分析查询变得容易。

• 许多现有的基于 SQL 的客户端，以及范围广泛的数据源和数据分析工具，都可以与 Redshift 一起使用。它具有稳定的架构，可以轻松地与各种商业智能工具进行交互。

• 由于 Redshift 旨在处理大量数据，因此其模块化设计使其能够轻松扩展。它的多层结构使一次处理多个查询变得简单。

• 可以从 Redshift 集群创建切片，允许对数据集进行更精细的检查

• Redshift 提供的性能指标可用于跟踪数据库的健康状况

为什么会有从Aurora到Redshift数据同步的需求

• Aurora 是一个基于行的数据库，这使得它最适合事务查询（OLTP）业务，比如Web 应用程序。想根据用户 ID 查找用户名？使用 Aurora 很容易做到。

• Redshift 是一个基于列的数据库，旨在高效处理对大量数据的复杂查询，这使得它最适合数据分析（OLAP）业务，比如电商网站用户行为分析。

• 比如我们可以用Aurora来实现电商网站的基本业务功能，用户管理，订单管理，商品管理等基本功能，然后用Redshift来实现用户行为分析，商品销售情况分析等等需要处理海量数据的查询功能。

AWS Data Migration Service

• AWS Database Migration Service (AWS DMS) 用于管理和迁移云上的数据库。

• 借助 AWS DMS，客户可以跨许多流行的商业数据库同步数据，包括 Microsoft SQL Server、Amazon Aurora 和 MySQL。

• 使用 AWS DMS，用户可以运行一次性迁移或设置持续同步。

• 在同步数据库之前，需要在云服务器上创建一个复制实例。此实例读取源数据并将其加载到目标。

如何使用AWS DMS从Aurora同步数据到Redshift

• 在实际开发过程中，我们的web服务器位于VPC public subnet中，允许任何客户端进行访问

• 用户数据库一般位于VPC的private subnet中，且只允许同一个VPC的web服务器进行访问，以保证数据库的安全

• DMS进行数据同步使用的instance也只需要位于private subnet，无需对外开放

• 因此我们首先需要创建一个VPC，vpc的架构如下：

接下来我们详细介绍每一部分需要创建的资源。

IAM

如果在资源创建过程中出现类似如下没有权限的错误，则需要在AWS Web Console IAM页面为用户添加相应的权限。

User: arn:aws:iam::****:user/*** is not authorized to perform: dms:CreateReplicationSubnetGroup on resource: arn:aws:dms:ap-southeast-1:****:subgrp:aurora-to-redshift-instance because no identity-based policy allows the dms:CreateReplicationSubnetGroup action (Service: AWSDatabaseMigrationService; Status Code: 400; Error Code: AccessDeniedException; ; Proxy: null)

本文中用到的权限如下：

• AmazonS3FullAccess： AWS Managed Policy

• AmazonVPCFullAccess： AWS Managed Policy

• AmazonRDSFullAccess： AWS Managed Policy

• AmazonRedshiftFullAccess： AWS Managed Policy

• DMS没有找到AWS Managed Policy，需要自己手动创建一下，内容如下：

{"Version": "2012-10-17","Statement": [{"Sid": "VisualEditor0","Effect": "Allow","Action": "dms:*","Resource": "*"}]
}

VPC

• vpc

• cird block： 10.10.0.0/16

• internet gateway

• public subnet

• 三个Availability Zone

• private subnet

• 三个Availability Zone

• public route table: public subnet关联的 route table

• destination： 0.0.0.0/0 target: internet-gateway-id(允许与外界进行通信)

• destination：10.10.0.0/16 local（内部通信）

• private route table：private subnet关联的 route table

• destination：10.10.0.0/16 local（内部通信）

• web server security group

• 允许任意ip对443端口进行访问

• 允许自己的ipdui22端口进行访问，以便ssh到服务器上向数据库插入数据

• private dms security group

• 只允许vpc内部访问

• private aurora security group

• 允许web server security group对3306端口进行访问

• 允许dms security group对3306端口进行访问

• private redshift aurora security group

• 允许web server security group对5439端口进行访问

• 允许dms security group对5439端口进行访问

VPC全部资源的serverless文件:

• custom:bucketNamePrefix 替换为自己的创建的bucket

• resources:Parameters:MyIP 替换为自己的public ip

service: aurora-to-redshift-vpccustom:bucketNamePrefix: "bucketName"provider:name: awsregion: ${opt:region, "ap-southeast-1"}stackName: ${self:service}deploymentBucket:name: com.${self:custom.bucketNamePrefix}.deploy-bucketserverSideEncryption: AES256resources:Parameters:VpcName:Type: StringDefault: "test-vpc"MyIP:Type: StringDefault: "your public ip"Resources:VPC:Type: "AWS::EC2::VPC"Properties:CidrBlock: "10.10.0.0/16"EnableDnsSupport: trueEnableDnsHostnames: trueInstanceTenancy: defaultTags:- Key: NameValue: !Sub "VPC_${VpcName}"# Internet GatewayInternetGateway:Type: "AWS::EC2::InternetGateway"Properties:Tags:- Key: NameValue: !Sub "VPC_${VpcName}_InternetGateway"VPCGatewayAttachment:Type: "AWS::EC2::VPCGatewayAttachment"Properties:VpcId: !Ref VPCInternetGatewayId: !Ref InternetGateway# web server security groupWebServerSecurityGroup:Type: AWS::EC2::SecurityGroupProperties:GroupDescription: Allow access from publicVpcId: !Ref VPCSecurityGroupIngress:- IpProtocol: tcpFromPort: 443ToPort: 443CidrIp: "0.0.0.0/0"- IpProtocol: tcpFromPort: 22ToPort: 22CidrIp: !Sub "${MyIP}/32"Tags:- Key: NameValue: !Sub "VPC_${VpcName}_WebServerSecurityGroup"# public route tableRouteTablePublic:Type: "AWS::EC2::RouteTable"Properties:VpcId: !Ref VPCTags:- Key: NameValue: !Sub "VPC_${VpcName}_RouteTablePublic"RouteTablePublicInternetRoute:Type: "AWS::EC2::Route"DependsOn: VPCGatewayAttachmentProperties:RouteTableId: !Ref RouteTablePublicDestinationCidrBlock: "0.0.0.0/0"GatewayId: !Ref InternetGateway# public subnetSubnetAPublic:Type: "AWS::EC2::Subnet"Properties:AvailabilityZone: !Select [0, !GetAZs ""]CidrBlock: "10.10.0.0/24"MapPublicIpOnLaunch: trueVpcId: !Ref VPCTags:- Key: NameValue: !Sub "VPC_${VpcName}_SubnetAPublic"RouteTableAssociationAPublic:Type: "AWS::EC2::SubnetRouteTableAssociation"Properties:SubnetId: !Ref SubnetAPublicRouteTableId: !Ref RouteTablePublicSubnetBPublic:Type: "AWS::EC2::Subnet"Properties:AvailabilityZone: !Select [1, !GetAZs ""]CidrBlock: "10.10.32.0/24"MapPublicIpOnLaunch: trueVpcId: !Ref VPCTags:- Key: NameValue: !Sub "VPC_${VpcName}_SubnetBPublic"RouteTableAssociationBPublic:Type: "AWS::EC2::SubnetRouteTableAssociation"Properties:SubnetId: !Ref SubnetBPublicRouteTableId: !Ref RouteTablePublicSubnetCPublic:Type: "AWS::EC2::Subnet"Properties:AvailabilityZone: !Select [2, !GetAZs ""]CidrBlock: "10.10.64.0/24"MapPublicIpOnLaunch: trueVpcId: !Ref VPCTags:- Key: NameValue: !Sub "VPC_${VpcName}_SubnetCPublic"RouteTableAssociationCPublic:Type: "AWS::EC2::SubnetRouteTableAssociation"Properties:SubnetId: !Ref SubnetCPublicRouteTableId: !Ref RouteTablePublic# instance security groupPrivateDMSInstanceSecurityGroup:Type: AWS::EC2::SecurityGroupProperties:GroupDescription: Allow http to client hostVpcId: !Ref VPCSecurityGroupIngress:- IpProtocol: tcpFromPort: 443ToPort: 443CidrIp: "10.10.0.0/16"Tags:- Key: NameValue: !Sub "VPC_${VpcName}_PrivateDMSInstanceSecurityGroup"# aurora security groupPrivateAuroraSecurityGroup:Type: AWS::EC2::SecurityGroupProperties:GroupDescription: Allow access from dms instance securty groupVpcId: !Ref VPCSecurityGroupIngress:- IpProtocol: tcpFromPort: 3306ToPort: 3306SourceSecurityGroupId: !Ref PrivateDMSInstanceSecurityGroup- IpProtocol: tcpFromPort: 3306ToPort: 3306SourceSecurityGroupId: !Ref WebServerSecurityGroupTags:- Key: NameValue: !Sub "VPC_${VpcName}_PrivateAuroraSecurityGroup"# redshift security groupPrivateRedshiftSecurityGroup:Type: AWS::EC2::SecurityGroupProperties:GroupDescription: Allow access from dms instance securty groupVpcId: !Ref VPCSecurityGroupIngress:- IpProtocol: tcpFromPort: 5439ToPort: 5439SourceSecurityGroupId: !Ref PrivateDMSInstanceSecurityGroup- IpProtocol: tcpFromPort: 5439ToPort: 5439SourceSecurityGroupId: !Ref WebServerSecurityGroupTags:- Key: NameValue: !Sub "VPC_${VpcName}_PrivateRedshiftSecurityGroup"# private route tableRouteTablePrivate:Type: "AWS::EC2::RouteTable"Properties:VpcId: !Ref VPCTags:- Key: NameValue: !Sub "VPC_${VpcName}_RouteTablePrivate"# private subnetSubnetAPrivate:Type: "AWS::EC2::Subnet"Properties:AvailabilityZone: !Select [0, !GetAZs ""]CidrBlock: "10.10.16.0/24"VpcId: !Ref VPCTags:- Key: NameValue: !Sub "VPC_${VpcName}_SubnetAPrivate"RouteTableAssociationAPrivate:Type: "AWS::EC2::SubnetRouteTableAssociation"Properties:SubnetId: !Ref SubnetAPrivateRouteTableId: !Ref RouteTablePrivateSubnetBPrivate:Type: "AWS::EC2::Subnet"Properties:AvailabilityZone: !Select [1, !GetAZs ""]CidrBlock: "10.10.48.0/24"VpcId: !Ref VPCTags:- Key: NameValue: !Sub "VPC_${VpcName}_SubnetBPrivate"RouteTableAssociationBPrivate:Type: "AWS::EC2::SubnetRouteTableAssociation"Properties:SubnetId: !Ref SubnetBPrivateRouteTableId: !Ref RouteTablePrivateSubnetCPrivate:Type: "AWS::EC2::Subnet"Properties:AvailabilityZone: !Select [2, !GetAZs ""]CidrBlock: "10.10.80.0/24"VpcId: !Ref VPCTags:- Key: NameValue: !Sub "VPC_${VpcName}_SubnetCPrivate"RouteTableAssociationCPrivate:Type: "AWS::EC2::SubnetRouteTableAssociation"Properties:SubnetId: !Ref SubnetCPrivateRouteTableId: !Ref RouteTablePrivateOutputs:VPC:Description: "VPC."Value: !Ref VPCExport:Name: !Sub "${self:provider.stackName}"SubnetsPublic:Description: "Subnets public."Value:!Join [",",[!Ref SubnetAPublic, !Ref SubnetBPublic, !Ref SubnetCPublic],]Export:Name: !Sub "${self:provider.stackName}-PublicSubnets"SubnetsPrivate:Description: "Subnets private."Value:!Join [",",[!Ref SubnetAPrivate, !Ref SubnetBPrivate, !Ref SubnetCPrivate],]Export:Name: !Sub "${self:provider.stackName}-PrivateSubnets"DefaultSecurityGroup:Description: "VPC Default Security Group"Value: !GetAtt VPC.DefaultSecurityGroupExport:Name: !Sub "${self:provider.stackName}-DefaultSecurityGroup"WebServerSecurityGroup:Description: "VPC Web Server Security Group"Value: !Ref WebServerSecurityGroupExport:Name: !Sub "${self:provider.stackName}-WebServerSecurityGroup"PrivateDMSInstanceSecurityGroup:Description: "The id of the InstanceSecurityGroup"Value: !Ref PrivateDMSInstanceSecurityGroupExport:Name: !Sub "${self:provider.stackName}-PrivateDMSInstanceSecurityGroup"PrivateAuroraSecurityGroup:Description: "The id of the AuroraSecurityGroup"Value: !Ref PrivateAuroraSecurityGroupExport:Name: !Sub "${self:provider.stackName}-PrivateAuroraSecurityGroup"PrivateRedshiftSecurityGroup:Description: "The id of the RedshiftSecurityGroup"Value: !Ref PrivateRedshiftSecurityGroupExport:Name: !Sub "${self:provider.stackName}-PrivateRedshiftSecurityGroup"

Aurora

• DB Cluster Parameter Group

• 要使用DMS对aurora进行数据同步，需要打开binlog相关的配置，默认的Cluster Parameter Group中binlog的配置是关闭的，并且默认Cluster Parameter Group是不可修改的

• 因此必须创建一个自己的Cluster Parameter Group ，并设置binlog相关的配置

• Family: aurora-mysql8.0

• Parameters:

• binlog_format: "ROW"

• binlog_row_image: "Full"

• binlog_checksum: "NONE"

• VpcSecurityGroupIds

• private aurora security group

DB Parameter Group

• Family: aurora-mysql8.0

• Subnet group

• 创建一个包含private subnet的private subnet group就可以

• DB Cluster

• subnet group

• private subnet group

Engine: aurora-mysql

• EngineVersion: 8.0.mysql_aurora.3.02.0

• 可以使用如下命令查看支持的EngineVersion：

aws rds describe-db-engine-versions --engine aurora --query "DBEngineVersions[].EngineVersion"

• DB Cluster Instance

• Engine: aurora-mysql

• DBInstanceClass: db.t2.small

• 可以使用如下命令查看支持的instance类型：

aws rds describe-orderable-db-instance-options --engine aurora-mysql --engine-version "8.0.mysql_aurora.3.02.2" --query "OrderableDBInstanceOptions
[].DBInstanceClass"

aurora全部资源的serverless文件:

• custom:bucketNamePrefix 替换为自己的创建的bucket

service: aurora-to-redshift-auroracustom:bucketNamePrefix: "bucketName"provider:name: awsregion: ${opt:region, "ap-southeast-1"}stackName: ${self:service}deploymentBucket:name: com.${self:custom.bucketNamePrefix}.deploy-bucketserverSideEncryption: AES256resources:Parameters:ServiceName:Type: StringDefault: aurora-to-redshiftResources:DBClusterParameterGroup:Type: "AWS::RDS::DBClusterParameterGroup"Properties:DBClusterParameterGroupName: cluster-aurora-mysql8-with-binlogDescription: DB Cluster Parameter Group with aurora-mysql8.0 engine, and binlog enabledFamily: aurora-mysql8.0Parameters:binlog_format: "ROW"binlog_row_image: "Full"binlog_checksum: "NONE"DBParameterGroup:Type: "AWS::RDS::DBParameterGroup"Properties:DBParameterGroupName: instance-mysql8Description: DB Instance Parameter GroupFamily: aurora-mysql8.0PrivateDBSubnetGroup:Type: "AWS::RDS::DBSubnetGroup"Properties:DBSubnetGroupName: private-subnetDBSubnetGroupDescription: "private subnet"SubnetIds:Fn::Split:- ","- Fn::ImportValue: !Sub "${ServiceName}-vpc-PrivateSubnets"RDSCluster:Type: "AWS::RDS::DBCluster"Properties:DBClusterIdentifier: test-data-sync-auroraDBClusterParameterGroupName:Ref: DBClusterParameterGroupDBSubnetGroupName:Ref: PrivateDBSubnetGroupVpcSecurityGroupIds:- Fn::ImportValue: !Sub "${ServiceName}-vpc-PrivateAuroraSecurityGroup"Engine: aurora-mysqlEngineVersion: 8.0.mysql_aurora.3.02.2DatabaseName: devMasterUserPassword: aurora_admin_2022MasterUsername: adminRDSDBInstance:Type: "AWS::RDS::DBInstance"Properties:DBClusterIdentifier:Ref: RDSClusterDBInstanceIdentifier: test-data-sync-aurora-instanceDBInstanceClass: db.t3.mediumDBParameterGroupName:Ref: DBParameterGroupDBSubnetGroupName:Ref: PrivateDBSubnetGroupEngine: aurora-mysqlOutputs:AuroraEndpoint:Description: "Aurora endpoint"Value: !GetAtt RDSCluster.Endpoint.AddressExport:Name: !Sub "${self:provider.stackName}-AuroraEndpoint"AuroraPort:Description: "Aurora port"Value: !GetAtt RDSCluster.Endpoint.PortExport:Name: !Sub "${self:provider.stackName}-AuroraPort"

• 在创建完资源后，连接数据库，执行一下sql创建数据

• 连接数据库需要创建一个位于vpc public subnet的服务器，创建过程参考这里：

• https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_Tutorials.WebServerDB.LaunchEC2.html

• vpc： 选择上面步骤中我们自己创建的vpc

• subnet：选择任意一个public subnet

• security group：web server security group

• key pair: 创建一个名为tutorial-ec2-instance-web-server的key pari，选择pem(ssh)格式

• 连接web服务器：参考https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Connecting.html#Aurora.Connecting.AuroraMySQL

• 打开终端或者SSH Client

• 找到创建key pair时下载的文件： tutorial-ec2-instance-web-server.pem

• 执行一下命令修改文件的执行权限

chmod 400 tutorial-ec2-instance-web-server.pem

• 使用Public DNS连接到ec2实例，可以在aws web console打开ec2实例的页面，点击右上角的Connect来找到如何连接

ssh -i "tutorial-ec2-instance-web-server.pem" ec2-user@${ec2_public_dns}.ap-southeast-1.compute.amazonaws.com

• 连接完成后，安装mysql

sudo yum install mysql

• 连接aurora，输入以下命令，然后输入密码继续

mysql -h {aurora-cluster}.ap-southeast-1.rds.amazonaws.com -P 3306 -u admin -p

• 执行以下sql插入数据

USE dev;
CREATE TABLE person (person_id int,last_name varchar(255),first_name varchar(255),city varchar(255)
);
INSERT INTO person VALUES (1, 'win', 'jessica', 'sh');

Redshift

• Cluster subnet group

• 创建一个包含private subnet的private subnet group

• Cluster

• ClusterSubnetGroupName

• 使用private subnet group

• VpcSecurityGroupIds

• 使用private redshift security group

• NodeType: dc2.large

• ClusterType: single-node

• redshift全部资源的serverless文件:

• custom:bucketNamePrefix 替换为自己的创建的bucket

service: aurora-to-redshift-redshiftcustom:bucketNamePrefix: "bucketName"provider:name: awsregion: ${opt:region, "ap-southeast-1"}stackName: ${self:service}deploymentBucket:name: com.${self:custom.bucketNamePrefix}.deploy-bucketserverSideEncryption: AES256resources:Parameters:ServiceName:Type: StringDefault: aurora-to-redshiftResources:ClusterSubnetGroup:Type: "AWS::Redshift::ClusterSubnetGroup"Properties:Description: Private Cluster Subnet GroupSubnetIds:Fn::Split:- ","- Fn::ImportValue: !Sub ${ServiceName}-vpc-PrivateSubnetsTags:- Key: NameValue: private-subnetCluster:Type: "AWS::Redshift::Cluster"Properties:ClusterIdentifier: test-data-sync-redshiftClusterSubnetGroupName: !Ref ClusterSubnetGroupVpcSecurityGroupIds:- Fn::ImportValue: !Sub ${ServiceName}-vpc-PrivateRedshiftSecurityGroupDBName: devMasterUsername: adminMasterUserPassword: Redshift_admin_2022NodeType: dc2.largeClusterType: single-nodeOutputs:RedshiftEndpoint:Description: "Redshift endpoint"Value: !GetAtt Cluster.Endpoint.AddressExport:Name: !Sub "${self:provider.stackName}-RedshiftEndpoint"RedshiftPort:Description: "Redshift port"Value: !GetAtt Cluster.Endpoint.PortExport:Name: !Sub "${self:provider.stackName}-RedshiftPort"

DMS

• Subnet group

• 创建一个包含private subnet的private subnet group

• Replication instance

• ReplicationInstanceClass: dms.t2.small

• ReplicationSubnetGroupIdentifier:

• 使用private subnet group

• VpcSecurityGroupIds

• 使用dms security group

• Source endpoint

• EndpointType: source

• EngineName: aurora

• ServerName: aurora cluster endpoint

• Port： 3306

• Target endpoint

• EndpointType: target

• EngineName: ednpoint

• ServerName: redshift cluster endpoint

• Port： 5439

• Database migration task

• MigrationType: full-load-and-cdc

• ReplicationInstanceArn

• TableMappings："{ \"rules\": [ { \"rule-type\": \"selection\", \"rule-id\": \"1\", \"rule-name\": \"1\", \"object-locator\": { \"schema-name\": \"dev\", \"table-name\": \"test-aurora-to-redshift\" }, \"rule-action\": \"include\" } ] }"

• ReplicationTaskSettings: "{ \"FullLoadSettings\": {\"TargetTablePrepMode\": \"DO_NOTHING\"}}"

• dms全部资源的serverless文件

• custom:bucketNamePrefix 替换为自己的创建的bucket

service: aurora-to-redshift-dmscustom:bucketNamePrefix: "bucketName"provider:name: awsregion: ${opt:region, "ap-southeast-1"}stackName: ${self:service}deploymentBucket:name: com.${self:custom.bucketNamePrefix}.deploy-bucketserverSideEncryption: AES256resources:Parameters:ServiceName:Type: StringDefault: aurora-to-redshiftResources:ReplicationSubnetGroup:Type: "AWS::DMS::ReplicationSubnetGroup"Properties:ReplicationSubnetGroupDescription: instance to sync data from aurora to redshiftReplicationSubnetGroupIdentifier: aurora-to-redshift-instanceSubnetIds:Fn::Split:- ","- Fn::ImportValue: !Sub ${ServiceName}-vpc-PrivateSubnetsReplicationInstance:Type: "AWS::DMS::ReplicationInstance"Properties:ReplicationInstanceIdentifier: aurora-to-redshift-instanceReplicationInstanceClass: dms.t2.smallReplicationSubnetGroupIdentifier: !Ref ReplicationSubnetGroupPubliclyAccessible: falseVpcSecurityGroupIds:- Fn::ImportValue: !Sub ${ServiceName}-vpc-PrivateDMSInstanceSecurityGroupAuroraSourceEndpoint:Type: "AWS::DMS::Endpoint"Properties:EndpointIdentifier: auroraEndpointType: sourceServerName:Fn::ImportValue: !Sub ${ServiceName}-aurora-AuroraEndpointPort:Fn::ImportValue: !Sub ${ServiceName}-aurora-AuroraPortUsername: adminPassword: aurora_admin_2022EngineName: auroraRedshiftTargetEndpoint:Type: "AWS::DMS::Endpoint"Properties:EndpointIdentifier: redshiftEndpointType: targetServerName:Fn::ImportValue: !Sub ${ServiceName}-redshift-RedshiftEndpointPort:Fn::ImportValue: !Sub ${ServiceName}-redshift-RedshiftPortDatabaseName: devUsername: adminPassword: Redshift_admin_2022EngineName: redshiftReplicationTask:Type: "AWS::DMS::ReplicationTask"Properties:ReplicationTaskIdentifier: aurora-to-redshift-taskMigrationType: full-load-and-cdcReplicationInstanceArn: !Ref ReplicationInstanceSourceEndpointArn: !Ref AuroraSourceEndpointTableMappings: '{ "rules": [ { "rule-type": "selection", "rule-id": "1", "rule-name": "1", "object-locator": { "schema-name": "dev", "table-name": "%" }, "rule-action": "include" } ] }'TargetEndpointArn: !Ref RedshiftTargetEndpointReplicationTaskSettings: '{ "FullLoadSettings": {"TargetTablePrepMode": "DO_NOTHING"}}'

数据同步

• 上述资源创建完毕后，可以在aws web console运行DMS的Database migration task，执行成功后status会是Load complete, replication ongoing

• 同步完成后，可以通过Redshift query editor来确认数据是否同步成功。

• 继续修改，添加，删除mysql数据，然后通过Redshift query editor来确认数据是否同步成功。

其他同步方法

Amazon Aurora zero-ETL integration with Amazon Redshift

• Amazon即将推出一个Aurora的新功能，实现与 Amazon Redshift 的零 ETL 集成，目前这个功能还没有正式推出，处于Preview阶段，且在美国东部（弗吉尼亚北部）区域可用。以下来自Amazon公告，原文链接在此：

• Amazon Aurora 现在支持与 Amazon Redshift 的零 ETL 集成，以使用 Amazon Redshift 对来自 Aurora 的数 PB 交易数据进行近乎实时的分析和机器学习 (ML)。在事务数据写入 Aurora 的几秒钟内，数据就可以在 Amazon Redshift 中使用，因此您不必构建和维护复杂的数据管道来执行提取、转换和加载 (ETL) 操作。

• 这种零 ETL 集成还使您能够在同一个新的或现有的 Amazon Redshift 实例中分析来自多个 Aurora 数据库集群的数据，以获得跨多个应用程序或分区的整体见解。通过近乎实时地访问交易数据，您可以利用 Amazon Redshift 的分析和功能，例如内置 ML、物化视图、数据共享以及对多个数据存储和数据湖的联合访问，从交易数据和其他数据中获得洞察力。

• Amazon Aurora 与 Amazon Redshift 的零 ETL 集成现已在美国东部（弗吉尼亚北部）区域推出兼容 MySQL 8.0 的 Amazon Aurora MySQL 3 的有限预览版。

AWS Data Pipeline

• AWS Data Pipeline也支持aurora到redshift的持续同步

• 相比于DMS的缺点

• AWS Data Pipeline目前只支持以下region

• 一个AWS Data Pipeline只能同步一张aurora表，aurora表名必须完全匹配

• 相比于DMS的优点

• 同步过程中支持自定义查询语句，可以对源表数据进行筛选之后同步到redshift

AWS Glue

• AWS Glue也支持aurora到redshift同步

• 相比于DMS的缺点

• 一个AWS Data Pipeline只能同步一张aurora表

• 不支持持续同步，只支持全表数据同步

• 需要在glue管理源表和目标表的scheme

• 相比于DMS的优点

• 可以在数据同步过程中对数据列应用mapping规则，可以按一定规则修改列值，可以删除某些列（目标表比源表少某些列），目标表与源表列名也可以不同

如何进行选择

• 如果需要持续同步且要使用的region不支持AWS Data Pipeline，则只能使用AWS DMS

• 如果需要持续同步且要使用的region支持AWS Data Pipeline，则可以根据其他需求决定使用哪个service，比如如果需要对数据进行筛选则可以选择AWS Data Pipeline

• 如果只需要整表一次性同步，且没有数据修改的需求，则可以选择任意一个

• 如果只需要整表一次性同步，且需要在同步过程中对数据进行一定规则的修改，则选择AWS Glue

github

https://github.com/JessicaWin/aws-in-action/tree/master/aurora-to-redshift