文章目录
aws(学习笔记第四十四课) opensearch
- 在
aws上部署opensearch
学习内容：
- 部署
opensearch - 并了解什么是
opensearch
1. 整体架构
1.1 代码链接
代码连接(opensearch-simple-domain)
1.2 整体架构
这里，
- 会构建一个
aws的opensearch服务。 - 自动生成管理员的密码，并将管理员密码报错在
aws secrets manager上。 - 这里，设定允许访问的
source ip adress。
2. 什么是opensearch
2.1 elastic search的aws实现
Elasticsearch 是一个强大的分布式搜索和分析引擎，基于 Apache Lucene 构建，广泛应用于全文搜索、日志分析、实时数据处理等场景。以下是它的 主要功能：
- 全文搜索
- 实时搜索与分析
- 分布式架构与高可用性
- 日志与监控分析
- 结构化 & 非结构化数据处理
- 机器学习与 AI 增强
- 安全与权限管理
2.2 elastic search的数据源
Elasticsearch 可以与多种类型的数据源集成，支持从不同系统中导入数据并进行索引和搜索。以下是 Elasticsearch 主要支持的数据源类型及其集成方式：
- 关系型数据库（MySQL、PostgreSQL、Oracle 等）
- NoSQL 数据库（MongoDB、Cassandra 等）
- 日志系统（Logstash、Filebeat、Fluentd）
- 消息队列（Kafka、RabbitMQ）
- 文件系统（CSV、JSON、日志文件）
- 云服务（AWS S3、Google Cloud Storage）
- 大数据组件（Hadoop、Spark、Flink）
- API 数据（RESTful 服务、爬虫数据）
2.3 开始学习elastic search
3. opensearch的cdk代码架构
3.1 cdk代码的整体架构
3.2 代码详细
3.2.1 创建对opensearch的访问限制
这里首先执行opensearch的版本，选择最新2.19。
OPENSEARCH_VERSION = "2.19"
# Add the authorized IP addresses (using CIDR format) that should
# be granted access to the OpenSearch Domain.
# Create an environment variable before running cdk deploy. E.g.:
OPENSEARCH_ALLOWED_IP='["33.45.123.8/32"]'
# allowed_ip_addresses = os.environ.get("OPENSEARCH_ALLOWED_IP", "x.x.x.x/32")
allowed_ip_addresses = OPENSEARCH_ALLOWED_IP
# Creating OpenSearch access policy to restrict
# access to a specific list of IPs. We are allowing all
# types of HTTP commands.
opensearch_access_policy = cdk_iam.PolicyStatement(
effect=cdk_iam.Effect.ALLOW,
principals=[cdk_iam.AnyPrincipal()],
actions=["es:ESHttp*"],
resources=[],
conditions={
"IpAddress": {
"aws:SourceIp": allowed_ip_addresses
}
}
)
3.2.2 创建对opensearch的访问密码
# Generating a secret and storing it with AWS Secrets Manager.
# https://aws.amazon.com/secrets-manager/
# To list secret using CLI and jq, run:
# aws secretsmanager list-secrets | jq ".SecretList[].Name"
# To retrieve a secret value using CLI and jq, run:
# aws secretsmanager get-secret-value --secret-id <secret-name>
secret_opensearch_admin_password = cdk_sm.Secret(
self, "OpenSearchDemoDomainAdminUser")
3.2.3 创建对opensearch的capacity config
# Capacity config documentation:
# https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_opensearchservice/CapacityConfig.html#aws_cdk.aws_opensearchservice.CapacityConfig
# Available instance types:
# https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-instance-types.html
capacity_config = cdk_opensearch.CapacityConfig(
master_nodes=3,
master_node_instance_type="t3.small.search",
data_nodes=3,
data_node_instance_type="t3.medium.search"
3.2.4 创建对opensearch的ebs config
# Available EBS options
# https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_opensearchservice/EbsOptions.html#aws_cdk.aws_opensearchservice.EbsOptions
ebs_config = EbsOptions(
volume_size=10,
volume_type=cdk_ec2.EbsDeviceVolumeType.GP3
)
3.2.5 设定opensearch的enable zone awareness
# Enabling zone awareness to allow data replication across AZ's.
# https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_opensearchservice/ZoneAwarenessConfig.html#aws_cdk.aws_opensearchservice.ZoneAwarenessConfig
zone_awareness_config = ZoneAwarenessConfig(
availability_zone_count=3,
enabled=True
)
3.2.6 设定encrypt setting和用户名密码
# Required when FGAC is enabled
encryption_config = EncryptionAtRestOptions(
enabled=True
)
# Required when FGAC is enabled
opensearch_admin_user = "admin-user"
advanced_security_config = AdvancedSecurityOptions(
master_user_name=opensearch_admin_user,
master_user_password=secret_opensearch_admin_password.secret_value
)
3.2.7 开始真正创建opensearch
# Required when FGAC is enabled
encryption_config = EncryptionAtRestOptions(
enabled=True
)
# Required when FGAC is enabled
opensearch_admin_user = "admin-user"
advanced_security_config = AdvancedSecurityOptions(
master_user_name=opensearch_admin_user,
master_user_password=secret_opensearch_admin_password.secret_value
)
3.2.8 将必要的output进行输出
cdk.CfnOutput(self,"OpenSearchDomainEndpoint", value=aos_domain.domain_endpoint)
cdk.CfnOutput(self,"OpenSearchDashboardsURL", value=(aos_domain.domain_endpoint + "/_dashboards"))
cdk.CfnOutput(self,"OpenSearchPasswordSecretName", value=secret_opensearch_admin_password.secret_name)
cdk.CfnOutput(self,"OpenSearchAdminUser", value=opensearch_admin_user)
4 执行cdk创建opensearch
4.1 执行cdk的注意点
这里，如果直接执行如下命令创建opensearch，会报错。
cdk --require-approval never deploy
错误信息如下：
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_IN_PROGRESS | AWS::Lambda::Function | AWS679f53fac002430cb0da5b7982bd2287 (AWS679f53fac002430cb0da5b7982bd22872D164C4C) Resource creation Initiated
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_FAILED | AWS::Lambda::Function | AWS679f53fac002430cb0da5b7982bd2287 (AWS679f53fac002430cb0da5b7982bd22872D164C4C) Resource handler returned message: "The runtime parameter of nodejs14.x is no longer supported for creating or updating AWS Lambda functions. We recommend you use a supported runtime while creating or updating functions. (Service: Lambda, Status Code: 400, Request ID: c49cc9d9-f4b1-42e4-8a98-af597a4aa3f4) (SDK Attempt Count: 1)" (RequestToken: 07111475-e581-91e9-185c-ce93ab079201, HandlerErrorCode: InvalidRequest)
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_IN_PROGRESS | AWS::Lambda::Function | DummyLambdaRuntimeSetter (DummyLambdaRuntimeSetter4B38A37F) Resource creation Initiated
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_FAILED | AWS::Lambda::Function | DummyLambdaRuntimeSetter (DummyLambdaRuntimeSetter4B38A37F) Resource creation cancelled
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_FAILED | AWS::OpenSearchService::Domain | OpensearchDemoDomain (OpensearchDemoDomainBEE1301C) Resource creation cancelled
，原因是，默认opensearch使用lambda是采用"nodejs14.x runtime，所以会报错。
4.2 执行cdk的错误回避对策(workaround)
cdk synth > template.yaml # 首先不直接部署cdk，而是将cdk输出到template.yaml
sed -i 's/nodejs14.x/nodejs18.x/g' template.yaml # 进行替换，升级到nodejs18
aws cloudformation deploy --template-file template.yaml --stack-name OpensearchSimpleDomainStack --capabilities CAPABILITY_IAM # 继续部署opensearch
4.3 创建opensearch的花费
创建了五六次opensearch，花费了$8,所以建议大家慎重测试opensearch！