amazon web services - Opensearch Ingestion Pipeline: reference a dynamic field - Stack Overflow

Opensearch Ingestion Pipeline reads from S3 bucket with a file:{"key1": "value1" ,

Opensearch Ingestion Pipeline reads from S3 bucket with a file:

{"key1": "value1" , "value1": "54", "key3": "value10"}
{"key1": "value3" , "value3": "65", "key3": "value9"}

Pipeline configuration

version: "2"
log-pipeline:
  source:
    s3:
      codec:
        newline: null
      compression: "none"
      aws:
        region: "eu-west-2"
        sts_role_arn: "arn:aws:iam::{account_no}:role/s3-os-ingestion-pipeline-role"
      acknowledgments: true
      scan:
        buckets:
          - bucket:
              name: "tempesdata"
      delete_s3_objects_on_read: false
  processor:
    - parse_json:
    - delete_entries:
        with_keys: ["s3", "message"]
  sink:
    - opensearch:
        hosts: [ "{host}" ]
        aws:
          sts_role_arn: "arn:aws:iam::{account_no}:role/s3-os-ingestion-pipeline-role"
          region: "eu-west-2"
          serverless: false
        index: "test"
        document_id: ${/key3} # UPDATE THIS

Problem is i want document id to be concat of key1's value's value + ${/key3}, Something like $/{${/key1}}-${/key3}. But this does not work. In the given example, I would want documents to be indexed with 54-value10, 65-value9

发布者:admin,转转请注明出处:http://www.yc00.com/questions/1745671682a4639442.html

相关推荐

发表回复

评论列表(0条)

  • 暂无评论

联系我们

400-800-8888

在线咨询: QQ交谈

邮件:admin@example.com

工作时间:周一至周五,9:30-18:30,节假日休息

关注微信