Argo Events로 Argo Rollouts 롤백을 감지해 Jira 인시던트와 Confluence 포스트모템을 자동 생성하는 방법

Argo Rollouts 롤백 감지
  → Notifications Engine이 Webhook 발송
  → Argo Events Webhook EventSource가 수신
  → EventBus(NATS JetStream)를 통해 Sensor에 전달
  → Sensor의 HTTP Trigger로 Jira REST API 호출 → 인시던트 이슈 생성
  → 동시에 Confluence REST API 호출 → 포스트모템 페이지 초기화

metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-rollout-degraded.webhook: ""

# argo-rollouts-notification-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-rollouts-notification-cm
  namespace: argo-rollouts
data:
  service.webhook.argo-events: |
    url: http://argo-events-webhook-eventsource-svc.argo-events.svc.cluster.local:12000/rollout
    headers:
      - name: Content-Type
        value: application/json
    retryMax: 3
    retryWaitMin: 5s
    retryWaitMax: 30s
 
  template.rollback-alert: |
    webhook:
      argo-events:
        method: POST
        body: |
          {
            "rolloutName": "{{.rollout.metadata.name}}",
            "namespace": "{{.rollout.metadata.namespace}}",
            "phase": "{{.rollout.status.phase}}",
            "message": "{{.rollout.status.message}}",
            "timestamp": "{{now | date \"2006-01-02T15:04:05Z07:00\"}}",
            "jiraProject": "OPS",
            "jiraIssueType": "Incident"
          }
 
  trigger.on-rollout-degraded: |
    - send: [rollback-alert]

# rollout-webhook-eventsource.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: rollout-webhook-eventsource
  namespace: argo-events
spec:
  service:
    ports:
      - port: 12000
        targetPort: 12000
  webhook:
    rollout:
      port: "12000"
      endpoint: /rollout
      method: POST

# rollout-resource-eventsource.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: rollout-watcher
  namespace: argo-events
spec:
  resource:
    rollout-degraded:
      group: argoproj.io
      version: v1alpha1
      resource: rollouts
      namespace: production
      eventTypes:
        - MODIFIED

# eventbus.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
  name: default
  namespace: argo-events
spec:
  nats:
    native:
      replicas: 3
      auth: token

# rollback-sensor.yaml
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: rollback-incident-sensor
  namespace: argo-events
spec:
  dependencies:
    - name: rollout-dep
      eventSourceName: rollout-webhook-eventsource  # 방식 B라면 rollout-watcher
      eventName: rollout
      filters:
        data:
          - path: body.phase
            type: string
            value:
              - "Degraded"
  triggers:
    # 트리거 1: Jira 인시던트 생성
    - template:
        name: create-jira-incident
        http:
          url: "https://$(JIRA_DOMAIN)/rest/api/3/issue"
          method: POST
          headers:
            - name: Authorization
              valueFrom:
                secretKeyRef:
                  name: atlassian-credentials
                  key: basic-auth
            - name: Content-Type
              value: application/json
          # payload: 이벤트 데이터를 request body의 특정 경로에 복사
          payload:
            - src:
                dependencyName: rollout-dep
                dataKey: body.jiraProject
              dest: body.fields.project.key
            - src:
                dependencyName: rollout-dep
                dataKey: body.jiraIssueType
              dest: body.fields.issuetype.name
            - src:
                dependencyName: rollout-dep
                dataKey: body.rolloutName
              dest: body.fields.summary
            - src:
                dependencyName: rollout-dep
                dataKey: body.namespace
              dest: body.fields.description.content[0].content[0].text
          # parameters: payload로 구성된 body 값을 prepend/append/override로 수정
          # payload와 달리 연산 변환이 가능하며, 같은 dest에 함께 쓰면 payload 적용 후 parameters가 덮습니다
          parameters:
            - src:
                dependencyName: rollout-dep
                dataKey: body.rolloutName
              dest: body.fields.summary
              operation: prepend
              value: "[INCIDENT] Rollback detected: "
        retryStrategy:
          steps: 3
          duration: 10s
 
    # 트리거 2: Confluence 포스트모템 페이지 초기화
    - template:
        name: create-confluence-postmortem
        http:
          url: "https://$(CONFLUENCE_DOMAIN)/wiki/rest/api/content"
          method: POST
          headers:
            - name: Authorization
              valueFrom:
                secretKeyRef:
                  name: atlassian-credentials
                  key: basic-auth
            - name: Content-Type
              value: application/json
          payload:
            - src:
                dependencyName: rollout-dep
                dataKey: body.rolloutName
              dest: body.title
          parameters:
            - src:
                dependencyName: rollout-dep
                dataKey: body.rolloutName
              dest: body.title
              operation: prepend
              value: "Postmortem — Rollback: "
        retryStrategy:
          steps: 3
          duration: 10s

# atlassian-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: atlassian-config
  namespace: argo-events
data:
  JIRA_DOMAIN: "your-team.atlassian.net"
  CONFLUENCE_DOMAIN: "your-team.atlassian.net"

{
  "type": "page",
  "title": "Postmortem — Rollback: my-service [2026-05-26]",
  "space": { "key": "INCIDENTS" },
  "ancestors": [{ "id": "123456" }],
  "body": {
    "storage": {
      "value": "<h2>인시던트 개요</h2><table><tr><th>항목</th><th>내용</th></tr><tr><td>서비스</td><td>{{rolloutName}}</td></tr><tr><td>네임스페이스</td><td>{{namespace}}</td></tr><tr><td>발생 시각</td><td>{{timestamp}}</td></tr><tr><td>연관 Jira</td><td>{{jiraIssueKey}}</td></tr></table><h2>타임라인</h2><p>(작성 필요)</p><h2>근본 원인</h2><p>(작성 필요)</p><h2>재발 방지 액션 아이템</h2><p>(작성 필요)</p>",
      "representation": "storage"
    }
  }
}

# atlassian-external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: atlassian-credentials
  namespace: argo-events
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: atlassian-credentials
  data:
    - secretKey: basic-auth
      remoteRef:
        key: prod/atlassian
        property: basic-auth-token