OpenSearch 또는 ECK 기반의 로깅 시스템 구축 및 고도화

1. 개요

• OpenSearch: Elasticsearch의 오픈소스 포크로, Kibana 대신 OpenSearch Dashboards 사용.
• ECK (Elastic Cloud on Kubernetes): Kubernetes 환경에서 Elasticsearch를 운영할 수 있도록 Elastic에서 제공하는 오픈소스 솔루션.

---

2. OpenSearch 기반 로깅 시스템 구축

2.1 필수 패키지 설치

# 패키지 업데이트
sudo yum update -y

# Docker 설치
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.io

# Docker 실행 및 부팅 시 자동 실행 설정
sudo systemctl start docker
sudo systemctl enable docker

# Docker Compose 설치
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# 확인
docker --version
docker-compose --version

2.2 OpenSearch 및 Dashboards 실행

docker-compose.yml 생성

version: '3'
services:
  opensearch:
    image: opensearchproject/opensearch:2.11.0
    container_name: opensearch
    environment:
      - discovery.type=single-node
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - opensearch-data:/usr/share/opensearch/data
    ports:
      - "9200:9200"
      - "9600:9600"
    restart: always

  dashboards:
    image: opensearchproject/opensearch-dashboards:2.11.0
    container_name: dashboards
    environment:
      - OPENSEARCH_HOSTS=http://opensearch:9200
    ports:
      - "5601:5601"
    restart: always

volumes:
  opensearch-data:

2.3 OpenSearch 실행

docker-compose up -d

확인

---

3. ECK 기반 Elasticsearch 구축 (K8S)

3.1 Kubernetes 설치

sudo yum update -y

# kubeadm, kubelet, kubectl 설치
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
EOF



sudo yum install -y kubelet kubeadm kubectl
sudo systemctl enable --now kubelet

3.2 ECK Operator 설치

kubectl create -f https://download.elastic.co/downloads/eck/2.16.1/crds.yaml

3.3 Elasticsearch 및 Kibana 배포

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-cluster
spec:
  version: 8.3.3
  nodeSets:
  - name: default
    count: 3
    config:
      node.roles: ["master", "data", "ingest"]
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 8.3.3
  count: 1
  elasticsearchRef:
    name: elasticsearch-cluster

적용:

kubectl apply -f elasticsearch-kibana.yaml

확인:

kubectl get pods -n elastic-system

---

4. Fluentd 기반 로깅 수집

4.1 Fluentd 설치

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent4.sh | sh
sudo systemctl start td-agent
sudo systemctl enable td-agent

4.2 Fluentd 설정 (OpenSearch 연동)

sudo vi /etc/td-agent/td-agent.conf

아래 내용 추가

<source>
  @type tail
  path /var/log/messages
  pos_file /var/log/td-agent/messages.pos
  tag messages
  format syslog
</source>

<match **>
  @type opensearch
  host localhost
  port 9200
  index_name fluentd-logs
  type_name _doc
  logstash_format true
</match>

Fluentd 재시작:

sudo systemctl restart td-agent

---

5. 고도화 (보안 및 성능 최적화)

5.1 인증 활성화

opensearch:
  environment:
    - "DISABLE_SECURITY_PLUGIN=false"
    
curl -XPUT -u admin:admin 'https://localhost:9200/_cluster/settings' -d '{
  "transient": {
    "cluster.routing.allocation.disk.threshold_enabled": false
  }
}'

5.2 성능 최적화

opensearch:
  environment:
    - "OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g"

---

6. 결론

• Docker 기반 OpenSearch 구성
• ECK 기반 Kubernetes 환경에서 Elasticsearch 구축
• Fluentd를 이용한 로그 수집 및 전송
• 보안 설정 및 성능 최적화

필요한 환경을 선택하여 구축하면 됩니다!