Skip to content

NimTechnology

Trình bày các công nghệ CLOUD một cách dễ hiểu.

  • Kubernetes & Container
    • Docker
    • Kubernetes
      • Ingress
      • Pod
    • Helm Chart
    • OAuth2 Proxy
    • Isito-EnvoyFilter
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Vault
    • Longhorn – Storage
    • VictoriaMetrics
    • MetalLB
    • Kong Gateway
  • CI/CD
    • ArgoCD
    • ArgoWorkflows
    • Argo Events
    • Spinnaker
    • Jenkins
    • Harbor
    • TeamCity
    • Git
      • Bitbucket
  • Coding
    • DevSecOps
    • Terraform
      • GCP – Google Cloud
      • AWS – Amazon Web Service
      • Azure Cloud
    • Golang
    • Laravel
    • Python
    • Jquery & JavaScript
    • Selenium
  • Log, Monitor & Tracing
    • DataDog
    • Prometheus
    • Grafana
    • ELK
      • Kibana
      • Logstash
  • BareMetal
    • NextCloud
  • Toggle search form

[Golang/Redis/Riot] Redis migration from old to new infrastructure.

Posted on October 10, 2024December 3, 2024 By nim No Comments on [Golang/Redis/Riot] Redis migration from old to new infrastructure.

Let’s convert the Python Redis migration code to Go, considering the go-redis library for Redis interactions. Here’s a step-by-step conversion from Python to Go.

Contents

Toggle
  • Plan:
  • Pseudocode:
  • Go Code:
  • Key Points in Go Code:
  • Dependencies:
  • Next Steps:

Plan:

  1. Connections: Use go-redis to connect to old and new Redis instances.
  2. Concurrency: Go uses goroutines, so we will replace multiprocessing.Pool with goroutines and a worker pool using Go channels.
  3. Key Migration: Use SCAN to retrieve keys in batches and DUMP/RESTORE to move data between Redis instances.
  4. TTL Handling: TTL handling logic remains the same.
  5. Graceful Termination: Use context cancellation or signal handling to terminate goroutines.

Pseudocode:

  • Establish Redis connections (old and new).
  • Create worker pool using goroutines.
  • Fetch keys in batches using SCAN.
  • For each key:
  • Get TTL.
  • Dump the key from old Redis.
  • Restore the key in new Redis with TTL.
  • Handle errors during restore.
  • Graceful shutdown with signal handling.

Go Code:

package main

import (
    "context"
    "fmt"
    "log"
    "os"
    "os/signal"
    "sync"
    "syscall"
    "time"

    "github.com/go-redis/redis/v8"
)

var (
    oldRedisHost = "SHARD_ENDPOINT_FOR_OLD_REDIS"
    newRedisHost = "SHARD_ENDPOINT_FOR_NEW_REDIS"
    rdbOld       *redis.Client
    rdbNew       *redis.Client
    ctx          = context.Background()
)

func initRedisClients() {
    rdbOld = redis.NewClient(&redis.Options{
        Addr: oldRedisHost + ":6379",
        DB:   0,
    })

    rdbNew = redis.NewClient(&redis.Options{
        Addr: newRedisHost + ":6379",
        DB:   0,
    })
}

// Migrate a single key
func migrateKey(key string, wg *sync.WaitGroup) {
    defer wg.Done()

    // Get TTL
    ttl, err := rdbOld.TTL(ctx, key).Result()
    if err != nil {
        log.Printf("Failed to get TTL for key %s: %v", key, err)
        return
    }
    // If TTL is -1 (no expiration), set TTL to 0
    if ttl < 0 {
        ttl = 0
    }

    // Dump the key from old Redis
    value, err := rdbOld.Dump(ctx, key).Result()
    if err != nil {
        log.Printf("Failed to dump key %s: %v", key, err)
        return
    }

    // Restore the key in new Redis
    err = rdbNew.RestoreReplace(ctx, key, ttl, value).Err()
    if err != nil {
        log.Printf("Failed to restore key %s: %v", key, err)
    }
}

func workerPool(keys <-chan string, wg *sync.WaitGroup) {
    for key := range keys {
        migrateKey(key, wg)
    }
}

// Graceful shutdown function to handle SIGINT (Ctrl+C)
func handleGracefulShutdown(cancel context.CancelFunc, wg *sync.WaitGroup) {
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

    // Block until a signal is received
    <-sigChan
    log.Println("Caught interrupt signal, terminating workers...")
    cancel()  // Trigger context cancellation
    wg.Wait() // Wait for workers to finish
}

func main() {
    // Initialize Redis clients
    initRedisClients()

    // Create a context that we can cancel
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    // Channel to send keys to worker pool
    keysChan := make(chan string)
    var wg sync.WaitGroup

    // Start worker pool (with 10 workers)
    numWorkers := 10
    for i := 0; i < numWorkers; i++ {
        go workerPool(keysChan, &wg)
    }

    // Handle graceful shutdown
    go handleGracefulShutdown(cancel, &wg)

    // SCAN keys in Redis and send them to workers
    var cursor uint64
    for {
        select {
        case <-ctx.Done():
            // Context is cancelled, exit loop
            close(keysChan)
            return
        default:
            // Scan 1000 keys at a time
            keys, nextCursor, err := rdbOld.Scan(ctx, cursor, "", 1000).Result()
            if err != nil {
                log.Printf("Failed to scan keys: %v", err)
                return
            }
            cursor = nextCursor

            // Send keys to worker pool
            for _, key := range keys {
                wg.Add(1)
                keysChan <- key
            }

            // Exit loop when scan is complete
            if cursor == 0 {
                break
            }
        }
    }

    // Wait for all workers to finish
    wg.Wait()
    close(keysChan)

    log.Println("Migration completed successfully.")
}

Key Points in Go Code:

  1. Redis Connections: We establish two connections using go-redis, one for the old Redis (rdbOld) and one for the new Redis (rdbNew).
  2. Worker Pool: We use goroutines to create a pool of workers. Each worker listens on a channel for keys to process.
  3. Key Migration:
  • For each key, we fetch the TTL and use Dump to get the key’s value.
  • We then Restore the key in the new Redis with the appropriate TTL.
  1. Graceful Shutdown: We handle SIGINT and SIGTERM signals to ensure the workers stop gracefully when interrupted (e.g., via Ctrl+C).
  2. Concurrency: We use a sync.WaitGroup to ensure that the main thread waits for all goroutines to complete their tasks.

Dependencies:

You will need the following dependency:

go get github.com/go-redis/redis/v8

Next Steps:

a. Would you like to test the migration in a controlled environment or add additional logging for tracking migration progress?
b. Should we add version compatibility checks between the old and new Redis versions to avoid ERR DUMP payload version or checksum errors?

Besides you can use Riot to migrate Redis’s data and It worked Elasticache of AWS:
https://redis.github.io/riot/#_install

riot replicate redis-nim-staging.yp7b3n.ng.0001.usw2.cache.amazonaws.com:6379 redis-nim-staging-test-replicate-0001-001.yp7b3n.0001.usw2.cache.amazonaws.com:6379 --target-cluster --mode live --info --progress log --show-diffs --threads 100 --batch 1 --struct
AWS - Amazon Web Service, Golang

Post navigation

Previous Post: [AWS Fault Injection Service] Crash Simulation on AWS
Next Post: An AWS IAM Security Tooling Reference [2024]

More Related Articles

[RabbitMQ] Queue announces “cluster is in minority” AWS - Amazon Web Service
[S3] You will naturally see files on S3 being deleted AWS - Amazon Web Service
[Kubernetes] the exciting things about K8S AWS - Amazon Web Service
[Metrics Server] Install metrics-server on Kubernetes. AWS - Amazon Web Service
[CoreDNS] How to improve the Coredns performance. AWS - Amazon Web Service
[AWS] Login Argocd via Cognito in AWS ArgoCD

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Tham Gia Group DevOps nhé!
Để Nim có nhiều động lực ra nhiều bài viết.
Để nhận được những thông báo mới nhất.

Recent Posts

  • [WordPress] Hướng dấn gửi mail trên WordPress thông qua gmail. June 15, 2025
  • [Bitbucket] Git Clone/Pull/Push with Bitbucket through API Token. June 12, 2025
  • [Teamcity] How to transfer the value from pipeline A to pipeline B June 9, 2025
  • [Windows] Remove the process that consumes too much CPU. June 3, 2025
  • Deploying Web-Based File Managers: File Browser and KubeFileBrowser with Docker and Kubernetes June 3, 2025

Archives

  • June 2025
  • May 2025
  • April 2025
  • March 2025
  • February 2025
  • January 2025
  • December 2024
  • November 2024
  • October 2024
  • September 2024
  • August 2024
  • July 2024
  • June 2024
  • May 2024
  • April 2024
  • March 2024
  • February 2024
  • January 2024
  • December 2023
  • November 2023
  • October 2023
  • September 2023
  • August 2023
  • July 2023
  • June 2023
  • May 2023
  • April 2023
  • March 2023
  • February 2023
  • January 2023
  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • December 2021
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • July 2021
  • June 2021

Categories

  • BareMetal
    • NextCloud
  • CI/CD
    • Argo Events
    • ArgoCD
    • ArgoWorkflows
    • Git
      • Bitbucket
    • Harbor
    • Jenkins
    • Spinnaker
    • TeamCity
  • Coding
    • DevSecOps
    • Golang
    • Jquery & JavaScript
    • Laravel
    • NextJS 14 & ReactJS & Type Script
    • Python
    • Selenium
    • Terraform
      • AWS – Amazon Web Service
      • Azure Cloud
      • GCP – Google Cloud
  • Kubernetes & Container
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Docker
    • Helm Chart
    • Isito-EnvoyFilter
    • Kong Gateway
    • Kubernetes
      • Ingress
      • Pod
    • Longhorn – Storage
    • MetalLB
    • OAuth2 Proxy
    • Vault
    • VictoriaMetrics
  • Log, Monitor & Tracing
    • DataDog
    • ELK
      • Kibana
      • Logstash
    • Fluent
    • Grafana
    • Prometheus
  • Uncategorized
  • Admin

Copyright © 2025 NimTechnology.