[Golang] Approach Channel in Calling another function with Golang

Bài toán toán là function A chạy 1 vòng For lấy ra từng giá trị nhỏ của mảng và send chúng đến function sử lý và nhận lại kết quả:

Nếu mảng của bạn có vài ngàn giá trị nhỏ thì việc gởi và sử lý tuần tự rất là mất thời gian.
nếu mỗi process tốn 1s nếu có 24000 processes thì bạn sẽ tốn 24000s = thì bạn tốn 6 giờ để sử lý

Lúc này bạn sẽ cần apply channel và goroutines để tăng Performance

Contents

1) Apply channels and goroutines for the For Loop to get results From another Function faster.

To track which file returned a true or false result, you need to modify your approach to include the file information along with the result. One way to do this is by using a struct to hold both the file path and the result. Here’s how you can implement it:

Define a Result Struct: Create a struct that holds both the file path and the result (a boolean).
Modify the Channel to Use the Struct: Change your results channel to carry the struct.
Adjust Goroutines and Result Processing Accordingly: Ensure that each goroutine sends this struct through the channel, and adjust the result processing loop to handle this struct.

Here’s the modified implementation:

First, define the struct:

type FileResult struct {
    FilePath string
    IsOld    bool
}

Then modify the HandlerDeleteFile function:

func (dl *DeleteFileHandler) HandlerDeleteFile() {
    rootPath := helper.GetEnvOrDefault("DOOM_PATH", "test_delete")
    minutesStr := helper.AgeToMs(helper.GetEnvOrDefault("RULE_AGE", "60m"))

    var cmd *exec.Cmd
    // [omitted: command initialization]

    output, err := cmd.StdoutPipe()
    // [omitted: error handling]

    if err := cmd.Start(); err != nil {
        fmt.Println("Error starting command:", err)
    }

    results := make(chan FileResult)
    var wg sync.WaitGroup

    scanner := bufio.NewScanner(output)
    for scanner.Scan() {
        line := scanner.Text()
        wg.Add(1)

        go func(filePath string) {
            defer wg.Done()
            isOld, err := checkOlFile(filePath, minutesStr)
            if err != nil {
                fmt.Println("Error checking file:", err)
                return
            }
            results <- FileResult{FilePath: filePath, IsOld: isOld}
        }(line)
    }

    go func() {
        wg.Wait()
        close(results)
    }()

    for result := range results {
        if result.IsOld {
            fmt.Printf("File %s is older than threshold\n", result.FilePath)
        } else {
            fmt.Printf("File %s is not older than threshold\n", result.FilePath)
        }
    }
}

In this revised code:

Each goroutine sends a FileResult struct to the results channel.
The struct contains the file path and a boolean indicating whether the file is older than the threshold.
The result processing loop reads from the results channel and prints the file path along with its status.

2) Provide the particular Goroutines to process.

To further improve the performance of your DeleteFileHandler in Go, especially in a scenario with a large number of files, we can consider the following enhancements:

Limit the Number of Concurrent Goroutines: Running too many goroutines simultaneously can lead to high memory usage and potentially decrease performance. Implement a worker pool pattern to limit the number of concurrent goroutines.
Buffered Channels: Use buffered channels to reduce the blocking on channel operations. This can help in cases where the rate of producing results is faster than the rate of processing them.
Efficient Error Handling: Instead of printing errors within goroutines, consider passing them back to the main goroutine for handling. This approach centralizes error handling and can make debugging easier.
Avoid Global Panic: Replace panic with error handling to avoid crashing the entire application on a single file error.

Here’s an updated implementation considering these points:

func (dl *DeleteFileHandler) HandlerDeleteFile() {
    rootPath := helper.GetEnvOrDefault("DOOM_PATH", "test_delete")
    minutesStr := helper.AgeToMs(helper.GetEnvOrDefault("RULE_AGE", "60m"))

    var cmd *exec.Cmd
    // [omitted: command initialization]

    output, err := cmd.StdoutPipe()
    // [omitted: error handling]

    if err := cmd.Start(); err != nil {
        fmt.Println("Error starting command:", err)
        return
    }

    // Buffered channels
    results := make(chan bool, 100) // Adjust buffer size as needed
    errors := make(chan error, 100)

    var wg sync.WaitGroup

    // Worker pool limit
    maxGoroutines := 10 // Adjust the number of workers as needed
    semaphore := make(chan struct{}, maxGoroutines)

    scanner := bufio.NewScanner(output)
    for scanner.Scan() {
        line := scanner.Text()

        wg.Add(1)
        semaphore <- struct{}{} // Acquire a token

        go func(filePath string) {
            defer wg.Done()
            defer func() { <-semaphore }() // Release the token

            result, err := checkOlFile(filePath, minutesStr)
            if err != nil {
                errors <- err
                return
            }
            results <- result
        }(line)
    }

    // Close channels when all goroutines are done
    go func() {
        wg.Wait()
        close(results)
        close(errors)
    }()

    // Process results and errors
    for {
        select {
        case result, ok := <-results:
            if !ok {
                results = nil
            } else if result {
                fmt.Println("File is older than threshold")
            }
        case err, ok := <-errors:
            if !ok {
                errors = nil
            } else {
                fmt.Println("Error checking file:", err)
            }
        }
        if results == nil && errors == nil {
            break
        }
    }
}

In this implementation:

A semaphore (semaphore channel) limits the number of concurrent goroutines.
Buffered channels results and errors are used to store the results and errors respectively.
The main goroutine waits for both results and errors channels to be closed before exiting.
A separate goroutine closes results and errors channels once all processing is done.