Skip to content

NimTechnology

Trình bày các công nghệ CLOUD một cách dễ hiểu.

  • Kubernetes & Container
    • Docker
    • Kubernetes
      • Ingress
      • Pod
    • Helm Chart
    • OAuth2 Proxy
    • Isito-EnvoyFilter
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Vault
    • Longhorn – Storage
    • VictoriaMetrics
    • MetalLB
    • Kong Gateway
  • CI/CD
    • ArgoCD
    • ArgoWorkflows
    • Argo Events
    • Spinnaker
    • Jenkins
    • Harbor
    • TeamCity
    • Git
      • Bitbucket
  • Coding
    • DevSecOps
    • Terraform
      • GCP – Google Cloud
      • AWS – Amazon Web Service
      • Azure Cloud
    • Golang
    • Laravel
    • Python
    • Jquery & JavaScript
    • Selenium
  • Log, Monitor & Tracing
    • DataDog
    • Prometheus
    • Grafana
    • ELK
      • Kibana
      • Logstash
  • BareMetal
    • NextCloud
  • Toggle search form

[Golang] List large files in the folder quickly on Golang

Posted on September 29, 2023September 30, 2023 By nim No Comments on [Golang] List large files in the folder quickly on Golang

Certainly! Using tools like find on Unix-based systems can be a more efficient way to list files, especially for very large directories. Here’s a guide on how to integrate the find command with a Go program:

Contents

Toggle
    • Step-by-Step Guide:
    • Example:
    • Notes:
  • find the old file

Step-by-Step Guide:

  1. Using the find Command:
    The find command is a powerful tool for searching files and directories. For instance, to list all files in a directory and its subdirectories, you can use:
   find /path/to/directory -type f
  1. Executing find from Go:
    You can use the os/exec package in Go to execute the find command and capture its output.
  2. Processing the Output in Go:
    Once you have the output from the find command, you can process it in Go as needed.

Example:

Here’s a simple Go program that uses the find command to list all files in a directory and then processes the output:

package main

import (
    "bufio"
    "fmt"
    "os"
    "os/exec"
)

func main() {
    // Define the directory to search
    dir := "/path/to/directory"

    // Execute the find command
	var cmd *exec.Cmd
	// Check the operating system and execute the appropriate command
	if runtime.GOOS == "windows" {
		cmd = exec.Command("cmd", "/c", "dir", "/b", "/s", rootPath)
	} else {
		cmd = exec.Command("find", rootPath, "-type", "f")
	}

    output, err := cmd.StdoutPipe()
    if err != nil {
        fmt.Println("Error creating stdout pipe:", err)
        return
    }

	stderr, err := cmd.StderrPipe()
	if err != nil {
		fmt.Println("Error creating stderr pipe:", err)
	}

    if err := cmd.Start(); err != nil {
        fmt.Println("Error starting command:", err)
        return
    }


    // Read and process the output
    scanner := bufio.NewScanner(output)
    for scanner.Scan() {
        line := scanner.Text()
        // Process each file path (in this case, just print it)
        fmt.Println(line)
    }

    if err := scanner.Err(); err != nil {
        fmt.Println("Error reading command output:", err)
        return
    }

    if err := cmd.Wait(); err != nil {
        fmt.Println("Error waiting for command to finish:", err)
        return
    }

	// Print the number of files
	fmt.Println("Number of files:", fileCount)
}

The line output, err := cmd.StdoutPipe() is used to create a pipe that allows you to capture the standard output (stdout) of a command executed by the cmd variable
-> output is typically an io.Reader interface, which allows you to read data from the pipe.

The line stderr, err := cmd.StderrPipe() is used to create a pipe that allows you to capture the standard error (stderr) output of a command executed by the cmd variable.

err := cmd.Start() is used to start the execution of an external command specified by the cmd variable and check if any error occurs during the start process

scanner := bufio.NewScanner(output): This line creates a new Scanner instance called scanner. The Scanner is part of the bufio package in Go and is used to read text data, line by line, from an input source. In this case, output is being used as the input source, which is the standard output pipe created earlier.

cmd.Wait(): This line of code invokes the Wait() method on the cmd instance. The Wait() method blocks the execution of your Go program until the external command has completed its execution. It ensures that your program doesn’t proceed further until the command has finished.

Notes:

  • This approach offloads the file listing task to the find command, which is optimized for this purpose.
  • The Go program captures the output of the find command and processes it line by line using a bufio.Scanner.
  • You can replace the fmt.Println(line) statement with any processing logic you need.
  • Make sure to handle errors appropriately, as shown in the example.
  • This approach is particularly useful when dealing with very large directories, as find can be faster than Go’s filepath.WalkDir for certain operations.

Remember that this approach is platform-dependent and will work on Unix-based systems that have the find command available. If you need a cross-platform solution, you’ll need to consider other methods.

find the old file

Nếu bạn có nhu cầu tìm file trên xxx phút thì tham khảo thêm cách dưới.

        minutes := 120  // replace with your desired value
	
	// Convert minutes to string and prepend with "+"
	minutesStr := "+" + strconv.Itoa(minutes)
	
	cmd := exec.Command("find", rootPath, "-type", "f", "-mmin", minutesStr)
Golang, Uncategorized

Post navigation

Previous Post: [Terraform / EKS] Build EKS and Karpenter by Terraform.
Next Post: [Docker] HOW TO INSTALL DOCKER ON WINDOWS SERVER 2022

More Related Articles

[GRPC/Golang] Learning GRPC and Golang through the easy and actual lessons. Golang
[Golang] Note mấy thứ hay ho về golang Coding
[Golang] Looking filepath.Walk to return any file and children folder in the folder Golang
[VScode] Hướng dẫn điều chỉnh font cho terminal Uncategorized
[Jenkins] Lưu ý nho nhỏ trong jenkins Jenkins
[Jenkins] Lesson 12: skipStagesAfterUnstable with Jenkins Pipeline Jenkins

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Tham Gia Group DevOps nhé!
Để Nim có nhiều động lực ra nhiều bài viết.
Để nhận được những thông báo mới nhất.

Recent Posts

  • [Laravel] Laravel Helpful June 26, 2025
  • [VScode] Hướng dẫn điều chỉnh font cho terminal June 20, 2025
  • [WordPress] Hướng dấn gửi mail trên WordPress thông qua gmail. June 15, 2025
  • [Bitbucket] Git Clone/Pull/Push with Bitbucket through API Token. June 12, 2025
  • [Teamcity] How to transfer the value from pipeline A to pipeline B June 9, 2025

Archives

  • June 2025
  • May 2025
  • April 2025
  • March 2025
  • February 2025
  • January 2025
  • December 2024
  • November 2024
  • October 2024
  • September 2024
  • August 2024
  • July 2024
  • June 2024
  • May 2024
  • April 2024
  • March 2024
  • February 2024
  • January 2024
  • December 2023
  • November 2023
  • October 2023
  • September 2023
  • August 2023
  • July 2023
  • June 2023
  • May 2023
  • April 2023
  • March 2023
  • February 2023
  • January 2023
  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • December 2021
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • July 2021
  • June 2021

Categories

  • BareMetal
    • NextCloud
  • CI/CD
    • Argo Events
    • ArgoCD
    • ArgoWorkflows
    • Git
      • Bitbucket
    • Harbor
    • Jenkins
    • Spinnaker
    • TeamCity
  • Coding
    • DevSecOps
    • Golang
    • Jquery & JavaScript
    • Laravel
    • NextJS 14 & ReactJS & Type Script
    • Python
    • Selenium
    • Terraform
      • AWS – Amazon Web Service
      • Azure Cloud
      • GCP – Google Cloud
  • Kubernetes & Container
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Docker
    • Helm Chart
    • Isito-EnvoyFilter
    • Kong Gateway
    • Kubernetes
      • Ingress
      • Pod
    • Longhorn – Storage
    • MetalLB
    • OAuth2 Proxy
    • Vault
    • VictoriaMetrics
  • Log, Monitor & Tracing
    • DataDog
    • ELK
      • Kibana
      • Logstash
    • Fluent
    • Grafana
    • Prometheus
  • Uncategorized
  • Admin

Copyright © 2025 NimTechnology.