[DevSecOps] Mutation testing

Look into Mutation testing.

Mutation testing is a type of software testing where certain statements in the source code are changed (or “mutated”) to determine if the test cases can detect the changes. The idea is to ensure that the test suite is robust and can catch defects in the code.

Here’s a basic overview of how mutation testing works:

1. Mutants Creation: The original source code is modified to create multiple versions called “mutants”. Each mutant has a small change, such as changing an arithmetic operator (e.g., + to -), inverting a condition (e.g., < to >=), or removing a line of code.
2. Run Tests on Mutants: The existing test suite is run against each mutant.
3. Analyze Results:
   • If a test fails for a mutant, that mutant is considered “killed”.
   • If all tests pass for a mutant, that mutant is considered “live”. This indicates a potential issue with the test suite because it was unable to detect the introduced defect.
4. Calculate Mutation Score: This is the ratio of killed mutants to the total number of mutants. A higher mutation score indicates a more effective test suite.

5. Use Results for Improvement: The results can be used to identify gaps in the test suite and improve test coverage.

Mutation testing provides insights into the quality of the test suite and can help in identifying areas where more tests might be needed. However, it can be computationally expensive because the test suite needs to be run against each mutant. As a result, it’s often used selectively or in combination with other testing techniques.

Example:

Certainly! Let’s consider a simple Go function and how we might apply mutation testing to it.

Original Code

Suppose we have a Go function that checks if a number is even:

package main

import "fmt"

func IsEven(n int) bool {
    return n%2 == 0
}

func main() {
    fmt.Println(IsEven(4))  // true
    fmt.Println(IsEven(5))  // false
}

Test for the Function

We’ll write a basic test for this function:

package main

import "testing"

func TestIsEven(t *testing.T) {
    if !IsEven(4) {
        t.Error("Expected 4 to be even")
    }
    if IsEven(5) {
        t.Error("Expected 5 to be not even")
    }
}

Mutation

Now, let’s introduce a mutation to the original code. One simple mutation would be to change the == operator to != in the IsEven function:

func IsEven(n int) bool {
    return n%2 != 0  // mutated code
}

Running the Test

When we run the test against this mutated code, the test will fail because the mutated function will return false for IsEven(4) and true for IsEven(5). This means our test has “killed” the mutant, which is a good thing.

Mutation Score

If this was the only mutation we introduced, our mutation score would be 100% because we killed 1 out of 1 mutant.

Real-world Mutation Testing in Go

In a real-world scenario, you’d use a mutation testing tool to automate the creation of mutants and the calculation of the mutation score. As of my last update in September 2021, there are tools like go-mutesting that can be used for mutation testing in Go. You’d install the tool, run it against your codebase, and it would automatically generate mutants, run tests against them, and provide a mutation score.

Tools:

1. Go-mutesting: This is a mutation testing tool for Go that you can find on GitHub. It helps in finding spots in your Go code that are not covered by your tests. GitHub: go-mutesting
2. Goblin: Although not exclusively a mutation testing tool, it is a testing framework for Go that might be used in conjunction with mutation testing efforts. GitHub: goblin

Please note that the state of tools can change, and new tools might have been developed since my last update. You might want to check for newer tools or updates to existing tools for mutation testing in Go.