Test Runner in Cucumber using TestNG

  

What is a Test Runner Class in Cucumber?

A Test Runner class in Cucumber is used to execute feature files and link them with step definition code. It acts as a bridge between:

• Feature files (written in Gherkin syntax) and

• Step Definition classes (Java methods mapped to steps).

In the context of Cucumber with TestNG, the Test Runner allows us to run scenarios just like regular TestNG test cases, and we get benefits like:

• Parallel execution,

• TestNG XML suite configuration,

• Integration with TestNG annotations (@BeforeClass, @AfterClass etc.).

Key Responsibilities of a Test Runner Class

1. Specify the location of feature files

2. Specify the location of step definitions

3. Configure other Cucumber options like plugins, tags, dryRun, etc.

4. Use @CucumberOptions to customize execution

5. Extend AbstractTestNGCucumberTests when using TestNG

Implementing Test Runner Class Using TestNG

Folder Structure Example:

project-root/

├── src/test/java/

│   ├── features/

│   │   └── login.feature

│   ├── stepDefinitions/

│   │   └── LoginSteps.java

│   └── runner/

│       └── TestRunner.java

Below are steps to Implement Test Runner in Cucumber using Testng 

1. Add Maven dependencies in pom.xml:

<dependencies>
    <!-- Cucumber dependencies -->
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-java</artifactId>
        <version>7.15.0</version>
    </dependency>

    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-testng</artifactId>
        <version>7.15.0</version>
    </dependency>

    <!-- TestNG dependency -->
    <dependency>
        <groupId>org.testng</groupId>
        <artifactId>testng</artifactId>
        <version>7.10.2</version>
        <scope>test</scope>
    </dependency>
</dependencies>

2. Write the Feature File (login.feature)

Feature: Login Functionality

  Scenario: Successful login with valid credentials
    Given user is on login page
    When user enters valid username and password
    Then user should be redirected to the homepage

3. Write Step Definitions (LoginSteps.java)

package stepDefinitions;

import io.cucumber.java.en.*;

public class LoginSteps {
    
    @Given("user is on login page")
    public void user_is_on_login_page() {
        System.out.println("User is on login page");
    }

    @When("user enters valid username and password")
    public void user_enters_valid_credentials() {
        System.out.println("User entered username and password");
    }

    @Then("user should be redirected to the homepage")
    public void user_should_see_homepage() {
        System.out.println("User is on homepage");
    }
}

4. Create Test Runner Class (TestRunner.java)

package runner;

import io.cucumber.testng.AbstractTestNGCucumberTests;
import io.cucumber.testng.CucumberOptions;

@CucumberOptions(
    features = "src/test/java/features",
    glue = "stepDefinitions",
    plugin = {
        "pretty",
        "html:target/cucumber-reports.html",
        "json:target/cucumber.json"
    },
    monochrome = true
)
public class TestRunner extends AbstractTestNGCucumberTests {
}

How to Run It?

Option 1: Run TestRunner.java directly from your IDE (Eclipse/IntelliJ)

Option 2: Add TestNG XML to run from command line or CI

<suite name="Cucumber Test Suite">
  <test name="Cucumber Scenarios">
    <classes>
      <class name="runner.TestRunner" />
    </classes>
  </test>
</suite>

Advantages of Using TestNG with Cucumber:

• Easily manage test suites using testng.xml

• Group and prioritize tests

• Run parallel tests using TestNG support

• Better integration with reporting tools and Jenkins

Important Points:

Property	Description
features	Path to .feature files
glue	Package for step definitions and hooks
plugin	Reporting plugins (e.g., HTML, JSON, pretty)
monochrome	Display readable console output
dryRun	true to check if step definitions exist for each step, without executing

How to use @Factory Annotation in TestNG

In TestNG, the @Factory annotation is used to create instances of test classes dynamically at runtime, particularly when you want to run the same test class multiple times with different sets of data or configurations .

Purpose of @Factory:

• Allows creation of multiple instances of a test class.

• Enables running the same tests with different input values.

• Useful for data-driven testing.

Key Characteristics:

• @Factory is applied on a method, not a class.

• The factory method should return an array of Object, where each object is an instance of a test class.

• Can be combined with constructors to pass parameters into the test class.

Example: Using @Factory in TestNG

Let's say you want to run the same test with different browser names:

Step 1: Test Class

import org.testng.annotations.Test;

public class BrowserTest {

    private String browser;

    // Constructor to accept the browser name
    public BrowserTest(String browser) {
        this.browser = browser;
    }

    @Test
    public void openBrowserTest() {
        System.out.println("Running test on: " + browser);
    }
}

Step 2: Factory Class

import org.testng.annotations.Factory;

public class BrowserTestFactory {

    @Factory
    public Object[] createInstances() {
        return new Object[] {
            new BrowserTest("Chrome"),
            new BrowserTest("Firefox"),
            new BrowserTest("Edge")
        };
    }
}

Step 3: testng.xml

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="Browser Suite">
    <test name="Browser Test Factory">
        <classes>
            <class name="BrowserTestFactory" />
        </classes>
    </test>
</suite>

Output:

Running test on: Chrome
Running test on: Firefox
Running test on: Edge

When to Use @Factory:

• You want to test the same functionality with different data.

• You prefer passing parameters via constructors rather than using @DataProvider.

• You need complex test class instantiation logic.

Difference Between @Factory and @DataProvider:

Feature	@Factory	@DataProvider
Usage	Instantiates test classes	Feeds data to test methods
Applied on	Method	Method
Returns	Object[] (instances)	Object[][] (data sets)
Constructor usage	Yes	No (generally, parameters in test methods)

Expected Exceptions in TestNG

  

What is Expected Exceptions in TestNG?

In TestNG, expected exceptions are used to verify that a method throws a specific exception during execution. This is useful when you're testing error-handling logic in your application.

If the specified exception is thrown, the test passes. If not (either no exception or a different one), the test fails.

Syntax:

You use expectedExceptions inside the @Test annotation:

@Test(expectedExceptions = ExceptionType.class)
public void testMethod() {
    // code that should throw ExceptionType
}

Example:

Let’s say you want to test whether a method throws an ArithmeticException when dividing by zero:

import org.testng.annotations.Test;

public class ExpectedExceptionExample {

    @Test(expectedExceptions = ArithmeticException.class)
    public void testDivideByZero() {
        int result = 10 / 0; // This will throw ArithmeticException
    }

    @Test
    public void testNoException() {
        int result = 10 / 2;
        System.out.println("Result: " + result); // This will not throw an exception
    }
}

Explanation:

• testDivideByZero() is expected to throw ArithmeticException. Since dividing by 0 does throw it, the test passes.

• testNoException() runs normally and also passes, as it's not expecting any exception.

Expected exceptions are useful for negative testing, such as:

• Handling of null inputs

• Invalid configurations

• Divisions by zero

• Custom business exceptions

Invocation Count in TestNG

  

What is invocationCount in TestNG?

In TestNG, the invocationCount attribute is used to execute a test method multiple times.

Explanation:

• invocationCount is an attribute of the @Test annotation.

• When set, it tells TestNG to run the same test method multiple times.

• This is useful for:

• Repeating tests for reliability (e.g., flaky tests).

• Load or stress testing a piece of code.

Syntax:

@Test(invocationCount = n)

Where n is the number of times the method should run.

Java Code example:

import org.testng.annotations.Test;

public class InvocationCountExample {

    @Test(invocationCount = 5)
    public void repeatedTest() {
        System.out.println("Running test method - Thread ID: " + Thread.currentThread().getId());
    }
}

Output:

Running test method - Thread ID: 1
Running test method - Thread ID: 1
Running test method - Thread ID: 1
Running test method - Thread ID: 1
Running test method - Thread ID: 1

You’ll see the method executed 5 times.

Advanced Usage with Parallel Execution (Optional):

If you want to execute these invocations in parallel, you can combine it with threadPoolSize.

@Test(invocationCount = 5, threadPoolSize = 3)
public void repeatedTestParallel() {
    System.out.println("Running test - Thread: " + Thread.currentThread().getId());
}

This will run 5 times using 3 threads in parallel.

Important Points:

Attribute	Description
invocationCount	Number of times to run the test method
threadPoolSize	Number of threads to run invocations concurrently

TestNG vs JUnit

 

Below is the detailed comparison between TestNG and JUnit, followed by Java code examples to demonstrate the key differences:

TestNG vs JUnit – Key Differences

Feature	TestNG	JUnit (JUnit 4/5)
Framework	Open Source Java-based Testing framework	Open Source Testing framework
Annotations	@BeforeSuite, @AfterClass, @Test, etc.	@Before, @After, @Test, etc.
Dependency Testing	Supports method dependencies using dependsOnMethods	Not directly supported
Supported Testing	Unit Testing, Functional Testing, Integration Testing, end-to-end Testing, etc.	Unit Testing
Suite Execution	Allows XML-based suite execution	Uses test runners like @RunWith
Parallel Execution	Built-in support with XML config	Requires third-party tools
Order of Tests	Supports ordering of test methods via a priority attribute	Does not support
Groups	Supports grouping of test	No built-in group support
Reports	Rich HTML/XML reports	Basic reports unless extended
Popular In	Selenium & Test Automation	General Unit Testing

TestNG Example

import org.testng.annotations.*;

public class TestNGExample {

    @BeforeClass
    public void setup() {
        System.out.println("TestNG: Setup before class");
    }

    @Test(priority = 1)
    public void loginTest() {
        System.out.println("TestNG: Executing login test");
    }

    @Test(priority = 2, dependsOnMethods = {"loginTest"})
    public void dashboardTest() {
        System.out.println("TestNG: Executing dashboard test");
    }

    @AfterClass
    public void teardown() {
        System.out.println("TestNG: Teardown after class");
    }
}

JUnit Example (JUnit 4)

import org.junit.*;

public class JUnitExample {

    @BeforeClass
    public static void setup() {
        System.out.println("JUnit: Setup before class");
    }

    @Test
    public void loginTest() {
        System.out.println("JUnit: Executing login test");
    }

    @Test
    public void dashboardTest() {
        System.out.println("JUnit: Executing dashboard test");
    }

    @AfterClass
    public static void teardown() {
        System.out.println("JUnit: Teardown after class");
    }
}

Note: JUnit doesn't support method dependency like TestNG.

Important Points:

• Use TestNG for: complex test suites, method dependencies, grouping, parameterization, and Selenium testing.

• Use JUnit for: standard unit testing, integration with Java frameworks, and simplicity.

Retry Failed Test Cases in TestNG

  

In TestNG, you can retry failed test cases automatically by implementing the IRetryAnalyzer interface. This is useful when you want to re-execute flaky or intermittently failing tests a certain number of times before marking them as failed.

Steps to Retry Failed Test Cases in TestNG

• Create a Retry Analyzer Class:

  • Implement the IRetryAnalyzer interface.

  • Override the retry() method which returns true if TestNG should retry the test.

• Attach Retry Analyzer to Test:

• You can attach it directly using @Test(retryAnalyzer = ...),
  or apply it globally using an IAnnotationTransformer.

Example 1: Basic Retry Analyzer with Retry Count = 3

RetryAnalyzer.java

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer {

    private int retryCount = 0;
    private static final int maxRetryCount = 3;

    @Override
    public boolean retry(ITestResult result) {
        if (retryCount < maxRetryCount) {
            System.out.println("Retrying test: " + result.getName() + " | Attempt: " + (retryCount + 1));
            retryCount++;
            return true;
        }
        return false;
    }
}

Example Test Class: TestRetryExample.java

import org.testng.Assert;
import org.testng.annotations.Test;

public class TestRetryExample {

    int attempt = 1;

    @Test(retryAnalyzer = RetryAnalyzer.class)
    public void testMethod() {
        System.out.println("Running test attempt: " + attempt);
        if (attempt < 3) {
            attempt++;
            Assert.fail("Failing the test intentionally");
        }
        Assert.assertTrue(true);
    }
}

Optional: Apply Retry Analyzer Globally

Instead of using @Test(retryAnalyzer = ...) on each test, you can apply the retry logic globally using IAnnotationTransformer.

RetryListener.java

import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

public class RetryListener implements IAnnotationTransformer {
    @Override
    public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
        annotation.setRetryAnalyzer(RetryAnalyzer.class);
    }
}

testng.xml

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="RetrySuite">
    <listeners>
        <listener class-name="RetryListener"/>
    </listeners>

    <test name="RetryTest">
        <classes>
            <class name="TestRetryExample"/>
        </classes>
    </test>
</suite>

Important Points:

Component	Purpose
IRetryAnalyzer	Handles retry logic for a failed test
retry()	Returns true if the test should be retried
@Test(retryAnalyzer = ...)	Assigns retry logic to a test
IAnnotationTransformer	Optional global retry setup via listener