filtering-execution-path

# How to run specific section/generator

The generator and generic path filtering was added in Catch2 3.13.0

Catch2 supports picking specific path through a test case by filtering sections and generator indices to run through. This is done by using one of the three commandline parameters, one or more times.

-c, --section           <section name>
-g, --generator-index   <index in generator>
-p, --path-filter       <path filter spec>

All the variants form a shared stack of filters, but if you use only -c/--section form to specify section filters, you will get the old behaviour, which does not affect generators at all. If you also use either -g/--generator-index, or -p/--path-filter, you will get the new behaviour, which can also filter generator elements.

Both the new and old filter behaviours include some potentially surprising things: * Code outside of sections being skipped will still be executed. E.g. any setup code in the TEST_CASE that lives outside of sections. * Path filters filter the prefix of the path. So if you specify single filter, it affects only the top level sections/generator, with their child sections/generators being unfiltered. * Path filters are independent of test case selection, Catch2 will try to follow the path filters in all selected test cases. This means that if you specify path filters without a test case filter, Catch2 will try to apply the path filters inside every registered test case.

Old behaviour

The old behaviour was deprecated in Catch2 3.13.0

-c, --section           <section name>

The argument to -c/--section can be any arbitrary string. When Catch2 is deciding whether to enter a section, it will check its trimmed name against the appropriate trimmed section filter. If they are the same, the section can be opened. If not, Catch2 will skip over that section.

Examples

Simple section nesting

Given

TEST_CASE( "foo" ) {
    REQUIRE( true );
    SECTION( "A" ) {
        SECTION( "A1" ) { REQUIRE( true ); }
        SECTION( "A2" ) { REQUIRE( true ); }
    }
    SECTION( "B" ) {
        SECTION( "B1" ) { REQUIRE( true ); }
        SECTION( "B2" ) { REQUIRE( true ); }
    }
}

./tests foo -c A runs section “A” and both of its subsections, resulting in 4 assertions.
./tests foo -c A -c B runs section “A”, but none of its subsections, resulting in 1 assertion (the one before “A”).
./tests foo -c A -c A1 runs section “A” and only the “A1” subsection, resulting in 2 assertions.

Sections with nested generators

Note that old behaviour completely ignores generators. This means both that they can’t be filtered, but also that they aren’t taken into account for the filter depth. In other words, given

TEST_CASE( "bar" ) {
    REQUIRE( true );
    SECTION( "A" ) { REQUIRE( true ); }
    SECTION( "B" ) {
        auto i = GENERATE( 1, 2, 3 );
        DYNAMIC_SECTION( "i=" << i ) {
            REQUIRE( true );
        }
    }
}

./tests bar -c A results in 2 assertions.
./tests bar -c B -c i=2 results in 4 assertions, because the whole generator in section “B” has to be used up, but the dynamic section is only entered when the generator returns 2 as the value for i.
./tests bar -c B -c i=4 results in 3 assertions, because the assertion outside of section is executed every time the test case is entered, and the generator forces the test case to rerun 3 times before it is used up, even though the dynamic section will never be entered.

Section with sibling generators

For cases where sections have sibling generators, the filtering can get even more surprising.

TEST_CASE( "qux" ) {
    REQUIRE( true );
    SECTION( "A" ) { REQUIRE( true ); }
    auto i = GENERATE( 1, 2, 3 );
    DYNAMIC_SECTION( "i=" << i ) {
        REQUIRE( true );
    }
}

./tests qux -c A results in 4 assertions, because section “A” is entered once, but the sibling generator has to be exhausted, and the first assertion is executed once per generator element.
./tests qux -c i=2 also results in 4 assertions. Once again, the generator has to be exhausted and the dynamic section is entered once.

New behaviour

The new behaviour was introduced in Catch2 3.13.0

-g, --generator-index   <index in generator>
-p, --path-filter       <path filter spec>

The argument to -g/--generator-index must be either a non-negative number, which is interpreted as the index of the desired element from the generator, or “*”, which allows all elements from the generator.

Providing index outside of the generator is an error.

The argument to -p/--path-filter must start with either “c:” for a section filter, or with “g:” for a generator filter. Everything past the colon is then parsed as either a section filter, or a generator filter.

Note that using p/--path-filter enables new filtering behaviour, even if it is only used to add section filters.

There is another important difference between filtering out sections and generators. A section can be left un-entered, but a generator always has to be active. For this reason, if generator fails a filter (e.g. there is a section filter at given depth instead), it has to stop the execution of the test case. Currently, this is done via SKIP() equivalent, causing the section to be considered skipped.

Examples

Nested generators

TEST_CASE( "waldo" ) {
    auto i = GENERATE( 1, 10, 100 );
    auto j = GENERATE( 2, 20, 200 );
    CAPTURE( i, j );
    REQUIRE( true );
}

./tests waldo -g 1 results in 3 assertions, with i := 10, because the second nested generator is unfiltered.
./tests waldo -g 1 -g 2 results in 1 assertion, with i := 10, j := 200.
./tests waldo -g * -g 2 results in 3 assertions, all with j := 200.
./tests waldo -g 1 -g * results in 3 assertions, all with i := 10.
./tests waldo -g 3 results in 1 failed assertion, because the first generator does not have 3rd element.
./tests waldo -g * -g 3 results in 3 failed assertions, as the second generator does not have 3rd element, but we have to exhaust the first generator.

Generator with a nested dynamic section

TEST_CASE( "grault" ) {
    REQUIRE( true );
    auto i = GENERATE( 1, 2, 3 );
    DYNAMIC_SECTION( "i=" << i ) {
        REQUIRE( true );
    }
}

./tests grault -p g:1 results in 2 assertions, as there is no filter on the dynamic section.
./tests grault -p g:1 -p c:i=2 results in 2 assertions, as the filter on the dynamic section matches the element given from the generator.
./tests grault -p g:1 -p c:i=3 results in 1 assertion, as the generator is limited to only try i := 2 and the dynamic section is filtered out.

Section with a sibling generator

Because generators have to stop test execution when they don’t pass filter, it is impossible to run only a section with sibling generator without triggering a test case skip. Consider this test case from an earlier example:

TEST_CASE( "qux" ) {
    REQUIRE( true );
    SECTION( "A" ) { REQUIRE( true ); }
    auto i = GENERATE( 1, 2, 3 );
    DYNAMIC_SECTION( "i=" << i ) {
        REQUIRE( true );
    }
}

./tests qux -p g:1 results in 2 assertions, as the dynamic section is entered only once.
./tests qux -p g:1 -p c:i=1 results in 1 assertion, as the dynamic section filter is incompatible with the generator filter.
./tests qux -p c:A results in 2 assertions and a skipped test case. This is because the generator is sibling to section “A”, and thus reads the same section filter. However, it is not a section and as thus cannot proceed.
./tests qux -p c:i=2 results in 1 assertion and a skipped test case. Once again, the first filter in the filter stack is a section filter, and thus the generator cannot proceed.

Compare this with the old filter behaviour, where ./tests qux -c i=2 would instead result in 4 assertions, because the generator would go through all elements.

Home