youys
/
cucumber-ruby
Not watched
1
0
Fork 0
Code
Releases 0 Wiki Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branch: master
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'master'
${ noResults }
cucumber-ruby/features/docs/cli/retry_failing_tests.feature

126 lines
3.3 kB
Raw Permalink Blame History 

  1. Feature: Retry failing tests

  2. 

  3.   Retry gives you a way to get through flaky tests that usually pass after a few runs.

  4.   This gives a development team a way forward other than disabling a valuable test.

  5. 

  6.   - Specify max retry count in option

  7.   - Output information to the screen

  8.   - Output retry information in test report

  9. 

  10.   Questions:

  11.   use a tag for flaky tests?  Global option to retry any test that fails?

  12. 

  13.   Background:

  14.     Given a scenario "Fails-once" that fails once, then passes

  15.     And a scenario "Fails-twice" that fails twice, then passes

  16.     And a scenario "Solid" that passes

  17. 

  18.   Scenario: Retry once, so Fails-once starts to pass

  19.     Given a scenario "Fails-forever" that fails

  20.     When I run `cucumber -q --retry 1 --format summary`

  21.     Then it should fail with:

  22.       """

  23.       4 scenarios (2 failed, 1 flaky, 1 passed)

  24.       """

  25.     And it should fail with:

  26.       """

  27.       Fails-forever

  28.         Fails-forever ✗

  29.         Fails-forever ✗

  30. 

  31.       Fails-once feature

  32.         Fails-once ✗

  33.         Fails-once ✓

  34. 

  35.       Fails-twice feature

  36.         Fails-twice ✗

  37.         Fails-twice ✗

  38. 

  39.       Solid

  40.         Solid ✓

  41.       """

  42. 

  43.   Scenario: Retry twice, so Fails-twice starts to pass too

  44.     Given a scenario "Fails-forever" that fails

  45.     When I run `cucumber -q --retry 2 --format summary`

  46.     Then it should fail with:

  47.       """

  48.       4 scenarios (1 failed, 2 flaky, 1 passed)

  49.       """

  50.     And it should fail with:

  51.       """

  52.       Fails-forever

  53.         Fails-forever ✗

  54.         Fails-forever ✗

  55.         Fails-forever ✗

  56. 

  57.       Fails-once feature

  58.         Fails-once ✗

  59.         Fails-once ✓

  60. 

  61.       Fails-twice feature

  62.         Fails-twice ✗

  63.         Fails-twice ✗

  64.         Fails-twice ✓

  65. 

  66.       Solid

  67.         Solid ✓

  68.       """

  69. 

  70.   Scenario: Flaky scenarios gives exit code zero in non-strict mode

  71.     When I run `cucumber -q --retry 2 --format summary`

  72.     Then it should pass with:

  73.       """

  74. 

  75. 

  76.       3 scenarios (2 flaky, 1 passed)

  77.       """

  78. 

  79.   Scenario: Flaky scenarios gives exit code zero in non-strict mode even when failing fast

  80.     When I run `cucumber -q --retry 2 --fail-fast --format summary`

  81.     Then it should pass with:

  82.       """

  83. 

  84. 

  85.       3 scenarios (2 flaky, 1 passed)

  86.       """

  87. 

  88.   Scenario: Flaky scenarios gives non-zero exit code in strict mode

  89.     When I run `cucumber -q --retry 2 --format summary --strict`

  90.     Then it should fail with:

  91.       """

  92.       Flaky Scenarios:

  93.       cucumber features/fails-once_feature.feature:2

  94.       cucumber features/fails-twice_feature.feature:2

  95. 

  96.       3 scenarios (2 flaky, 1 passed)

  97.       """

  98. 

  99.   Scenario: Retry each test but suspend retrying after 2 failing tests, so later tests are not retried

  100.     Given a scenario "Fails-forever-1" that fails

  101.     And a scenario "Fails-forever-2" that fails

  102.     When I run `cucumber -q --retry 1 --retry-total 2 --format summary`

  103.     Then it should fail with:

  104.       """

  105.       5 scenarios (4 failed, 1 passed)

  106.       """

  107.     And it should fail with:

  108.       """

  109.       Fails-forever-1

  110.         Fails-forever-1 ✗

  111.         Fails-forever-1 ✗

  112. 

  113.       Fails-forever-2

  114.         Fails-forever-2 ✗

  115.         Fails-forever-2 ✗

  116. 

  117.       Fails-once feature

  118.         Fails-once ✗

  119. 

  120.       Fails-twice feature

  121.         Fails-twice ✗

  122. 

  123.       Solid

  124.         Solid ✓

  125.       """