youys
/
cucumber-ruby
Not watched
1
0
Fork 0
Code
Releases 0 Wiki Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
Tree: 4641dd3d32
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '4641dd3d32'
${ noResults }
cucumber-ruby/features/docs/formatters/junit_formatter.feature

536 lines
18 kB
Raw Blame History 

  1. Feature: JUnit output formatter

  2.   In order for developers to create test reports with ant

  3.   Cucumber should be able to output JUnit xml files

  4. 

  5.   Background:

  6.     Given the standard step definitions

  7.     And a file named "features/one_passing_one_failing.feature" with:

  8.       """

  9.       Feature: One passing scenario, one failing scenario

  10. 

  11.         Scenario: Passing

  12.           Given this step passes

  13. 

  14.         Scenario: Failing

  15.           Given this step fails

  16.       """

  17.     And a file named "features/some_subdirectory/one_passing_one_failing.feature" with:

  18.       """

  19.       Feature: Subdirectory - One passing scenario, one failing scenario

  20. 

  21.         Scenario: Passing

  22.           Given this step passes

  23. 

  24.         Scenario: Failing

  25.           Given this step fails

  26.       """

  27.     And a file named "features/pending.feature" with:

  28.       """

  29.       Feature: Pending step

  30. 

  31.         Scenario: Pending

  32.           Given this step is pending

  33. 

  34.         Scenario: Undefined

  35.           Given this step is undefined

  36.       """

  37.     And a file named "features/pending.feature" with:

  38.       """

  39.       Feature: Pending step

  40. 

  41.         Scenario: Pending

  42.           Given this step is pending

  43. 

  44.         Scenario: Undefined

  45.           Given this step is undefined

  46.       """

  47.     And a file named "features/scenario_outline.feature" with:

  48.       """

  49.       Feature: Scenario outlines

  50. 

  51.         Scenario Outline: Using scenario outlines

  52.           Given this step <type>

  53. 

  54.           Examples:

  55.             | type         |

  56.             | passes       |

  57.             | fails        |

  58.             | is pending   |

  59.             | is undefined |

  60.       """

  61.     And a file named "features/encoding_output.feature" with:

  62.       """

  63.       Feature: One passing scenario, one failing scenario

  64. 

  65.         Scenario: Passing

  66.           Given a passing ctrl scenario

  67.       """

  68.     And a file named "features/step_definitions/junit_steps.rb" with:

  69.       """

  70.       Given /^Given a passing ctrl scenario/ do

  71.         Kernel.puts "encoding"

  72.         Kernel.puts "pickle".encode("UTF-16")

  73.       end

  74.       """

  75. 

  76.   Scenario: one feature, one passing scenario, one failing scenario

  77.     When I run `cucumber --format junit --out tmp/ features/one_passing_one_failing.feature`

  78.     Then it should fail with:

  79.       """

  80. 

  81.       """

  82.     And the junit output file "tmp/TEST-features-one_passing_one_failing.xml" should contain:

  83.       """

  84.       <?xml version="1.0" encoding="UTF-8"?>

  85.       <testsuite failures="1" errors="0" skipped="0" tests="2" time="0.05" name="One passing scenario, one failing scenario">

  86.       <testcase classname="One passing scenario, one failing scenario" name="Passing" time="0.05">

  87.         <system-out>

  88.           <![CDATA[]]>

  89.         </system-out>

  90.         <system-err>

  91.           <![CDATA[]]>

  92.         </system-err>

  93.       </testcase>

  94.       <testcase classname="One passing scenario, one failing scenario" name="Failing" time="0.05">

  95.         <failure message="failed Failing" type="failed">

  96.           <![CDATA[Scenario: Failing

  97. 

  98.       Given this step fails

  99. 

  100.       Message:

  101.       ]]>

  102.           <![CDATA[ (RuntimeError)

  103.       ./features/step_definitions/steps.rb:4:in `/^this step fails$/'

  104.       features/one_passing_one_failing.feature:7:in `this step fails']]>

  105.         </failure>

  106.         <system-out>

  107.           <![CDATA[]]>

  108.         </system-out>

  109.         <system-err>

  110.           <![CDATA[]]>

  111.         </system-err>

  112.       </testcase>

  113.       </testsuite>

  114. 

  115.       """

  116. 

  117.   Scenario: one feature, one passing scenario, one failing scenario, add file attribute

  118.     When I run `cucumber --format junit,fileattribute=true --out tmp/ features/one_passing_one_failing.feature`

  119.     Then it should fail with:

  120.       """

  121. 

  122.       """

  123.     And the junit output file "tmp/TEST-features-one_passing_one_failing.xml" should contain:

  124.       """

  125.       <?xml version="1.0" encoding="UTF-8"?>

  126.       <testsuite failures="1" errors="0" skipped="0" tests="2" time="0.05" name="One passing scenario, one failing scenario">

  127.       <testcase classname="One passing scenario, one failing scenario" name="Passing" time="0.05" file="features/one_passing_one_failing.feature">

  128.         <system-out>

  129.           <![CDATA[]]>

  130.         </system-out>

  131.         <system-err>

  132.           <![CDATA[]]>

  133.         </system-err>

  134.       </testcase>

  135.       <testcase classname="One passing scenario, one failing scenario" name="Failing" time="0.05" file="features/one_passing_one_failing.feature">

  136.         <failure message="failed Failing" type="failed">

  137.           <![CDATA[Scenario: Failing

  138. 

  139.       Given this step fails

  140. 

  141.       Message:

  142.       ]]>

  143.           <![CDATA[ (RuntimeError)

  144.       ./features/step_definitions/steps.rb:4:in `/^this step fails$/'

  145.       features/one_passing_one_failing.feature:7:in `this step fails']]>

  146.         </failure>

  147.         <system-out>

  148.           <![CDATA[]]>

  149.         </system-out>

  150.         <system-err>

  151.           <![CDATA[]]>

  152.         </system-err>

  153.       </testcase>

  154.       </testsuite>

  155. 

  156.       """

  157. 

  158.   Scenario: one feature in a subdirectory, one passing scenario, one failing scenario

  159.     When I run `cucumber --format junit --out tmp/ features/some_subdirectory/one_passing_one_failing.feature --require features`

  160.     Then it should fail with:

  161.       """

  162. 

  163.       """

  164.     And the junit output file "tmp/TEST-features-some_subdirectory-one_passing_one_failing.xml" should contain:

  165.       """

  166.       <?xml version="1.0" encoding="UTF-8"?>

  167.       <testsuite failures="1" errors="0" skipped="0" tests="2" time="0.05" name="Subdirectory - One passing scenario, one failing scenario">

  168.       <testcase classname="Subdirectory - One passing scenario, one failing scenario" name="Passing" time="0.05">

  169.         <system-out>

  170.           <![CDATA[]]>

  171.         </system-out>

  172.         <system-err>

  173.           <![CDATA[]]>

  174.         </system-err>

  175.       </testcase>

  176.       <testcase classname="Subdirectory - One passing scenario, one failing scenario" name="Failing" time="0.05">

  177.         <failure message="failed Failing" type="failed">

  178.           <![CDATA[Scenario: Failing

  179. 

  180.       Given this step fails

  181. 

  182.       Message:

  183.       ]]>

  184.           <![CDATA[ (RuntimeError)

  185.       ./features/step_definitions/steps.rb:4:in `/^this step fails$/'

  186.       features/some_subdirectory/one_passing_one_failing.feature:7:in `this step fails']]>

  187.         </failure>

  188.         <system-out>

  189.           <![CDATA[]]>

  190.         </system-out>

  191.         <system-err>

  192.           <![CDATA[]]>

  193.         </system-err>

  194.       </testcase>

  195.       </testsuite>

  196. 

  197.       """

  198. 

  199.   Scenario: pending and undefined steps are reported as skipped

  200.     When I run `cucumber --format junit --out tmp/ features/pending.feature`

  201.     Then it should pass with:

  202.       """

  203. 

  204.       """

  205.     And the junit output file "tmp/TEST-features-pending.xml" should contain:

  206.       """

  207.       <?xml version="1.0" encoding="UTF-8"?>

  208.       <testsuite failures="0" errors="0" skipped="2" tests="2" time="0.05" name="Pending step">

  209.       <testcase classname="Pending step" name="Pending" time="0.05">

  210.         <skipped/>

  211.         <system-out>

  212.           <![CDATA[]]>

  213.         </system-out>

  214.         <system-err>

  215.           <![CDATA[]]>

  216.         </system-err>

  217.       </testcase>

  218.       <testcase classname="Pending step" name="Undefined" time="0.05">

  219.         <skipped/>

  220.         <system-out>

  221.           <![CDATA[]]>

  222.         </system-out>

  223.         <system-err>

  224.           <![CDATA[]]>

  225.         </system-err>

  226.       </testcase>

  227.       </testsuite>

  228. 

  229.       """

  230. 

  231.   Scenario: pending and undefined steps add fileattribute

  232.     When I run `cucumber --format junit,fileattribute=true --out tmp/ features/pending.feature`

  233.     Then it should pass with:

  234.       """

  235. 

  236.       """

  237.     And the junit output file "tmp/TEST-features-pending.xml" should contain:

  238.       """

  239.       <?xml version="1.0" encoding="UTF-8"?>

  240.       <testsuite failures="0" errors="0" skipped="2" tests="2" time="0.05" name="Pending step">

  241.       <testcase classname="Pending step" name="Pending" time="0.05" file="features/pending.feature">

  242.         <skipped/>

  243.         <system-out>

  244.           <![CDATA[]]>

  245.         </system-out>

  246.         <system-err>

  247.           <![CDATA[]]>

  248.         </system-err>

  249.       </testcase>

  250.       <testcase classname="Pending step" name="Undefined" time="0.05" file="features/pending.feature">

  251.         <skipped/>

  252.         <system-out>

  253.           <![CDATA[]]>

  254.         </system-out>

  255.         <system-err>

  256.           <![CDATA[]]>

  257.         </system-err>

  258.       </testcase>

  259.       </testsuite>

  260. 

  261.       """

  262. 

  263.   Scenario: pending and undefined steps with strict option should fail

  264.     When I run `cucumber --format junit --out tmp/ features/pending.feature --strict`

  265.     Then it should fail with:

  266.       """

  267. 

  268.       """

  269.     And the junit output file "tmp/TEST-features-pending.xml" should contain:

  270.       """

  271.       <?xml version="1.0" encoding="UTF-8"?>

  272.       <testsuite failures="2" errors="0" skipped="0" tests="2" time="0.05" name="Pending step">

  273.       <testcase classname="Pending step" name="Pending" time="0.05">

  274.         <failure message="pending Pending" type="pending">

  275.           <![CDATA[Scenario: Pending

  276. 

  277.       Given this step is pending

  278. 

  279.       Message:

  280.       ]]>

  281.           <![CDATA[TODO (Cucumber::Pending)

  282.       ./features/step_definitions/steps.rb:3:in `/^this step is pending$/'

  283.       features/pending.feature:4:in `this step is pending']]>

  284.         </failure>

  285.         <system-out>

  286.           <![CDATA[]]>

  287.         </system-out>

  288.         <system-err>

  289.           <![CDATA[]]>

  290.         </system-err>

  291.       </testcase>

  292.       <testcase classname="Pending step" name="Undefined" time="0.05">

  293.         <failure message="undefined Undefined" type="undefined">

  294.           <![CDATA[Scenario: Undefined

  295. 

  296.       Given this step is undefined

  297. 

  298.       Message:

  299.       ]]>

  300.           <![CDATA[Undefined step: "this step is undefined" (Cucumber::Core::Test::Result::Undefined)

  301.       features/pending.feature:7:in `this step is undefined']]>

  302.         </failure>

  303.         <system-out>

  304.           <![CDATA[]]>

  305.         </system-out>

  306.         <system-err>

  307.           <![CDATA[]]>

  308.         </system-err>

  309.       </testcase>

  310.       </testsuite>

  311. 

  312.       """

  313. 

  314.   Scenario: run all features

  315.     When I run `cucumber --format junit --out tmp/ features`

  316.     Then it should fail with:

  317.       """

  318. 

  319.       """

  320.     And a file named "tmp/TEST-features-one_passing_one_failing.xml" should exist

  321.     And a file named "tmp/TEST-features-pending.xml" should exist

  322. 

  323.   Scenario: show correct error message if no --out is passed

  324.     When I run `cucumber --format junit features`

  325.     Then the stderr should not contain:

  326.       """

  327. can't convert .* into String \(TypeError\)

  328.       """

  329.     And the stderr should contain:

  330.       """

  331. You *must* specify --out DIR for the junit formatter

  332.       """

  333. 

  334.   Scenario: strict mode, one feature, one scenario outline, four examples: one passing, one failing, one pending, one undefined

  335.     When I run `cucumber --strict --format junit --out tmp/ features/scenario_outline.feature`

  336.     Then it should fail with:

  337.       """

  338. 

  339.       """

  340.     And the junit output file "tmp/TEST-features-scenario_outline.xml" should contain:

  341.       """

  342.       <?xml version="1.0" encoding="UTF-8"?>

  343.       <testsuite failures="3" errors="0" skipped="0" tests="4" time="0.05" name="Scenario outlines">

  344.       <testcase classname="Scenario outlines" name="Using scenario outlines (outline example : | passes |)" time="0.05">

  345.         <system-out>

  346.           <![CDATA[]]>

  347.         </system-out>

  348.         <system-err>

  349.           <![CDATA[]]>

  350.         </system-err>

  351.       </testcase>

  352.       <testcase classname="Scenario outlines" name="Using scenario outlines (outline example : | fails |)" time="0.05">

  353.         <failure message="failed Using scenario outlines (outline example : | fails |)" type="failed">

  354.           <![CDATA[Scenario Outline: Using scenario outlines

  355. 

  356.       Example row: | fails |

  357. 

  358.       Message:

  359.       ]]>

  360.           <![CDATA[ (RuntimeError)

  361.       ./features/step_definitions/steps.rb:4:in `/^this step fails$/'

  362.       features/scenario_outline.feature:9:4:in `this step fails']]>

  363.         </failure>

  364.         <system-out>

  365.           <![CDATA[]]>

  366.         </system-out>

  367.         <system-err>

  368.           <![CDATA[]]>

  369.         </system-err>

  370.       </testcase>

  371.       <testcase classname="Scenario outlines" name="Using scenario outlines (outline example : | is pending |)" time="0.05">

  372.         <failure message="pending Using scenario outlines (outline example : | is pending |)" type="pending">

  373.           <![CDATA[Scenario Outline: Using scenario outlines

  374. 

  375.       Example row: | is pending |

  376. 

  377.       Message:

  378.       ]]>

  379.           <![CDATA[TODO (Cucumber::Pending)

  380.       ./features/step_definitions/steps.rb:3:in `/^this step is pending$/'

  381.       features/scenario_outline.feature:10:4:in `this step is pending']]>

  382.         </failure>

  383.         <system-out>

  384.           <![CDATA[]]>

  385.         </system-out>

  386.         <system-err>

  387.           <![CDATA[]]>

  388.         </system-err>

  389.       </testcase>

  390.       <testcase classname="Scenario outlines" name="Using scenario outlines (outline example : | is undefined |)" time="0.05">

  391.         <failure message="undefined Using scenario outlines (outline example : | is undefined |)" type="undefined">

  392.           <![CDATA[Scenario Outline: Using scenario outlines

  393. 

  394.       Example row: | is undefined |

  395. 

  396.       Message:

  397.       ]]>

  398.           <![CDATA[Undefined step: "this step is undefined" (Cucumber::Core::Test::Result::Undefined)

  399.       features/scenario_outline.feature:11:4:in `this step is undefined']]>

  400.         </failure>

  401.         <system-out>

  402.           <![CDATA[]]>

  403.         </system-out>

  404.         <system-err>

  405.           <![CDATA[]]>

  406.         </system-err>

  407.       </testcase>

  408.       </testsuite>

  409. 

  410.       """

  411. 

  412.   Scenario: strict mode with --expand option, one feature, one scenario outline, four examples: one passing, one failing, one pending, one undefined

  413.     When I run `cucumber --strict --expand --format junit --out tmp/ features/scenario_outline.feature --publish-quiet`

  414.     Then it should fail with exactly:

  415.       """

  416. 

  417.       """

  418.     And the junit output file "tmp/TEST-features-scenario_outline.xml" should contain:

  419.       """

  420.       <?xml version="1.0" encoding="UTF-8"?>

  421.       <testsuite failures="3" errors="0" skipped="0" tests="4" time="0.05" name="Scenario outlines">

  422.       <testcase classname="Scenario outlines" name="Using scenario outlines (outline example : | passes |)" time="0.05">

  423.         <system-out>

  424.           <![CDATA[]]>

  425.         </system-out>

  426.         <system-err>

  427.           <![CDATA[]]>

  428.         </system-err>

  429.       </testcase>

  430.       <testcase classname="Scenario outlines" name="Using scenario outlines (outline example : | fails |)" time="0.05">

  431.         <failure message="failed Using scenario outlines (outline example : | fails |)" type="failed">

  432.           <![CDATA[Scenario Outline: Using scenario outlines

  433. 

  434.       Example row: | fails |

  435. 

  436.       Message:

  437.       ]]>

  438.           <![CDATA[ (RuntimeError)

  439.       ./features/step_definitions/steps.rb:4:in `/^this step fails$/'

  440.       features/scenario_outline.feature:9:4:in `this step fails']]>

  441.         </failure>

  442.         <system-out>

  443.           <![CDATA[]]>

  444.         </system-out>

  445.         <system-err>

  446.           <![CDATA[]]>

  447.         </system-err>

  448.       </testcase>

  449.       <testcase classname="Scenario outlines" name="Using scenario outlines (outline example : | is pending |)" time="0.05">

  450.         <failure message="pending Using scenario outlines (outline example : | is pending |)" type="pending">

  451.           <![CDATA[Scenario Outline: Using scenario outlines

  452. 

  453.       Example row: | is pending |

  454. 

  455.       Message:

  456.       ]]>

  457.           <![CDATA[TODO (Cucumber::Pending)

  458.       ./features/step_definitions/steps.rb:3:in `/^this step is pending$/'

  459.       features/scenario_outline.feature:10:4:in `this step is pending']]>

  460.         </failure>

  461.         <system-out>

  462.           <![CDATA[]]>

  463.         </system-out>

  464.         <system-err>

  465.           <![CDATA[]]>

  466.         </system-err>

  467.       </testcase>

  468.       <testcase classname="Scenario outlines" name="Using scenario outlines (outline example : | is undefined |)" time="0.05">

  469.         <failure message="undefined Using scenario outlines (outline example : | is undefined |)" type="undefined">

  470.           <![CDATA[Scenario Outline: Using scenario outlines

  471. 

  472.       Example row: | is undefined |

  473. 

  474.       Message:

  475.       ]]>

  476.           <![CDATA[Undefined step: "this step is undefined" (Cucumber::Core::Test::Result::Undefined)

  477.       features/scenario_outline.feature:11:4:in `this step is undefined']]>

  478.         </failure>

  479.         <system-out>

  480.           <![CDATA[]]>

  481.         </system-out>

  482.         <system-err>

  483.           <![CDATA[]]>

  484.         </system-err>

  485.       </testcase>

  486.       </testsuite>

  487. 

  488.       """

  489. 

  490.   Scenario: run test cases from different features interweaved

  491.     When I run `cucumber --format junit --out tmp/ features/one_passing_one_failing.feature:3 features/pending.feature:3 features/one_passing_one_failing.feature:6`

  492.     Then it should fail with:

  493.       """

  494. 

  495.       """

  496.     And the junit output file "tmp/TEST-features-one_passing_one_failing.xml" should contain:

  497.       """

  498.       <?xml version="1.0" encoding="UTF-8"?>

  499.       <testsuite failures="1" errors="0" skipped="0" tests="2" time="0.05" name="One passing scenario, one failing scenario">

  500.       <testcase classname="One passing scenario, one failing scenario" name="Passing" time="0.05">

  501.         <system-out>

  502.           <![CDATA[]]>

  503.         </system-out>

  504.         <system-err>

  505.           <![CDATA[]]>

  506.         </system-err>

  507.       </testcase>

  508.       <testcase classname="One passing scenario, one failing scenario" name="Failing" time="0.05">

  509.         <failure message="failed Failing" type="failed">

  510.           <![CDATA[Scenario: Failing

  511. 

  512.       Given this step fails

  513. 

  514.       Message:

  515.       ]]>

  516.           <![CDATA[ (RuntimeError)

  517.       ./features/step_definitions/steps.rb:4:in `/^this step fails$/'

  518.       features/one_passing_one_failing.feature:7:in `this step fails']]>

  519.         </failure>

  520.         <system-out>

  521.           <![CDATA[]]>

  522.         </system-out>

  523.         <system-err>

  524.           <![CDATA[]]>

  525.         </system-err>

  526.       </testcase>

  527.       </testsuite>

  528. 

  529.       """

  530.     And a file named "tmp/TEST-features-pending.xml" should exist

  531. 

  532.   @wip-jruby

  533.   Scenario: mixed encoding outputs are handled across all supported ruby versions, failing on jruby currently

  534.     When I run `cucumber --format junit --out tmp/ features/encoding_output.feature`

  535.     Then it should pass