[Webtest] cruisecontrol

[Dierk Koenig]

Hi David,

> i am using webtest within cruisecontrol and I'd like to make
> cruisecontrol
> build fail if there is any failing webtest step. my build.xml
> tasks are like
> these:
>
>  - check repository for changes
>  - if no changes wait and repeat
>  - checkout project
>  - build and unit testing (with phpunit or junit)
>  - deploy on an integration server
>  - functional testing on the integration server (with webtest)

I guess you have some more steps here, that you don't want to
get executed, if the webtests fail. (e.g. tagging the
current version)

> if everything works fine cruisecontrol sends a "Build Successful"
> email, and
> if something goes wrong it ought send a "Build Failed" message.
> an error in
> junit testing breaks the build and cruisecontrol sends the "Build Failed"
> message, but, and that's my point, an error in webtest testing
> does not break
> the build. why?

By default, WebTests are supposed to work as user acceptance tests
rather than technical verifications. Acceptance tests are likely
not to run 100% from the time they were delivered. Therefore,
WebTest just reports about the progress in terms of running
acceptance tests rather than breaking the build.
This is in contrast to Unit Tests. These are supposed to run
100% with every build.

You can steer this behaviour of WebTest with the
"haltonerror" and "haltonfailure" attributes of the config step.
see http://webtest.canoo.com/webtest/manual/syntax.html#2

For Build number 231 you will find the report of the concerned
selftest step under
http://webtest.canoo.com/selftest/TestResultOverview.html#testspec22

hope that helps and happy testing
Mittie