[Webtest] New Warning test result
Marc Guillemot <email@example.com>
Tue, 5 Jun 2007 03:50:17 -0700 (PDT)
ups, I've already been faster to answer, sorry.
I don't understand the relationship between WT-244, non breaking failures
We've discussed here for some time (I think this was Denis' suggestion) to
add the possibility to collect verifications failures to make a <webtest>
failed only at the end and have the possibility to display all verification
failures in one time. I don't know if an issue has been opened for that or
not. If I correctly understand you, you would like to have such results as
let say yellow (because neither green nor red) in the overview report? I
have to think more about that, but it could make sense to me too. I've
already had tests where I had to implement a workaround due to a known issue
in the application and where I wasn't satisfied to see them green (thanks to
the workaround) in the report.
The report format surely needs to be changed for that because currently only
1 error/failure is possible per <webtest> and the result of a <webtest> can
only be successful or failed. So you can't achieve that just by extending
Please open a new issue for that.
Nate Oster wrote:
> Marc and WebTest committers,
> Congrats on WebTest2.5! The reporting improvements are fantastic, and
> we're already switching over to the mostly-self-documenting webtest.xml
> file for running our scripts. Now that the results reporting code is
> stabilized again, I think it might be time to reconsider adding explicit
> support for Warning as a test result.
> There are a number of recurring issues and current JIRA enhancement
> requests that would be greatly simplified by introducing a new result
> type to WebTest. For example, JIRA WT-244 "Perform a set of steps on
> every response," would be pretty simple to implement with some Groovy
> code if you had a simple way of *not failing* a test case when one of
> the validation steps failed. You could simply recast the "failure"
> exception as a "warning" instead.
> Similarly, my current project wants to create a "Workaround" step, so
> that we can mark code that's expected to fail because of a known bug and
> provide an alterative set of steps. However, we don't want these
> "patched" test scripts to simply PASS(!) - we want to indicate that
> something is "suspect" about them. A simple Warning result would be
> perfect for this.
> So, "Warning" would be a lot like JUnit's "Error" result. It's not a
> "Failure" of the test, but it's not the expected result either. I know
> that sounds odd with a declarative-procedural test automation tool, but
> it actually exposes a generic mechanism and consistent reporting that
> could really improve design, maintainability, and extensibility of our
> I'm especially interested in how easy it would be to take advantage of
> this result type by extending Step with a Groovy macro. The "Workaround
> step" idea, for example, would be easy to implement by extending
> GroupStep with a little Groovy. We tried to introduce this ourselves,
> but the results reporting code was undergoing so much change, and we're
> so unfamiliar with the WebTest internals, that it proved difficult to
> keep up.
> What do you think? Should I raise a Feature request to introduce a
> "Warning" test result?
> Nate Oster
> To: firstname.lastname@example.org
> From: Marc Guillemot <email@example.com>
> Date: Mon, 16 Apr 2007 09:44:22 +0200
> Subject: [Webtest] Re: Resume after failing step
> Reply-To: firstname.lastname@example.org
> Reply-To: Marc Guillemot <email@example.com>
> Hi Christoph,
> if you really want to have "non failing" errors (I still believe that it
> is better to split your tests, but I don't know all the details of your
> case), you can easily extend WebTest with something like that:
> <groovyScript description="define custom step">
> public class MySafeFailureContainer
> extends com.canoo.webtest.steps.control
> public void doExecute() throws Exception
> catch (final Exception e)
> // just dismiss
> and then you can use this custom task inside your steps just like any
> .. some steps...
> Here groovyScript refers to the Groovy Ant task (not the WebTest step)
> as defined by the not-yet-but-very-soon-documented utility webtest.xml
> form the latest builds. The call should naturally occur before usage of
> the task.
> As already said, this will generate reports that are difficult to
> analyse. On the other side this shows how easy it is to extend WebTest
> Marc Guillemot wrote:
>> Seems to be similar to Denis' case and would make sense (if wisely
>> used). Perhaps can it be implemented based on WT-251 but this requires
>> some changes in the xml report format as currently only one
>> error/failure is saved.
>> Michal wrote:
>>> when I started to work with webtest I also missed extended
>>> For example in our tests after we load page we do common verification
>>> for valid urls, 404 errors, we load css, images, etc...
>>> If this test fails it is not critical to whole <webtest> section, we
>>> would like to know about the error but at the same time we would like
>>> continue testing.
>>> Therefore it would be excellent if we would be able to create a step
>>> like <verifyTitle text="my title" onfailure="continuetests" /> which
>>> would indicate that this step is not critical for the whole webtest
>>> steps should be executed as if <verifyTitle> passed even if it
>>> What do you think?
>>> Marc Guillemot wrote:
>>>> haltonfailure and haltonerror both allow to configure if the build
>>>> should fail or not. Ie if Ant should stop the execution after
>>>> In both cases the execution of the steps is stopped after the
>>>> Generally I think quite strange to continue a test when something
>>>> failed. How to consider the results at the end? Correct or not
>>>> Nevertheless it may make sense in some cases. Denis mentioned a case
>>>> me last week where he had some <verifyXxxx/> to check the Italian
>>>> translation of some text. The application behaved correctly but some
>>>> the texts where wrong and it would have been more useful to complete
>>>> execution and show all wrong texts in one time.
>>>> is your case comparable with Denis' one?
>>>> Perhaps should you refactor your tests. If you have a long way to go
>>>> a particular page and want to perform different actions from there,
>>>> about splitting it? A first test would just go to the particular
>>>> and store on file session information (probably the cookies that
>>>> to hold the session). Then each single "test from the particular
>>>> could just reuse this session by setting the cookie value from what
>>>> read from file. This would give you a far better information in
>>>> you would see precisely which single "test from the particular page"
>>>> works and which fails.
> WebTest mailing list
View this message in context: http://www.nabble.com/New-Warning-test-result-tf3779767.html#a10967103
Sent from the WebTest mailing list archive at Nabble.com.