[Webtest] OutOfMemoryError

Greg Vatman webtest@lists.canoo.com
Wed, 29 Sep 2004 14:59:19 -0700


After much painful debugging, I have finally discovered the problem and =
found the solution.

During one of my OutOfMemoryError messages, I received some extra =
information about the heap:

compacting perm gen  total 65536K, used 65535K [0x14010000, 0x18010000, =
0x18010000)
   the space 65536K,  99% used [0x14010000, 0x1800fd98, 0x1800fe00, =
0x18010000)

It seems the permanent generation is spaced used by Sun's HotSpot to =
optimize performance and is seperate from heap sizes set by using Xmx =
and Xms options.  This explains why I never reach my maximum heap size =
before getting OutOfMemoryError.

Ant is loading a huge number of jars and using many classes and methods, =
all of which are being dumped into the permanent space.  The default =
size is 32M for clients and 64M for servers.  I guess that is why some =
people solved their issues by using the -server option.

My solution was to use the following option in ANT_OPTS: =
-XX:MaxPermSize=3D128m

Now I'm able to run the entire suite without error.


-----Original Message-----
From: webtest-admin@gate2.canoo.com
[mailto:webtest-admin@gate2.canoo.com]On Behalf Of Luca Scheuring
Sent: Tuesday, September 28, 2004 12:26 AM
To: webtest@gate2.canoo.com
Subject: AW: [Webtest] OutOfMemoryError


I had a very similar problem recently and when you say that you have a =
huge
allocation in StringBuffer, it's sounds familiar to me.

first I was running my testsuite on windows where it worked fine, memory
consumption around 50MB. than I tried to run the same tests on linux. =
there,
I had the OutOfMemoryError. I saw that some strings in the report =
xml-file
were extremely long, up to 100'000 characters. this lead to a huge =
report
file which caused the high memory consumption. the problem was that the
language settings of my linux shell was set to LANG=3Den_US.UTF-8. since =
there
exist some =F6=E4=FC characters in my properties-files, this made string
processing (don't know where exactly) freaking out. setting LANG=3Den_US
solved the problem.

maybe you have a similar problem.

--luca

> -----Urspr=FCngliche Nachricht-----
> Von: webtest-admin@lists.canoo.com=20
> [mailto:webtest-admin@lists.canoo.com] Im Auftrag von Greg Vatman
> Gesendet: Dienstag, 28. September 2004 00:44
> An: webtest@gate2.canoo.com
> Betreff: RE: [Webtest] OutOfMemoryError
>=20
> Well, I'm about out of solutions.
>=20
> * I only ever used one call to taskdefs
> * I tried splitting the xml files out by project (1-2 test=20
> cases per project)
> * I set property webtest.resultreporterclass =3D=20
> com.canoo.webtest.reporting.PlainTextReporter
> * tried using -XX:+AggressiveHeap
> * tried setting heap sizes with -Xms and -Xmx
>=20
> Nothing has helped.  Memory consumption always seems to=20
> increase to around 150MB and I get an OutOfMemoryError.
>=20
> Running the script through hprof shows a huge char allocation=20
> in StringBuffer:
>=20
>          percent         live       alloc'ed  stack class
>  rank   self  accum    bytes objs   bytes objs trace name
>    46  0.40% 43.02%    39000  238 32523352 209686   129 [C
>=20
> which maps to this stack trace:
>=20
> TRACE 129:
>         java.lang.StringBuffer.expandCapacity(StringBuffer.java:202)
>         java.lang.StringBuffer.append(StringBuffer.java:401)
>         java.util.zip.ZipFile.getEntry(ZipFile.java:149)
>         java.util.jar.JarFile.getEntry(JarFile.java:194)
>         java.util.jar.JarFile.getJarEntry(JarFile.java:181)
>        =20
> sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:668)
>         sun.misc.URLClassPath.getResource(URLClassPath.java:157)
>         java.net.URLClassLoader$1.run(URLClassLoader.java:191)
>        =20
> java.security.AccessController.doPrivileged(AccessController.j
> ava:Native method)
>=20
>=20
> That is alot of unused chars...
>=20
> I'm out of ideas at this point.  Is anyone else running a=20
> large suite with many steps?  Has anyone been able to solve a=20
> similar issue?  I really hoping I don't have to scrap my=20
> WebTest project.
>=20
> Thanks,
> Greg
>=20
> -----Original Message-----
> From: webtest-admin@gate2.canoo.com
> [mailto:webtest-admin@gate2.canoo.com]On Behalf Of Dierk Koenig
> Sent: Friday, September 24, 2004 12:18 AM
> To: webtest@gate2.canoo.com
> Subject: RE: [Webtest] OutOfMemoryError
>=20
>=20
> Hi Greg,
>=20
> there are two major reasons for excessive memory consumption:
> 1) too many calls to taskdef
> 2) large result (xml) files
>=20
> ad 1)
> every time your tests hit a taskdef a new classloader is=20
> constructed reading all webtest (httunit, etc.) classes in Memory.
> Try having only one taskdef. The /selftests dir that comes=20
> with the doc.zip give an example on how to do that.
>=20
> ad 2)
> when collecting too many results in one xml file the "append"=20
> to that file first reads the file, parses it, does the append=20
> and writes it back.
> This can result in a large memory consumption for large suites.
> Solutions:
> a) use the PlainTextReporter if its output is good enough for your
>    purposes
> b) use a new xml file for each test or sub-suite and merge them
>    afterwards
> I'm afraid we don't have an example for this, yet.
>=20
> In our environment 2000 Steps is an upper limit for what we=20
> put into one result.xml. A few hundred would be a typical value.
>=20
> cheers
> Mittie
>=20
> -----Original Message-----
> From: webtest-admin@lists.canoo.com=20
> [mailto:webtest-admin@lists.canoo.com]On
> Behalf Of Greg Vatman
> Sent: Donnerstag, 23. September 2004 21:49
> To: webtest@gate2.canoo.com
> Subject: [Webtest] OutOfMemoryError
>=20
>=20
> I'm attemping to run about 35 automated test cases, but I=20
> keep getting an OutOfMemoryError exception.  These functional=20
> tests are pretty large and have many steps involved.  I've=20
> tried increasing the JVM heap size using -Xms and -Xmx (in=20
> ANT_OPTS), but it doesn't help.
> The memory error always seems to occur after the 28th test=20
> (at which point a little over 2000 test steps have been executed).
> I've check the mail archive and checked for previous issues:
>  * showhtmlparseroutput is set to false
>  * &definition; is only used in the main test suite file
>  * using the latest version of Ant (1.6.2) and WebTest (507)=20
> Is anyone having success running a large set of tests with=20
> many steps?  Does anyone have any other suggestions?
> Thanks in advance.
> - Greg
>=20
> _______________________________________________
> WebTest mailing list
> WebTest@lists.canoo.com
> http://lists.canoo.com/mailman/listinfo/webtest
> _______________________________________________
> WebTest mailing list
> WebTest@lists.canoo.com
> http://lists.canoo.com/mailman/listinfo/webtest
>=20

_______________________________________________
WebTest mailing list
WebTest@lists.canoo.com
http://lists.canoo.com/mailman/listinfo/webtest