name

[Ruby] Prevent WWW::Mechanize from stealing your memory

Wednesday, September 3, 2008 12:49:48 PM

Today i fought an intense battle with the garbage collector of Ruby, wondering, why all those images and pages which i downloaded with WWW::Mechanize seemed to fill my memory, though i removed all reference to them.
I unassigned them, let them create in forked processes, which i killed afterwards, and ran the GC manually every second.
Still my little programm wished for nothing less than to fill all available memory.

The solution to this was rather simple: WWW::Mechanize holds a history of visited pages, so one can use convenient features as the browser-like back() method.
Since i did not use this at all, i just had to assign 0 to history_size on the creation of my agent, to stop calling for swapmemory:

agent = WWW::Mechanize.new{ |agent|
	agent.history.max_size=0
}

I still wonder, why anyone should want this to be unassigned and therefore an infinite history, since i do not yet know anyone who possesses infinite memory for that one.

convmv is your friendKohlenstoffstahl - Werbeunfug oder reales Wort?

Comments

Unregistered user # Friday, November 28, 2008 10:09:15 AM

Annonomos writes: Thanks Daniel !! I was having the same problem with my memory filling over 4 gig. and slowing my computer down to a crawl. I added these lines to my application to isolate where my problem originated. puts "****************** start memory report" objects = Hash.new(0) p ObjectSpace.each_object{|obj| objects[obj.class] += 1} pp objects.sort_by{|k,v| -v} puts "*****************end memory report" and saw a growing number of Hpricot on each loop of my web site gathering, for example [Hpricot::Elem, 100071], the only thing I had using Hpricot was the mechinize so I searched google for "mechinize memory" and found you. I added the line as you suggested: agent.history.max_size=0 and now it's working with no more problems with constant memory usage. You saved me a lot of time. I'm not sure how long it would have taken me to figure that out. Thanks again. scotty0028663

Unregistered user # Tuesday, February 3, 2009 9:17:13 PM

Anonymous writes: Just found this. Genius! Thank you so much -

Unregistered user # Saturday, October 23, 2010 4:24:23 PM

Sergey Kovalev writes: Helped me a lot. Should be agent.history.max_size=1 if you want to parse HTML, fill forms, etc.

Unregistered user # Thursday, September 8, 2011 9:35:20 PM

Anonymous writes: Awesome. Thanks!