PEPPLER.ORG
Michael Peppler
Sybase Consulting
Menu
Home
Sybase on Linux
Install Guide for Sybase on Linux
General Sybase Resources
General Perl Resources
Freeware
Sybperl
Sybase::Simple
DBD::Sybase
BCP Tool
Bug Tracker
Mailing List Archive
Downloads Directory
FAQs
Sybase on Linux FAQ
Sybperl FAQ
Personal
Michael Peppler's resume

sybperl-l Archive

Up    Prev    Next    

From: JMiller at pressherald dot com
Subject: RE: memory problems - SOLVED
Date: May 6 2002 4:50PM

thank you all for your responses - 

I upgraded perl to 5.6.1 and sybperl to 2.14.
I had tried to upgrade in the past but could never
get it to compile - a newer gcc solved that problem.
my script now takes up about 15 MB - instead of 200+ !!!     :) 

thanks again!

JJM


> -----Original Message-----
> From:	Michael Peppler [SMTP:mpeppler@peppler.org]
> Sent:	Saturday, May 04, 2002 6:19 PM
> To:	SybPerl Discussion List
> Subject:	RE: memory problems
> 
> On Fri, 2002-05-03 at 08:38, JMiller@pressherald.com wrote:
> > 
> > Hi Michael - 
> > thanks for responding - we are running perl 5.003 
> 
> That's pretty old. If you get the chance - upgrade to 5.6.1 (and the
> latest sybperl).
> 
> > %pgs = "";
> > %pages = "";
> 
> As we mentioned elsewhere - this should be
> 
> %pgs = ();
> %pages = ();
> 
> However, below you use $pages as an array, not a hash, so you should
> really write:
> 
> @pages = ();
> 
> > $numresp= 0;
> > $dbh->ct_execute($query);
> >         while ($dbh->ct_results($res_type) == CS_SUCCEED) {
> >                 next unless $dbh->ct_fetchable($res_type);
> > 		while ((%pgs) = $dbh->ct_fetch(CS_TRUE)) {
> > 			$pages[$numresp]{id} = $pgs{id};
> 
> You're working too hard:
> 
> 			push(@pages, {%pgs});
>                  }
>           }
> 
> $numresp = scalar(@pages);
> 
> > now here is the meat of the script - it loops through %pages and gets
> text
> > data based on that
> > and then save the data to a file and then go back and get the next row
> from
> > the db.
> > 
> > $numpages = @pages;
> > $loop = 0;
> > while ( $loop < $numpages ) {
> > 
> > 	$dbh->ct_sql("set textsize 2048000");
> > 	$currentpage = $pages[$loop]{id};
> > 	$query1 = qq{
> > 		select id, name, text, layoutdt
> > 		from story, storypages where story.id = storypages.storyid
> > 		and storypages.pagesid = $currentpage
> > 		and name not like "ag.%" and name not like "%folio%"
> > 		};
> > 
> > 	$dbh->ct_execute($query1);
> >       while ($dbh->ct_results($res_type) == CS_SUCCEED) {
> >       	next unless $dbh->ct_fetchable($res_type);
> > 		while ((%story) = $dbh->ct_fetch(CS_TRUE)) {
> > 
> > 			@spline = "";
> 
> That should be          @spline = ();
> 
> 
> > thanks for any help or suggestions you might have.
> 
> Actually I strongly suggest that you run this script with the -w flag,
> and with 
> 
> use strict;
> 
> at the top. This will force you to declare your variables as lexicals
> (my $foo), which in turn will tell perl to free them when they go out of
> scope. This is particularly important for the second part of the script
> where you fetch large chunks of data, but perl probably never realizes
> that you are done with them.
> 
> Michael
> 
> 
> > > -----Original Message-----
> > > From:	Michael Peppler [SMTP:mpeppler@peppler.org]
> > > Sent:	Friday, May 03, 2002 10:37 AM
> > > To:	SybPerl Discussion List
> > > Subject:	Re: memory problems
> > > 
> > > On Fri, 2002-05-03 at 06:16, JMiller@pressherald.com wrote:
> > > > 
> > > > Hi, 
> > > > I have a sybperl (ver 2.06) script that chews up gobs of memory 
> > > > on my Solaris system.  I am retrieving  text fields from sybase.
> > > > I read each result into a hash and process it.  my script will
> > > > reach 250+ MB plus before it runs out of memory.  this was
> > > > exacerbated when I had to bump up the textsize on the query
> > > > to 2048000.  I have tried to reinitialize the hash ( %hash = ""; ) 
> > > > after each fetch but it doesn't help.
> > > 
> > > John Gilmore-Baldwin's comments are on the money:
> > > 
> > > You empty a hash by doing:
> > > 
> > > %hash = ();
> > > 
> > > Reconnecting to Sybase won't help. Sybperl itself *shouldn't* be
> leaking
> > > (although 2.06 is pretty old, so it's not impossible that it has some
> > > problems in that respect).
> > > 
> > > (as an aside, you disconnect from Sybase in a sybperl script either by
> > > calling $dbh->dbclose() [Sybase::DBlib] or by having the handle go out
> > > of scope/become undefined: $dbh = undef;)
> > > 
> > > Is there any way for you to post a section of the script that
> > > illustrates the problem? There might be some issue with variables
> > > getting set but never getting freed due to the way they are used.
> > > 
> > > Also I'm guessing that you have a fairly old version of perl, and that
> > > may also contribute to the problem.
> > > 
> > > Michael
> > > -- 
> > > Michael Peppler                              Data Migrations, Inc.
> > > mpeppler@peppler.org           *or*          mpeppler@mbay.net
> > > http://www.mbay.net/~mpeppler
> > > International Sybase User Group: http://www.isug.com
> > 
> -- 
> Michael Peppler                              Data Migrations, Inc.
> mpeppler@peppler.org           *or*          mpeppler@mbay.net
> http://www.mbay.net/~mpeppler
> International Sybase User Group: http://www.isug.com