PEPPLER.ORG
Michael Peppler
Sybase Consulting
Menu
Home
Sybase on Linux
Install Guide for Sybase on Linux
General Sybase Resources
General Perl Resources
Freeware
Sybperl
Sybase::Simple
DBD::Sybase
BCP Tool
Bug Tracker
Mailing List Archive
Downloads Directory
FAQs
Sybase on Linux FAQ
Sybperl FAQ
Personal
Michael Peppler's resume

sybperl-l Archive

Up    Prev    Next    

From: Michael Peppler <mpeppler at peppler dot org>
Subject: RE: memory problems
Date: May 4 2002 10:18PM

On Fri, 2002-05-03 at 08:38, JMiller@pressherald.com wrote:
> 
> Hi Michael - 
> thanks for responding - we are running perl 5.003 

That's pretty old. If you get the chance - upgrade to 5.6.1 (and the
latest sybperl).

> %pgs = "";
> %pages = "";

As we mentioned elsewhere - this should be

%pgs = ();
%pages = ();

However, below you use $pages as an array, not a hash, so you should
really write:

@pages = ();

> $numresp= 0;
> $dbh->ct_execute($query);
>         while ($dbh->ct_results($res_type) == CS_SUCCEED) {
>                 next unless $dbh->ct_fetchable($res_type);
> 		while ((%pgs) = $dbh->ct_fetch(CS_TRUE)) {
> 			$pages[$numresp]{id} = $pgs{id};

You're working too hard:

			push(@pages, {%pgs});
                 }
          }

$numresp = scalar(@pages);

> now here is the meat of the script - it loops through %pages and gets text
> data based on that
> and then save the data to a file and then go back and get the next row from
> the db.
> 
> $numpages = @pages;
> $loop = 0;
> while ( $loop < $numpages ) {
> 
> 	$dbh->ct_sql("set textsize 2048000");
> 	$currentpage = $pages[$loop]{id};
> 	$query1 = qq{
> 		select id, name, text, layoutdt
> 		from story, storypages where story.id = storypages.storyid
> 		and storypages.pagesid = $currentpage
> 		and name not like "ag.%" and name not like "%folio%"
> 		};
> 
> 	$dbh->ct_execute($query1);
>       while ($dbh->ct_results($res_type) == CS_SUCCEED) {
>       	next unless $dbh->ct_fetchable($res_type);
> 		while ((%story) = $dbh->ct_fetch(CS_TRUE)) {
> 
> 			@spline = "";

That should be          @spline = ();


> thanks for any help or suggestions you might have.

Actually I strongly suggest that you run this script with the -w flag,
and with 

use strict;

at the top. This will force you to declare your variables as lexicals
(my $foo), which in turn will tell perl to free them when they go out of
scope. This is particularly important for the second part of the script
where you fetch large chunks of data, but perl probably never realizes
that you are done with them.

Michael


> > -----Original Message-----
> > From:	Michael Peppler [SMTP:mpeppler@peppler.org]
> > Sent:	Friday, May 03, 2002 10:37 AM
> > To:	SybPerl Discussion List
> > Subject:	Re: memory problems
> > 
> > On Fri, 2002-05-03 at 06:16, JMiller@pressherald.com wrote:
> > > 
> > > Hi, 
> > > I have a sybperl (ver 2.06) script that chews up gobs of memory 
> > > on my Solaris system.  I am retrieving  text fields from sybase.
> > > I read each result into a hash and process it.  my script will
> > > reach 250+ MB plus before it runs out of memory.  this was
> > > exacerbated when I had to bump up the textsize on the query
> > > to 2048000.  I have tried to reinitialize the hash ( %hash = ""; ) 
> > > after each fetch but it doesn't help.
> > 
> > John Gilmore-Baldwin's comments are on the money:
> > 
> > You empty a hash by doing:
> > 
> > %hash = ();
> > 
> > Reconnecting to Sybase won't help. Sybperl itself *shouldn't* be leaking
> > (although 2.06 is pretty old, so it's not impossible that it has some
> > problems in that respect).
> > 
> > (as an aside, you disconnect from Sybase in a sybperl script either by
> > calling $dbh->dbclose() [Sybase::DBlib] or by having the handle go out
> > of scope/become undefined: $dbh = undef;)
> > 
> > Is there any way for you to post a section of the script that
> > illustrates the problem? There might be some issue with variables
> > getting set but never getting freed due to the way they are used.
> > 
> > Also I'm guessing that you have a fairly old version of perl, and that
> > may also contribute to the problem.
> > 
> > Michael
> > -- 
> > Michael Peppler                              Data Migrations, Inc.
> > mpeppler@peppler.org           *or*          mpeppler@mbay.net
> > http://www.mbay.net/~mpeppler
> > International Sybase User Group: http://www.isug.com
> 
-- 
Michael Peppler                              Data Migrations, Inc.
mpeppler@peppler.org           *or*          mpeppler@mbay.net
http://www.mbay.net/~mpeppler
International Sybase User Group: http://www.isug.com