sybperl-l Archive

From: JMiller at pressherald dot com
Subject: RE: memory problems
Date: May 3 2002 3:38PM

Hi Michael - 
thanks for responding - we are running perl 5.003 


here is my connection piece

	my $dbh = new Sybase::CTlib $user, $password, $server, $g_program, {
		CON_PROPS => {
			CS_HOSTNAME		=> "$g_hostname",
			CS_PACKETSIZE		=> 2048,
			CS_SEC_ENCRYPTION	=> CS_TRUE
		}
	};

I then get a list of items from sybase  that I want to process and store
them in the %pages hash.

$query = qq{
	select distinct pages.id,pages.sectionpagenum,newsplan.publication,
	newsplan.rundate,newsplan.edition, section.name \"sectionname\"  
	from newsplan,section,pages,storypages
	where newsplan.name like \"%$newsplan%\" 
	and newsplan.id = section.planid
	and section.id = pages.sectionid 
	and pages.id = storypages.pagesid
	and pages.pagedone <> \"\"
	order by section.name, pages.sectionpagenum,newsplan.edition
                };

%pgs = "";
%pages = "";

$numresp= 0;
$dbh->ct_execute($query);
        while ($dbh->ct_results($res_type) == CS_SUCCEED) {
                next unless $dbh->ct_fetchable($res_type);
		while ((%pgs) = $dbh->ct_fetch(CS_TRUE)) {
			$pages[$numresp]{id} = $pgs{id};
			$pages[$numresp]{sectionpagenum} =
$pgs{sectionpagenum};
			$pages[$numresp]{publication} = $pgs{publication};
			$pages[$numresp]{rundate} = $pgs{rundate};
			$pages[$numresp]{edition} = $pgs{edition};
			$pages[$numresp]{sectionname} = $pgs{sectionname};
			$numresp++;
		}
	  }


now here is the meat of the script - it loops through %pages and gets text
data based on that
and then save the data to a file and then go back and get the next row from
the db.

$numpages = @pages;
$loop = 0;
while ( $loop < $numpages ) {

	$dbh->ct_sql("set textsize 2048000");
	$currentpage = $pages[$loop]{id};
	$query1 = qq{
		select id, name, text, layoutdt
		from story, storypages where story.id = storypages.storyid
		and storypages.pagesid = $currentpage
		and name not like "ag.%" and name not like "%folio%"
		};

	$dbh->ct_execute($query1);
      while ($dbh->ct_results($res_type) == CS_SUCCEED) {
      	next unless $dbh->ct_fetchable($res_type);
		while ((%story) = $dbh->ct_fetch(CS_TRUE)) {

			@spline = "";
			@spline = split(/\012/,$story{layoutdt});
			$newtext = "";
			$newheadline = "";
			$newcaption = "";
			$newmugline = "";
			$newquickread = "";
			$newnuggetbox = "";
			##$goodtext = 0;
			$curline = 0;
			$numlines = @spline;
			while ( $numlines > $curline ) {
				if ( $spline[$curline] =~ /\007text\007/i )
{
					$newtext =
$newtext.$spline[$curline];
				}
				elsif ( $spline[$curline] =~
/\007headline\007/i ) {
					$newheadline =
$newheadline.$spline[$curline];
				}
			++$curline;
			}
			@spline = "";;


$newstoryid = "$story{id}"."001";

		open(STORYFILE,">$outdir/$storyname.txt") || abort("could
not open output save file file $outdir/$storyname\n $!");
		print STORYFILE
"{PUBLICATION}\n$pages[$loop]{publication}\n{DATE}\n$pages[$loop]{rundate}\n
{SDATE}\n$sdate\n{EDITION}\n$pages[$loop]{edition}\n{SECTION}\n$pages[$loop]
{section}\n{PAGE}\n$pagenumber\n{STORYID}\n$newstoryid\n{SLUGNAME}\n$story{n
ame}\n{IMAGES}\n@graphics\n{LAYOUT}\n$newtext\n$newheadline\n$newcaption\n$n
ewmugline\n$newquickread\n$newnuggetbox\n{TEXT2}\n$story{text}\n";
		print LOG "Story -  $story{name} ORIGINAL ID - $story{id}
newstoryid - $newstoryid
:$pages[$loop]{rundate}:$pages[$loop]{edition}:$pages[$loop]{sectionname}:$p
ages[$loop]{sectionpagenum}\n";
		close(STORYFILE);
		
		}
	$story{layoutdt} = "";
	%story = (); 
	%story = undef; 
	}
	$loop++;
}


thanks for any help or suggestions you might have.

John Miller



> -----Original Message-----
> From:	Michael Peppler [SMTP:mpeppler@peppler.org]
> Sent:	Friday, May 03, 2002 10:37 AM
> To:	SybPerl Discussion List
> Subject:	Re: memory problems
> 
> On Fri, 2002-05-03 at 06:16, JMiller@pressherald.com wrote:
> > 
> > Hi, 
> > I have a sybperl (ver 2.06) script that chews up gobs of memory 
> > on my Solaris system.  I am retrieving  text fields from sybase.
> > I read each result into a hash and process it.  my script will
> > reach 250+ MB plus before it runs out of memory.  this was
> > exacerbated when I had to bump up the textsize on the query
> > to 2048000.  I have tried to reinitialize the hash ( %hash = ""; ) 
> > after each fetch but it doesn't help.
> 
> John Gilmore-Baldwin's comments are on the money:
> 
> You empty a hash by doing:
> 
> %hash = ();
> 
> Reconnecting to Sybase won't help. Sybperl itself *shouldn't* be leaking
> (although 2.06 is pretty old, so it's not impossible that it has some
> problems in that respect).
> 
> (as an aside, you disconnect from Sybase in a sybperl script either by
> calling $dbh->dbclose() [Sybase::DBlib] or by having the handle go out
> of scope/become undefined: $dbh = undef;)
> 
> Is there any way for you to post a section of the script that
> illustrates the problem? There might be some issue with variables
> getting set but never getting freed due to the way they are used.
> 
> Also I'm guessing that you have a fairly old version of perl, and that
> may also contribute to the problem.
> 
> Michael
> -- 
> Michael Peppler                              Data Migrations, Inc.
> mpeppler@peppler.org           *or*          mpeppler@mbay.net
> http://www.mbay.net/~mpeppler
> International Sybase User Group: http://www.isug.com