Michael Peppler
Sybase Consulting
Sybase on Linux
Install Guide for Sybase on Linux
General Sybase Resources
General Perl Resources
BCP Tool
Bug Tracker
Mailing List Archive
Downloads Directory
Sybase on Linux FAQ
Sybperl FAQ
Michael Peppler's resume

sybperl-l Archive

Up    Prev    Next    

From: "Sabherwal, Balvinder K dot " <BKSabherwal at lmus dot leggmason dot com>
Subject: RE: out of memory -- CTlib
Date: Feb 8 2001 4:00PM

You can open a file in write mode and put all your records in that file
using just one variable rather than using an array. once you have completed
fetching the records from the database, close the file u r writing to and
open it back in a read mode and get one record at a time from the file and
do the steps u want to.



> Any sugestions or help would be welcome.
> I am using ct_lib to select large look-up tables from the data base for
> feed
> processing.  I tend to assign all of the info in the data base into a hash
> keyed on a specific value in the database and then read the file line by
> line using the key as a quick lookup. What I am running into however is
> that
> if I try to read in more than 100,000 records or so I get an 'Out of
> Memory!' error.  Is there a more efficient way to read in a large number
> of
> records into a hash table?  Any help or suggestions would be most welcome.
> Thanks 
> Mark
> sub get_records {	
> 	my ($dbh) = @_; 
> 	my ($y, @record, $restype, $sql);
> 	$sql =	"SELECT  a bunch of rows " .
> 		"FROM  table "; 			 
> 	$dbh->ct_execute($sql);
> 	while ($dbh->ct_results($restype) == CS_SUCCEED) {
> 			next unless $dbh->ct_fetchable($restype);
> 		while (@record = $dbh->ct_fetch){
> 			if ( !$record_list{$record[3]}) {
> 				$record_list{$record[3]} = [@record];
> ###############  record[3] is the key value
> 			 } else { 
> 				print OUT
> "=\"$record[3]\",=\"$record[0]\",=\"Duplicate\"";
> 			}		
> 			$y++;
> 			if (!($y % 10000) && ($y !=0)) {
> 				print "$y Records processed at " , `date`;
> 			}
> 		}
> 	}
> 	$dbh->ct_cancel(CS_CANCEL_ALL);
> 	print "read $y total records\n";
> }
IMPORTANT:  The security of electronic mail  sent through the Internet 
is not guaranteed.  Legg Mason therefore recommends that you do not 
send confidential information to us via electronic mail, including social 
security numbers, account numbers, and personal identification numbers.    

Delivery, and timely delivery, of electronic mail is also not 
guaranteed.  Legg Mason therefore recommends that you do not send time-sensitive 
or action-oriented messages to us via electronic mail, including 
authorization to  "buy" or "sell" a security or instructions to conduct any 
other financial transaction.  Such requests, orders or instructions will 
not be processed until Legg Mason can confirm your instructions or 
obtain appropriate written documentation where necessary.