|
Home > Archive > dBASE Questions and Answers > November 2005 > Maximum number of records
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Maximum number of records
|
|
| Pieter van Heerden 2005-11-14, 7:23 am |
| I work in Windows XP Pro and the newest dBase.
dBase information ssys that theoretically the program can manage "about" 2 x 10^9 records. This is limited by the BDE to about 1 billion (1 x 10^9(?)) records.
Because I need big file capacity for working with weather data, I have trid to build a big file to check on a maximum size under my circumstances by repeatedly appending records from one file (1271 records) to a newly created unindexed file through the dB
ase UpdateSet class. Source = File1.dbf; Destination = File2.dbf.
This works fine until the destination file reaches the following apparent maxima:
File size: about 2.1 GB
Records: about 2.3 x 10^6
At about that stage I get an error message telling me that the "Data structure corruption" occured and appendng new records from File1 to File2 stops.
This is very far below the BDE maximum. How do I increase the capacity to at least strive for the BDE capacity?
| |
| Marilyn Price 2005-11-14, 7:23 am |
| In article <t8t#TYQ6FHA.1232@news-server>, psvh@mweb.co.za says...
> I work in Windows XP Pro and the newest dBase.
>
> dBase information ssys that theoretically the program can manage
> "about" 2 x 10^9 records. This is limited by the BDE to about
> 1 billion (1 x 10^9(?)) records.
The first item is bytes (2 GB). The second is records (1 gig). The
limit is whichever is reached first.
> This works fine until the destination file reaches the following apparent maxima:
>
> File size: about 2.1 GB
> Records: about 2.3 x 10^6
>
> At about that stage I get an error message telling me that the
> "Data structure corruption" occured and appendng new records from
> File1 to File2 stops.
>
> This is very far below the BDE maximum. How do I increase
> the capacity to at least strive for the BDE capacity?
No, it is not. It has reached the filesize maximum of 2 GB.
--
Marilyn Price
M. P. Data
| |
| Jean-Pierre Martel 2005-11-14, 9:24 am |
| In article <t8t#TYQ6FHA.1232@news-server>, psvh@mweb.co.za says...
>
> How do I increase the capacity to at least strive for the BDE capacity?
If you need to work on tables bigger than what the BDE allows, the
solution could be to use dBASE as a front-end to access only a subset of
the data stored in a huge client/server database like Interbase, Oracle,
Informix, Sybase, MSSQL, or DB2, for example.
Jean-Pierre Martel, editor
The dBASE Developers Bulletin
Blue Star dBASE Plus Core Concepts Graduate
| |
| Pieter van Heerden 2005-11-17, 3:24 am |
| Marilyn Price Wrote:
> In article <t8t#TYQ6FHA.1232@news-server>, psvh@mweb.co.za says...
>
> The first item is bytes (2 GB). The second is records (1 gig). The
> limit is whichever is reached first.
>
>
> No, it is not. It has reached the filesize maximum of 2 GB.
>
> --
> Marilyn Price
> M. P. Data
Thanks Marilyn. I was blinded by the FAT32 max file size of 4 GB (and bigger on the NTFS system). An unfair question (perhaps): When is dBase expected to lift the 2 GB file size limit, or is that also a BDE limitation?
Interesting observations:
1. In Ken Mayer's "The dBase Book" max file size indicated for DBF files is 1 Gb, so there seems to have been an increase in allowed size since the book was written! Max file size for Paradox tables is given as double that of DBF files.
2. Having an index active when appending large files, apart from slowing the process down, results in the program "bombing out" far below (at about 10% of allowed max in my case) the max level of file size and, I assume, will also be the case for allowed
records. In the case of appending large amounts of data, the solution is of course to append to tables that do not have an index operating. Rebuilding an index afterwards takes care of this problem.
| |
| David Kerber 2005-11-17, 9:24 am |
| In article <zsxO36z6FHA.1276@news-server>, psvh@mweb.co.za says...
....
>
> Thanks Marilyn. I was blinded by the FAT32 max file size of 4 GB (and bigger on the NTFS system). An unfair question (perhaps): When is dBase expected to lift the 2 GB file size limit, or is that also a BDE limitation?
It is a BDE limitation.
>
> Interesting observations:
>
> 1. In Ken Mayer's "The dBase Book" max file size indicated for DBF files is 1 Gb, so there seems to have been an increase in allowed size since the book was written! Max file size for Paradox tables is given as double that of DBF files.
No, the limit has been the same since at least the early 90's: 1G
records or 2G Bytes, whichever comes first (and it's always the 2G Bytes
which comes first). If Ken says otherwise, it's a typo.
....
--
Remove the ns_ from if replying by e-mail (but keep posts in the
newsgroups if possible).
| |
| Charles Crume 2005-11-17, 11:23 am |
|
"David Kerber" < ns_dkerber@ns_Warren
RogersAssociates.com> wrote in message
news:MPG. 1de6416dcf72fcec989d
67@news.dbase.com...
> In article <zsxO36z6FHA.1276@news-server>, psvh@mweb.co.za says...
>
> ...
>
>
> It is a BDE limitation.
>
>
> No, the limit has been the same since at least the early 90's: 1G
> records or 2G Bytes, whichever comes first (and it's always the 2G Bytes
> which comes first). If Ken says otherwise, it's a typo.
And the only way you can stuff 1G of records into a dbf is if the table
contains a single 1 byte logical field -- not much use in real life (I
forget why, but each record uses 1 byte for something or other, ergo 1G of 1
byte logical records = 2G size file -- the max).
Charles...
| |
| David Kerber 2005-11-17, 1:23 pm |
| In article <lY5CRj56FHA.1520@news-server>,
NOccsSPAM@charlescru
mesoftware.com says...
>
> "David Kerber" < ns_dkerber@ns_Warren
RogersAssociates.com> wrote in message
> news:MPG. 1de6416dcf72fcec989d
67@news.dbase.com...
>
> And the only way you can stuff 1G of records into a dbf is if the table
> contains a single 1 byte logical field -- not much use in real life (I
> forget why, but each record uses 1 byte for something or other, ergo 1G of 1
> byte logical records = 2G size file -- the max).
That extra byte is the "Deleted" flag. Then add the header info which
takes up some small amount of space, and no matter what you do, you
can't *quite* get 1G records in 2G Bytes. I suppose I should try some
day to see just how many I could get, just for grins. It shouldn't take
more than a few hours; I did one with around 150M records for testing a
few months ago.
--
Remove the ns_ from if replying by e-mail (but keep posts in the
newsgroups if possible).
| |
| Gerald Lightsey 2005-11-19, 3:23 am |
| On Thu, 17 Nov 2005 01:18:24 -0500, in the dbase.how-to group, Pieter
van Heerden said...
> 2. Having an index active when appending large files, apart from
> slowing the process down, results in the program "bombing out" far
> below (at about 10% of allowed max in my case) the max level of file
> size and, I assume, will also be the case for allowed records. In
> the case of appending large amounts of data, the solution is of
> course to append to tables that do not have an index operating.
> Rebuilding an index afterwards takes care of this problem.
Have you tried using the "REINDEX" qualifier within the command?
Gerald
| |
| David Kerber 2005-11-29, 8:24 pm |
| In article <MPG. 1de695d339f4ea15989d
68@news.dbase.com>,
ns_dkerber@ns_Warren
RogersAssociates.com says...
> In article <lY5CRj56FHA.1520@news-server>,
> NOccsSPAM@charlescru
mesoftware.com says...
>
> That extra byte is the "Deleted" flag. Then add the header info which
> takes up some small amount of space, and no matter what you do, you
> can't *quite* get 1G records in 2G Bytes. I suppose I should try some
> day to see just how many I could get, just for grins. It shouldn't take
> more than a few hours; I did one with around 150M records for testing a
> few months ago.
Here are my results using dBase IV:
I created a .dbf with one logical record. The file size before adding
any records was 66 bytes, so I estimated that I would get 1G - 33
records. Then in a continuous loop I did an APPEND BLANK. It took
several hours (about six, though I didn't keep close track) to get to
the point of 1,000,000,000 records, and surprisingly, it kept going.
Apparently for dBase, 1 billion means 2^30 (1,073,741,824), not 10^9
(1,000,000,000), even though the manual says otherwise.
<A little later>:
This is really interesting: my dBase IV test routine has just exceeded
the dBase IV .dbf file limits: it has 2217025536 bytes, and 1109626410
records, and is still growing.
I'll let you all know how this turns out, if my computer survives
<Grin>.
--
Remove the ns_ from if replying by e-mail (but keep posts in the
newsgroups if possible).
|
|
|
|
|