DBCC CHECKDB multiple errors

SQL2K
SP4

Error: 7987, Severity: 22, State: 3
A possible database consistency problem has been detected

Ran DBCC CHECKDB on WV database on both servers (the dbs are not copies
of each other but are based on the same information, 1 is on the
development server, the other on the production server) but received
the same error messages for the same tables in both.  At the end of the
message was:

CHECKDB found 0 allocation errors and 8 consistency errors in database
'WV'.
 repair_allow_data_lo
ss is the minimum repair level for the errors found
by DBCC CHECKDB (WV ).

When I ran DBCC DBREINDEX (as an alternative to allowing data loss) on
the tables indicated, it appeared to fix a few tables that only had 1
consistency error (they no longer show up with inconsistencies) but
there are 2 tables that still show up with multiple errors.

I haven't found a post with all the error messages (8928, 8944, 8964)
and I'm wondering what the best approach to this is to prevent any
data loss.

Ironically, I've never seen msg 8964 before this week, and we
installed SP4 last week, which is supposed to take care of this bug.
Could this have caused the bug?

DBCC results for 'sde.GDB_STRINGDOMAINS'.
Server: Msg 8928, Level 16, State 1, Line 1
Object ID 1648724926, index ID 0: Page (1:155093) could not be
processed. See other errors for details.
Server: Msg 8944, Level 16, State 1, Line 1
Table error: Object ID 1648724926, index ID 0, page (1:155093), row 25.
Test (offsetNull >= BASEOFFSET) failed. Values are 0 and 4.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313603), slot 10, text ID 41680896 is not referenced.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313603), slot 11, text ID 1340932096 is not referenced.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313603), slot 12, text ID 23986176 is not referenced.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313604), slot 0, text ID 96010240 is not referenced.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313604), slot 1, text ID 817430528 is not referenced.
There are 0 rows in 0 pages for object 'sde.GDB_STRINGDOMAINS'.


DBCC results for 'wvowner.f8'.
Server: Msg 8928, Level 16, State 1, Line 1
Object ID 1648724926, index ID 0: Page (1:155093) could not be
processed. See other errors for details.
Server: Msg 8944, Level 16, State 1, Line 1
Table error: Object ID 1648724926, index ID 0, page (1:155093), row 25.
Test (offsetNull >= BASEOFFSET) failed. Values are 0 and 4.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313603), slot 10, text ID 41680896 is not referenced.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313603), slot 11, text ID 1340932096 is not referenced.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313603), slot 12, text ID 23986176 is not referenced.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313604), slot 0, text ID 96010240 is not referenced.
Server: Msg 8964, Level 16, State 1, Line 1
Table error: Object ID 1648724926. The text, ntext, or image node at
page (1:313604), slot 1, text ID 817430528 is not referenced.
There are 1078620 rows in 30728 pages for object 'wvowner.f8'.
CHECKDB found 0 allocation errors and 7 consistency errors in table
'wvowner.f8' (object ID 1648724926).

For both of these tables, CHECKDB has found a row whose structure we cannot
validate (row 25 on page 1:155093).  Because the structure of the rest of
the page is suspect due to this problem, DBCC will not process the page any
further.  The 8964 errors reported are a result of our not processing any of
the data on this page.  Running  REPAIR_ALLOW_DATA_LO
SS will result in this
page being deallocated.  Because it is a data page, you will lose data.
(FYI, error 7987 is the run-time check that is equivalent to the corruption
that CHECKDB is reporting.  It is a new error in SP4.)

The only way to ensure that you don't lose any data is to restore your last
known good database backups, plus any transaction log backups.

> Ironically, I've never seen msg 8964 before this week, and we
> installed SP4 last week, which is supposed to take care of this bug.
> Could this have caused the bug?

I'm not sure I understand this statement.  Which bug are you referring to?

Thanks,
--
Ryan Stonecipher
Microsoft Sql Server Storage Engine, DBCC

This posting is provided "AS IS" with no warranties, and confers no rights.

<naomimsm@gmail.com> wrote in message
news:1122478968.737696.309790@f14g2000cwb.googlegroups.com...
> SQL2K
> SP4
>
> Error: 7987, Severity: 22, State: 3
> A possible database consistency problem has been detected
>
> Ran DBCC CHECKDB on WV database on both servers (the dbs are not copies
> of each other but are based on the same information, 1 is on the
> development server, the other on the production server) but received
> the same error messages for the same tables in both.  At the end of the
> message was:
>
> CHECKDB found 0 allocation errors and 8 consistency errors in database
> 'WV'.
>  repair_allow_data_lo
ss is the minimum repair level for the errors found
> by DBCC CHECKDB (WV ).
>
> When I ran DBCC DBREINDEX (as an alternative to allowing data loss) on
> the tables indicated, it appeared to fix a few tables that only had 1
> consistency error (they no longer show up with inconsistencies) but
> there are 2 tables that still show up with multiple errors.
>
> I haven't found a post with all the error messages (8928, 8944, 8964)
> and I'm wondering what the best approach to this is to prevent any
> data loss.
>
> Ironically, I've never seen msg 8964 before this week, and we
> installed SP4 last week, which is supposed to take care of this bug.
> Could this have caused the bug?
>
> DBCC results for 'sde.GDB_STRINGDOMAINS'.
> Server: Msg 8928, Level 16, State 1, Line 1
> Object ID 1648724926, index ID 0: Page (1:155093) could not be
> processed. See other errors for details.
> Server: Msg 8944, Level 16, State 1, Line 1
> Table error: Object ID 1648724926, index ID 0, page (1:155093), row 25.
> Test (offsetNull >= BASEOFFSET) failed. Values are 0 and 4.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313603), slot 10, text ID 41680896 is not referenced.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313603), slot 11, text ID 1340932096 is not referenced.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313603), slot 12, text ID 23986176 is not referenced.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313604), slot 0, text ID 96010240 is not referenced.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313604), slot 1, text ID 817430528 is not referenced.
> There are 0 rows in 0 pages for object 'sde.GDB_STRINGDOMAINS'.
>
>
> DBCC results for 'wvowner.f8'.
> Server: Msg 8928, Level 16, State 1, Line 1
> Object ID 1648724926, index ID 0: Page (1:155093) could not be
> processed. See other errors for details.
> Server: Msg 8944, Level 16, State 1, Line 1
> Table error: Object ID 1648724926, index ID 0, page (1:155093), row 25.
> Test (offsetNull >= BASEOFFSET) failed. Values are 0 and 4.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313603), slot 10, text ID 41680896 is not referenced.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313603), slot 11, text ID 1340932096 is not referenced.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313603), slot 12, text ID 23986176 is not referenced.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313604), slot 0, text ID 96010240 is not referenced.
> Server: Msg 8964, Level 16, State 1, Line 1
> Table error: Object ID 1648724926. The text, ntext, or image node at
> page (1:313604), slot 1, text ID 817430528 is not referenced.
> There are 1078620 rows in 30728 pages for object 'wvowner.f8'.
> CHECKDB found 0 allocation errors and 7 consistency errors in table
> 'wvowner.f8' (object ID 1648724926).
>

Ryan, we, too, have had DBCC CHECKDB errors, and the errors we have had
seemed to disappear on subsquent runs of DBCC CHECKDB. Why? Does some
automatic fix happen with some types of errors, or is something getting
swept under the rug?

How does one track down the cause of CHECKDB errors?

Can virus protection software (Avast in our case) ever be a cause of DB
corruption?

The error(s) that disappeared are as follows:

[2] Database VMed1: Check Data and Index Linkage...
[Microsoft SQL-DMO (ODBC SQLState: 42000)] Error 8928: [Microsoft]&#
91;ODBC SQL
Server Driver][SQL Server]Object ID 2050874423, index ID 2: Page (1:2134
9)
could not be processed. See other errors for details.
[Microsoft][ODBC SQL Server Driver][SQL Server]Table error: Obje
ct ID
2050874423, index ID 2, page (1:21349), row 339. Test (ColumnOffsets <=
(nextRec - pRec)) failed. Values are 13 and 3.
[Microsoft][ODBC SQL Server Driver][SQL Server]CHECKDB found 0 a
llocation
errors and 2 consistency errors in table 'NodeRelation' (object ID
2050874423).
[Microsoft][ODBC SQL Server Driver][SQL Server]CHECKDB found 0 a
llocation
errors and 2 consistency errors in database 'VMed1'.
[Microsoft][ODBC SQL Server Driver][SQL  Server]repair_allow_
data
_loss is the
minimum repair level for the errors found by DBCC CHECKDB (VMed1 ).

Thanks,

Randy Neall


"Ryan Stonecipher [MSFT]" <ryanston@microsoft.com> wrote in message
news:eihMfCukFHA.3380@TK2MSFTNGP12.phx.gbl...
> For both of these tables, CHECKDB has found a row whose structure we
 cannot
> validate (row 25 on page 1:155093).  Because the structure of the rest of
> the page is suspect due to this problem, DBCC will not process the page
any
> further.  The 8964 errors reported are a result of our not processing any
of
> the data on this page.  Running  REPAIR_ALLOW_DATA_LO
SS will result in this
> page being deallocated.  Because it is a data page, you will lose data.
> (FYI, error 7987 is the run-time check that is equivalent to the
 corruption
> that CHECKDB is reporting.  It is a new error in SP4.)
>
> The only way to ensure that you don't lose any data is to restore your
last
> known good database backups, plus any transaction log backups.
> 
>
> I'm not sure I understand this statement.  Which bug are you referring to?
>
> Thanks,
> --
> Ryan Stonecipher
> Microsoft Sql Server Storage Engine, DBCC
>
> This posting is provided "AS IS" with no warranties, and confers no
rights.
>
> <naomimsm@gmail.com> wrote in message
> news:1122478968.737696.309790@f14g2000cwb.googlegroups.com... 
>
>

Randy,

There are a couple of possibilities:  the first is hardware, the second is
normal workload processing.

In some circumstances, corruption problems can exist in memory only.  For
instance, bad disk firmware can generate single-byte or single-bit
discrepancies when a page is read from disk.  If CHECKDB is run on this
data, then it can generate corruption errors.  Subsequent reads of the same
page may come back just fine.  In SQL Server 2005, we've introduced a page
checksum feature that will help to identify these issues when they happen.
Bad memory can also cause problems like this...

In other cases, true corruption can exist in your database, but your normal
maintenance plans (that rebuild indexes, for instance) could eliminate the
corruption by regenerating the index pages during the rebuild.  (The index
rebuild "fix" really only works for nonclustered indexes; if you have hard
corruption in your data pages, then a rebuild won't fix the issue.)

In your case, either is possible.  Since this was a nonclustered index,
could a maintenance rebuild have fixed your problem?  Have you run complete
hardware diagnostics on your IO stack?

--
Ryan Stonecipher
Microsoft Sql Server Storage Engine, DBCC

This posting is provided "AS IS" with no warranties, and confers no rights.

"Randolph Neall" < randolphneall@veraci
tycomputing.com> wrote in message
news:OK2GwxukFHA.3148@TK2MSFTNGP09.phx.gbl...
> Ryan, we, too, have had DBCC CHECKDB errors, and the errors we have had
> seemed to disappear on subsquent runs of DBCC CHECKDB. Why? Does some
> automatic fix happen with some types of errors, or is something getting
> swept under the rug?
>
> How does one track down the cause of CHECKDB errors?
>
> Can virus protection software (Avast in our case) ever be a cause of DB
> corruption?
>
> The error(s) that disappeared are as follows:
>
> [2] Database VMed1: Check Data and Index Linkage...
> [Microsoft SQL-DMO (ODBC SQLState: 42000)] Error 8928: [Microsoft]
[ODBC
> SQL
> Server Driver][SQL Server]Object ID 2050874423, index ID 2: Page (1:21
349)
> could not be processed. See other errors for details.
> [Microsoft][ODBC SQL Server Driver][SQL Server]Table error: Ob
ject ID
> 2050874423, index ID 2, page (1:21349), row 339. Test (ColumnOffsets <=
> (nextRec - pRec)) failed. Values are 13 and 3.
> [Microsoft][ODBC SQL Server Driver][SQL Server]CHECKDB found 0
 allocation
> errors and 2 consistency errors in table 'NodeRelation' (object ID
> 2050874423).
> [Microsoft][ODBC SQL Server Driver][SQL Server]CHECKDB found 0
 allocation
> errors and 2 consistency errors in database 'VMed1'.
> [Microsoft][ODBC SQL Server Driver][SQL  Server]repair_allow_
da
ta_loss is
> the
> minimum repair level for the errors found by DBCC CHECKDB (VMed1 ).
>
> Thanks,
>
> Randy Neall
>
>
> "Ryan Stonecipher [MSFT]" <ryanston@microsoft.com> wrote in message
> news:eihMfCukFHA.3380@TK2MSFTNGP12.phx.gbl... 
> cannot 
> any 
> of 
> corruption 
> last 
> rights. 
>
>

Thanks much, Ryan. In fact we ARE rebuilding indexes every night (possibly
overkill), and this error occurs AFTER that rebuild, I mean immediately
after (30 minutes) in the wee hours when no one is working. So it appears
that we got the problem as a direct result of the rebuild itself.
Apparently, the following night, the rebuild happens again, and that one
evidently fixes it.

We will check out our hardware with whatever diagnostics are available.

The errors we have seen tend to always be on the same table and possibly the
same index. I'm wondering if some table/index designs are more vulnerable
and unstable than others. This particular table has about seven integers
with overlapping indexes on all columns, including a single-column identity
clustered index and an alternate key on  six of the integer columns, plus
others.

Thanks,

Randy Neall

I would strongly recommend that you open up a case with Microsoft support in
this case (http://support.microsoft.com), as it doesn't sound at all like
normal behavior.  To answer your question, there aren't any particular index
structures that are more susceptible to this problem than others.  I would
get in touch with the experts in PSS and have them help you narrow down the
problem.

Thanks,
--
Ryan Stonecipher
Microsoft Sql Server Storage Engine, DBCC

This posting is provided "AS IS" with no warranties, and confers no rights.

"Brandon Owensby" <123@abc.com> wrote in message
news:Og2k4B5kFHA.1148@TK2MSFTNGP12.phx.gbl...
> Thanks much, Ryan. In fact we ARE rebuilding indexes every night (possibly
> overkill), and this error occurs AFTER that rebuild, I mean immediately
> after (30 minutes) in the wee hours when no one is working. So it appears
> that we got the problem as a direct result of the rebuild itself.
> Apparently, the following night, the rebuild happens again, and that one
> evidently fixes it.
>
> We will check out our hardware with whatever diagnostics are available.
>
> The errors we have seen tend to always be on the same table and possibly
> the same index. I'm wondering if some table/index designs are more
> vulnerable and unstable than others. This particular table has about seven
> integers with overlapping indexes on all columns, including a
> single-column identity clustered index and an alternate key on  six of the
> integer columns, plus others.
>
> Thanks,
>
> Randy Neall
>
>
>