Problem:
As I was doing an RMAN backup, Oracle abruptly quit the backup throwing an ORA-19566 error (claiming I'd have corrupt data blocks in my SYSTEM tablespace).
The alert log said:
Hex dump of (file 10Corrupt block relative dba: 0x0280373c (file 10, block 14140)
Bad check value found during backing up datafile
Data in bad block:
type: 6 format: 2 rdba: 0x0280373c
last change scn: 0x0000.0005606d seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x606d0601
check value in block header: 0x18f3
computed block checksum: 0xff00
Reread of blocknum=14140, file=/path/to/my/orasystem/SYSTEM.DBF. found same corrupt data
Reread of blocknum=14140, file=/path/to/my/orasystem/SYSTEM.DBF. found same corrupt data
Reread of blocknum=14140, file=/path/to/my/orasystem/SYSTEM.DBF. found same corrupt data
Reread of blocknum=14140, file=/path/to/my/orasystem/SYSTEM.DBF. found same corrupt data
Reread of blocknum=14140, file=/path/to/my/orasystem/SYSTEM.DBF. found same corrupt data
I queried V_$DATABASE_BLOCK_CORRUPTION which returned 0 rows. I queried DBA_EXTENTS to see which segment was corrupt. This way I found out the corrupt table: "IDL_UB1$".
To be sure I ran the "dbv" (dbverify) command line tool right after the failed backup. It also claimed I had a corrupt block in my SYSTEM tablespace. Peculiarly enough, after checking all my other tablespaces too, and then checking the SYSTEM tablespace a second time, dbv didn't find any corrupt block. So what was the real issue behind this alleged corrupt block?
I retried the RMAN backup. This time Oracle would tell me that another tablespace had corrupt blocks.
The alert.log again contained errors like "Reread of blocknum=...".
2 days later the linux machine wouldn't boot because of a defect harddisk.
So it turned out these errors had their roots in a semi-defect harddisk.
|