We starting seeing a ton of these waits on an Exadata system:
cell single block physical read
Normally the predominant wait is
cell smart table scan
Simple table full scans were doing the single block reads, whereas they normally would do smart scans.
Oracle support had us run this command to identify cell servers with “quarantined” plans:
cellcli -e list quarantine
They identified a cell server with several of these plans and had us run this command in cellcli:
drop quarantine all
Ideally you would patch up your cell servers, etc to the current release and this would reduce the number of quarantined plans.
Oracle note 1349167.1 contains an example of why a cell server would crash and cause a plan to be quarantined.
Evidently if you have a certain number of quarantined plans (6 I think) this fact becomes “visible” to the database servers and they start using single block reads instead of the multiblock smart scans. Dropping the quarantined plans reduces the number of plans below the threshhold and the database goes back to doing smart scans.