Adaptive Optimization Limitation Example

I’ve been reading up on Oracle 12c to get certified and to help advise my company on potential uses for the new version.  I’ve been looking forward to researching the new Adaptive Optimization features because it makes so much sense that the database should change its plans when it finds differences between the expected number of rows each part of a plan sees and the actual number of rows.

I’ve written blog posts in the past about limitations of the optimizer related to its ability to determine the number of rows (cardinality) that steps in a plan would see.  I took the scripts from some of these and ran them on a 12c instance to see if the new features would cause any of the inefficient plans to change to the obvious efficient plans.

Sadly, none of my examples ran differently on 12c.  I don’t doubt that there are examples that run better because of the new features but the ones I constructed earlier didn’t see any improvement.  So, I thought I would blog about one such example.

Here is the original blog post with an example run on 11.2 Oracle: url

Here is the same script run on 12c: zip

Here is the query with the bad plan:

select B.DIVNUM 
from DIVISION A,SALES B 
where a.DIVNUM=B.DIVNUM and                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
A.DIVNAME='Mayberry'                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    

Plan hash value: 480645376                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

--------------------------------------------------------------------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
| Id  | Operation          | Name     | Rows  | Bytes | Cost (%CPU)|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
--------------------------------------------------------------------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
|   0 | SELECT STATEMENT   |          |       |       |   421 (100)|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
|*  1 |  HASH JOIN         |          |   500K|  8300K|   421   (2)|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
|*  2 |   TABLE ACCESS FULL| DIVISION |     1 |    14 |     2   (0)|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
|   3 |   TABLE ACCESS FULL| SALES    |  1000K|  2929K|   417   (1)|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
--------------------------------------------------------------------

There is only 1 SALES row that has the DIVNUM associated with DIVNAME=’Mayberry’. There 1,000,001 SALES rows  and there is an index on SALES.DIVNUM so an index scan would be the most efficient access method and a nested loops join the most efficient join method. But the 12c optimizer chooses a hash join and full table scan instead.

According to the 12c SQL Tuning manual there are two types of Adaptive Optimization that might help in this case: Adaptive Plans and Adaptive Statistics.  I tried to tweak my test script to get Adaptive Statistics to kick in by commenting out the dbms_stats calls but it didn’t help.  Also, I tried running the query several times in a row but it never changed plan.

I can see why Adaptive Plans wouldn’t work.  How long will it let the full scan of SALES go before it decides to switch to a nested loops join with an index scan?  If it gets half way through the table it is too late.  So, I’m not sure how Adaptive Plans could change the plan on the fly when it expects a lot of rows and only finds a few.

On the Adaptive Statistics I guess this is just a case that it still can’t handle.  I guess it is like a histogram across joins case that would be pretty complex to solve in general.

Anyway, this all reminds me of when I first learned about histograms.  I got all excited that histograms would solve all my query performance problems and then came crashing down to earth when I realized it wasn’t the case.  I think the analogy fits.  Histograms improved cardinality estimates and can help in certain cases.  I think the new adaptive features will help improve plans by using real cardinality figures where it can, but they aren’t a cure-all.

I’m not sure that getting cardinality right is a solvable problem in the general case.  You have to have a fast optimizer so there are limits to how much it can do.

I ran all this as the user SYSTEM on the base 12c 64 bit Linux install with all the default parameters unchanged on the standard default database.

– Bobby

Posted in Uncategorized | 2 Comments

New Features Guide Highlights

I just finished reading the main section of the Oracle 12c New Features Guide.  I read pages 17-107 of the version I have – Section 1 titled “Oracle Database 12c Release 1 (12.1) New Features”.  I underlined and asterisked the things that stuck out as interesting in this pass and thought I would include them in this post:

1.1.4.7 New PL/SQL DBMS_UTILITY.EXPAND_SQL_TEXT Procedure

Expands views into SQL automatically.

1.1.6.1 Default Values for Columns Based on Oracle Sequences

Use sequence in column definition for default values.

1.1.6.4 Increased Size Limit for VARCHAR2, NVARCHAR2, and RAW 
Data Types

32K varchar2 columns

1.1.8.6 JDBC Support for Database Resident Connection Pool

Possible alternative to shared servers

1.2.3.1 Asynchronous Global Index Maintenance for DROP and 
TRUNCATE Partition

Global index not made unusable by partition maintenance

1.2.4.1 Adaptive Query Optimization

Plans change when DB sees that its cardinality estimates
were wrong.

1.2.4.15 Session-Private Statistics for Global Temporary Tables

Gather stats on global temp tables for your session only - cool.

1.3.4.3 SecureFiles is the Default for LOB Storage

Not sure what the means, but good to know that the default 
changed.

1.4.1 Database Consolidation

Subsections 2-6 give some clues to the way the multitenant 
architecture works.

1.4.3.1 Cloning a Database

Sounds similar to Delphix

1.4.4.3 Oracle Data Pump Change Table Compression at Import Time

Imported data can be compressed using HCC on target.

1.5.5.7 Multiple Indexes on Same Set of Columns

Can have different kinds of indexes on same set of columns 
(same order I assume)

1.5.9.1 Active Database Duplication Enhancements

Faster clone of an open database using RMAN

1.6.1.1 Enterprise Manager Database Express

Sounds like a better EM tool - would like to check it out
and review the 2-Day dba manuals which show uses of it.

1.6.3.1 Queryable Patch Inventory

Don't have to do lsniventory to see your patches?

1.8.3.1 Multi-Process Multi-Threaded Oracle

Wondering what platforms this is on and what it really means.

1.9.7.1 Unified Context-Based Database Audit Architecture

Sounds like this may replace some third party tools.  Improved 
auditing.

1.12.1.2 Parallel Upgrade

May speed up upgrade by parallelising

I read all 91 pages but there were sections that didn’t really interest me since they are about features we don’t use such as Spatial.  If you are interested in 12c I encourage you to read this as I did.  I printed out about 10 pages at a time.  It’s only 91 pages so it doesn’t take forever to read it.

– Bobby

Posted in Uncategorized | Leave a comment

Bug 13930580 Example

It looks like we hit 13930580 on one of our systems and I wanted to put some example output showing log file sync times going above 10 milliseconds (10,000 microseconds) after the log writer switches to polling.

Here is the output in the lgwr trace file showing the switch to polling:

*** 2013-12-27 01:50:47.374
kcrfw_update_adaptive_sync_mode: post->poll long#=1 sync#=1 
sync=55336 poll=1913
4 rw=5506 rw+=5506 ack=0 min_sleep=19134

*** 2013-12-27 01:50:47.374
Log file sync switching to polling
Current scheduling delay is 1 usec
Current approximate redo synch write rate is 0 per sec

Here is a report showing the number of log file sync waits and their average time in microseconds:

END_INTERVAL_TIME          number of waits ave microseconds
-------------------------- --------------- ----------------
26-DEC-13 03.00.19.333 PM           976539       2279.06861
26-DEC-13 04.00.43.475 PM           681212       2029.32406
26-DEC-13 05.00.07.975 PM           343932       1575.26284
26-DEC-13 06.00.34.772 PM           163911       1850.74354
26-DEC-13 07.01.00.911 PM            73151       3815.28597
26-DEC-13 08.00.24.463 PM            39304       5038.05427
26-DEC-13 09.00.48.970 PM            32122       5677.00557
26-DEC-13 10.00.13.491 PM           472349       2353.95857
26-DEC-13 11.00.40.521 PM            18679       18655.5294
27-DEC-13 12.00.06.276 AM            19618       17046.2287
27-DEC-13 01.00.30.299 AM            18983       5721.99178
27-DEC-13 02.00.54.261 AM            17724       17106.3415
27-DEC-13 03.00.18.282 AM             9088       25342.7271
27-DEC-13 04.00.42.218 AM            14365          12128.7
27-DEC-13 05.00.06.391 AM            16323       12879.8831
27-DEC-13 06.00.31.379 AM            43758       15326.7298
27-DEC-13 07.00.56.027 AM            83819       14796.2851
27-DEC-13 08.00.20.637 AM           168718       13506.4363
27-DEC-13 09.00.47.262 AM           302827       19116.9491
27-DEC-13 10.00.14.014 AM           480347       19358.6655
27-DEC-13 11.00.41.178 AM           512777       15952.2358
27-DEC-13 12.00.08.220 PM           511091       13799.5512
27-DEC-13 01.00.38.131 PM           576341       10183.4347
27-DEC-13 02.00.06.308 PM           524568       10251.1259

Notice how the average wait time goes above 10,000 microseconds consistently once the log writer switches to polling between 1 and 2 am on 12/27/2013.  I didn’t show all the output but this long log file sync wait time has continued since the switch.

Also, these long log file sync times don’t correspond to long log file parallel write times.  Here are the number and averages of log file parallel write waits for the same time frame:

END_INTERVAL_TIME          number of waits ave microseconds
-------------------------- --------------- ----------------
26-DEC-13 03.00.19.333 PM           902849       1426.66601
26-DEC-13 04.00.43.475 PM           659701       1394.87763
26-DEC-13 05.00.07.975 PM           344245       1294.92401
26-DEC-13 06.00.34.772 PM           166643       1586.64944
26-DEC-13 07.01.00.911 PM            80457       4019.29429
26-DEC-13 08.00.24.463 PM            46409       5580.67827
26-DEC-13 09.00.48.970 PM            69218       5115.20904
26-DEC-13 10.00.13.491 PM           475297       2219.80541
26-DEC-13 11.00.40.521 PM            40943        19405.052
27-DEC-13 12.00.06.276 AM            38835       18160.8073
27-DEC-13 01.00.30.299 AM            24734       6321.38425
27-DEC-13 02.00.54.261 AM            33617       11723.6698
27-DEC-13 03.00.18.282 AM            36469       17485.2614
27-DEC-13 04.00.42.218 AM            19344       6955.27042
27-DEC-13 05.00.06.391 AM            17857       4399.75718
27-DEC-13 06.00.31.379 AM            45098       4923.02763
27-DEC-13 07.00.56.027 AM            83700       3610.39713
27-DEC-13 08.00.20.637 AM           160919       2841.31507
27-DEC-13 09.00.47.262 AM           266405       3523.86855
27-DEC-13 10.00.14.014 AM           384795        3367.5075
27-DEC-13 11.00.41.178 AM           437806       2729.84248
27-DEC-13 12.00.08.220 PM           448261       2442.81012
27-DEC-13 01.00.38.131 PM           511648       1880.74418
27-DEC-13 02.00.06.308 PM           481106       1919.21158

The average I/O time – log file parallel write – is pretty low when the system is active (more than 100,000 waits per hour) – usually less than 4000 microseconds and yet log file sync is always more than 10,000 after the switch to polling. Also, the CPU on the system is consistently less than 30% used so it isn’t a system load issue.

Here are some Oracle support documents related to this issue:

Adaptive Switching Between Log Write Methods can Cause 
'log file sync' Waits (Doc ID 1462942.1)

Waits for "log file sync" with Adaptive Polling vs Post/Wait 
Choice Enabled (Doc ID 1541136.1)

Bug 13930580: LGWR IS BLOCKING SESSIONS

Here is the script I used to get the wait output:

set linesize 32000
set pagesize 1000
set long 2000000000
set longchunksize 1000
set head off;
set verify off;
set termout off;

column u new_value us noprint;
column n new_value ns noprint;

select name n from v$database;
select user u from dual;
set sqlprompt &ns:&us>

set head on
set echo on
set termout on
set trimspool on

UNDEFINE WAITNAME
UNDEFINE MINIMUMWAITS

spool "&ns.&&WAITNAME..log"

column END_INTERVAL_TIME format a26

select sn.END_INTERVAL_TIME,
(after.total_waits-before.total_waits) "number of waits",
(after.time_waited_micro-before.time_waited_micro)/
(after.total_waits-before.total_waits) "ave microseconds",
before.event_name "wait name"
from DBA_HIST_SYSTEM_EVENT before, 
DBA_HIST_SYSTEM_EVENT after,
DBA_HIST_SNAPSHOT sn
where before.event_name='&&WAITNAME' and
after.event_name=before.event_name and
after.snap_id=before.snap_id+1 and
after.instance_number=1 and
before.instance_number=after.instance_number and
after.snap_id=sn.snap_id and
after.instance_number=sn.instance_number and
(after.total_waits-before.total_waits) > &&MINIMUMWAITS
order by after.snap_id;

spool off

I gave log file sync as the WAITNAME and 1 as MINIMUMWAITS for the first output.  I changed WAITNAME to log file parallel write for the second one with 1 still for MINIMUMWAITS.

It looks like there is a new feature in 11.2 that was finally turned on by default in 11.2.0.3.  The work around is to set a hidden parameter to false to turn off the new feature.  Check out the Oracle support docs I listed for details.

– Bobby

p.s. I forgot to mention that when I tested on a test database with and without this new feature the log file sync times were lower with the new feature, as they should be.

With _use_adaptive_log_file_sync=TRUE (feature enabled):

Top 5 Timed Foreground Events

Event Waits Time(s) Avg wait (ms) % DB time Wait Class
log file sync 639,598 3,466 5 86.74 Commit
DB CPU 397 9.93
buffer exterminate 683 14 21 0.36 Other
virtual circuit wait 912 12 13 0.31 Network
SQL*Net message to client 1,293,196 7 0 0.17 Network

With _use_adaptive_log_file_sync=FALSE (disabled as in earlier versions of 11.2):

Top 5 Timed Foreground Events

Event Waits Time(s) Avg wait (ms) % DB time Wait Class
log file sync 639,644 3,553 6 87.31 Commit
DB CPU 367 9.02
buffer exterminate 1,364 28 21 0.69 Other
buffer busy waits 2,679 15 6 0.37 Concurrency
virtual circuit wait 903 13 15 0.32 Network

With the new feature enabled log file sync was 5 milliseconds instead of 6 without it.  So, the new feature does speed up log file sync waits when it is working normally.  But, there must be some bug condition where it degrades to greater than 10 millisecond log file syncs.

Posted in Uncategorized | 5 Comments

Delphix upgrade in twenty minutes

Just got off of a Webex with Delphix support.  They upgraded our Delphix server from version 3.0.3.1 to 3.2.5.1.  It took about twenty minutes.  Pretty nice compared to an Oracle database upgrade I think!

The only thing that took some time was that I had to be sure I had enough space.  It looks like Delphix needs your disk space to be no more than 85% utilized to fully function.  We ended up extending our four 1.5 terabyte luns to 1.9 TB each to give us space.  Then I cleaned up some archive log space by running a snapsync on each source database to completion.  Our system just needed a little TLC to get some space free again.

But, the upgrade itself, running the scripts and rebooting the server, took 20 minutes and was all done by Delphix support for no extra charge.  Sweet.

– Bobby

Posted in Uncategorized | 3 Comments

Cool Picture of Instance/Database from 12c Concepts

I just think this is a cool picture of an Oracle 12c instance and database from Oracle’s 12c Concepts manual (not my own work):

Instance and Database from 12c Concepts

This is from the concepts manual found here: url

– Bobby

Posted in Uncategorized | 2 Comments

Finished reading Oracle Core

Just finished reading the book by Jonathan Lewis titled “Oracle Core: Essential Internals for DBAs and Developers“.  I think I picked it up at the Collaborate 13 conference in Denver last April but haven’t had time (or taken the time) to read it.

Reading a book like Oracle Core can be a challenge because it is pretty dense with example scripts and outputs including dumps in hex.  So, I decided to take the strategy of pushing myself to crash through the book without carefully following every example.  I may only have absorbed about 10% of the material but if I didn’t jam through it I would have gotten 0%!

I picked up Oracle Core because I had read another book by the same author titled “Cost-Based Oracle Fundamentals” which has paid for itself 100 times over in terms of helping me tune queries.  I highly recommend Cost-Based Oracle Fundamentals without reservation.  But, like Oracle Core it can be a challenge to just sit down and read it and follow every SQL example and output.  Probably it would be worth making a first pass focusing on just the English language text and skimming the examples, maybe delving into the examples of most interest.

In the case of Oracle Core I haven’t yet put it to practical use but I’m glad to have at least skimmed through it.  Now I know what’s in it and can refer back to it when needed.

Next I hope to start reading up on Oracle 12c since I plan to get my certification this year.  But, I wanted to finish Oracle Core before I moved on, even if I only read it at a high level.

– Bobby

Posted in Uncategorized | 3 Comments

Top Ten Posts So Far

Just for fun I’ve pasted in a table listing the top 10 most viewed posts on this blog as links and including total number views since this blog began in March 2012.  I based this on WordPress’s statistics so I’m not sure exactly how the blog software collects the numbers but it is fun to get some positive feedback.  Hopefully it means people are getting something out of it.  I’m certainly enjoying putting it together.  Here are the links ordered by views as listed on the right:

cell single block physical read 3,738
REGEXP_LIKE Example 2,822
Finding query with high temp space usage using ASH views 2,232
DBA_HIST_ACTIVE_SESS_HISTORY 2,097
CPU queuing and library cache: mutex X waits 1,801
DBMS_SPACE.SPACE_USAGE 1,748
Resource Manager wait events 1,566
Fast way to copy data into a table 1,166
Delphix First Month 1,074
use_nl and use_hash hints for inner tables of joins 1,047

Anyway, I thought I would list the top ten posts on this blog if you want to read the ones that have the most views and possibly are the most useful.

– Bobby

 

 

 

Posted in Uncategorized | Leave a comment

Exadata compression reduces row chaining > 255 columns

After I did my previous post on how adding a column to a table with more than 255 columns can cause a ton of row chaining the question came up about Exadata and the form of compression we use on our > 255 column tables.  Would we see the same sort of row chaining on our tables that have more than 255 columns, especially after we add new columns and populate them with data?

So, I reran the same scripts from the previous post unchanged and with the addition of QUERY HIGH compression of the tables after they are loaded with data.  Our real tables on our Exadata system are compressed in this way so I wanted to see if, assuming the tables are re-compressed when updates require it, will we still see row chaining.

Bottom line is that after compressing the tables for query high row chaining was almost eliminated even with more than 255 columns and after adding a new column to a table with more than 255 columns.

Here are the test results from the previous post – 16K block, 11.2.0.3 HP-UX Itanium 64-bit.  These are the increase in the table fetch continued row statistic after running a query with the given column in the where clause and forcing a serial full scan.

Table         Column 1 Column 2 Column 3 Last Column-1 Last Column

TEST255C             0        0        0             0           0

TEST256C             0    83333    83333         83333       83333

TEST256CPLUS1        0    83333  1999999       1999999     1999999

Here is on Exadata – V2 quarter rack with 11.2.0.2 BP20 and no compression.  Also, block size is 8K – half of what we had before.

Table         Column 1 Column 2 Column 3 Last Column-1 Last Column

TEST255C             0        0        0             0          14

TEST256C             0   166666   166666        166666      166666

TEST256CPLUS1        0   166666  1999999       1999999     1999999

So, without compression basically the same results.  I think the smaller block is probably why there is more chaining for the same amount of data.

But check out the dramatic reduction in chaining if you re-compress the table in each case with QUERY HIGH compression:

Table         Column 1 Column 2 Column 3 Last Column-1 Last Column

TEST255C            46       46       46           181         182

TEST256C            42       42       42           172         172

TEST256CPLUS1       41       41       41           172         172

Negligible chaining regardless and the elapsed times are all less than .2 seconds.

...>grep Elapsed *.log
nexttolastcolumns.log:Elapsed: 00:00:00.09
nexttolastcolumns.log:Elapsed: 00:00:00.08
nexttolastcolumns.log:Elapsed: 00:00:00.09
test255c.log:Elapsed: 00:00:00.09
test256c.log:Elapsed: 00:00:00.09
test256cplus1.log:Elapsed: 00:00:00.07
testfirstcolumns.log:Elapsed: 00:00:00.11
testfirstcolumns.log:Elapsed: 00:00:00.11
testfirstcolumns.log:Elapsed: 00:00:00.09
testsecondcolumns.log:Elapsed: 00:00:00.08
testsecondcolumns.log:Elapsed: 00:00:00.08
testsecondcolumns.log:Elapsed: 00:00:00.09
testthirdcolumns.log:Elapsed: 00:00:00.09
testthirdcolumns.log:Elapsed: 00:00:00.08
testthirdcolumns.log:Elapsed: 00:00:00.08

By comparison the uncompressed Exadata elapsed times were in the seconds with chaining:

...>grep Elapsed *.log
nexttolastcolumns.log:Elapsed: 00:00:00.99
nexttolastcolumns.log:Elapsed: 00:00:01.83
nexttolastcolumns.log:Elapsed: 00:00:02.81
test255c.log:Elapsed: 00:00:01.07
test256c.log:Elapsed: 00:00:01.83
test256cplus1.log:Elapsed: 00:00:03.19
testfirstcolumns.log:Elapsed: 00:00:00.72
testfirstcolumns.log:Elapsed: 00:00:10.97
testfirstcolumns.log:Elapsed: 00:00:00.51
testsecondcolumns.log:Elapsed: 00:00:04.80
testsecondcolumns.log:Elapsed: 00:00:04.63
testsecondcolumns.log:Elapsed: 00:00:01.18
testthirdcolumns.log:Elapsed: 00:00:04.36
testthirdcolumns.log:Elapsed: 00:00:04.56
testthirdcolumns.log:Elapsed: 00:00:02.21

On our non-Exadata system the chaining resulted in times in minutes with the most chaining:

...>grep Elapsed *.log
nexttolastcolumns.log:Elapsed: 00:00:22.89
nexttolastcolumns.log:Elapsed: 00:00:30.15
nexttolastcolumns.log:Elapsed: 00:04:19.77
test255c.log:Elapsed: 00:00:02.49
test256c.log:Elapsed: 00:00:06.79
test256cplus1.log:Elapsed: 00:00:09.41
testfirstcolumns.log:Elapsed: 00:00:20.91
testfirstcolumns.log:Elapsed: 00:00:23.24
testfirstcolumns.log:Elapsed: 00:00:28.78
testsecondcolumns.log:Elapsed: 00:00:11.34
testsecondcolumns.log:Elapsed: 00:00:15.91
testsecondcolumns.log:Elapsed: 00:00:18.46
testthirdcolumns.log:Elapsed: 00:00:13.29
testthirdcolumns.log:Elapsed: 00:00:17.95
testthirdcolumns.log:Elapsed: 00:04:12.92

So, if we can keep our tables with more than 255 columns compressed for query high row chaining doesn’t appear to be a factor, even if we add new columns and populate them.  I believe this is because QUERY HIGH compression spreads the data for the rows in several blocks across the blocks anyway so they really aren’t stored in the same way that uncompressed rows are stored with all the columns for a row kept together.

– Bobby

Posted in Uncategorized | 4 Comments

Exchange Partition Example

I put together an example of how to do an exchange partition to meet a specific need in our environment.  It might be helpful to others though your situation may vary.  It is a self-contained test script in a zip.

There is documentation in the script but here are the criteria:

-- example of partition exchange with following conditions:
-- 
-- Range partitioned table
-- Indexes PK, regular, bitmapped - all local
-- validated FK
-- trigger
-- parallel 8 nologging
--
-- tables:
-- ptab - partitioned table
-- ftab - table related by FK
-- etab - table to be exchanged in
--

Here is the actual exchange:

-- exchange etab with ptab partition p1:

alter table ptab 
exchange partition p1 
with table etab
including indexes 
without validation;

One side effect is that the constraints that were ‘VALIDATED’ before are now ‘NOT VALIDATED’:

SQL> -- show constraint status
SQL> 
SQL> select constraint_name,status,validated
  2  from user_constraints
  3  where table_name in ('PTAB','FTAB');

CONSTRAINT_NAME                STATUS   VALIDATED
------------------------------ -------- -------------
FTAB_PK                        ENABLED  VALIDATED
PTAB_PK                        ENABLED  NOT VALIDATED
FK_C3                          ENABLED  NOT VALIDATED

I built this script to reorganize a partitioned table so it assumes you just want to copy the rows reorganizing them more efficiently in the blocks.  This was tested on 11.2.0.3.

– Bobby

Posted in Uncategorized | Leave a comment

Full scan performance on tables with more than 255 columns

Production issue

We have some production tables with more than 255 columns and are experiencing performance degradation on queries that access certain columns.  I was able to show that a full table scan with a where clause condition including one column ran in 6 minutes but with another column ran over 20 minutes.  Also, I noticed that the slower full scans were seeing “db file sequential read” waits and that the statistic “table fetch continued rows” was steadily growing.

A coworker told me that they thought the difference in performance was caused by us having more than 255 columns in the table and asked if we should reorder the columns to improve performance since it seemed like the faster column was at the beginning of the table.

I picked a recent partition of the table and tested a full scan on each of the over 300 columns and found that the first 15 columns performed much better than all the rest.

Then I dumped out a block from a partition and found that the rows were split into 255 column chunks and smaller chunks.  But, the interesting thing is that the 255 column chunk had the last 255 columns and not the first 255.

All this led to the building of a test case to figure out how this works.  Here is the zip.

Take away

Before I get into a long description of all the steps I took in my test case let me first describe what I believe is the take away:

  1. My testing assumes your table has more than 255 columns and less than 2 * 255
  2. A full scan with a where clause condition on one of the last 255 columns will experience row chaining
  3. If you add new columns to the table and populate them with data the chaining experienced on the last 255 columns will be greatly increased
  4. If you reorganize the table by copying the data to a new table then the first N-255 columns will not experience row chaining.  (N=total number of columns).

So, if you have a table with N columns and N>255 and <2*255 then best performance would be found if you reorganize the rows in the table after you add columns and populate them.  Also, ideal performance will be achieved if the columns you use in your where clause are in the first N-255 columns.

Test Case

Tables

test255c – 255 columns
test256c – 256 columns
test256cplus1 – 256 columns initially, then one added and populated with data

Each of these tables are loaded with a million rows of data.

Test scripts – run in this order

test255c.sql – 255 columns, builds table, tests last column
test256c.sql – 256 columns, builds table, tests last column
test256cplus1.sql – 257 columns, builds table, adds column, tests last column
testfirstcolumns.sql – retest first column all three tables
testsecondcolumns.sql – retest second column all three tables
testthirdcolumns.sql – retest third column all three tables
nexttolastcolumns.sql – retest next to last column all three tables

blockdumps255c.trc – one data block from test255c
blockdumps256c.trc – one data block from test256c
blockdumps256cplus1.trc – one data block from test256cplus1

The key test results relate to the statistic “table fetch continued rows”.  I’ve summarized the results of sequential full table scans on each of these tables in this chart:

Table         Column 1 Column 2 Column 3 Last Column-1 Last Column

TEST255C             0        0        0             0           0

TEST256C             0    83333    83333         83333       83333

TEST256CPLUS1        0    83333  1999999       1999999     1999999

So, in the 255 column example there was no row chaining.  In the 256 column example there was some row chaining but small compared to the total number of rows.  In the example where a column had been added to make 257 columns the last 255 columns experienced a lot of row chaining compared to the number of rows.  Unfortunately for us our production situation is represented by the last example.  Queries against our first 15 columns don’t have chaining just like the first column in the example, but many other columns are chaining like crazy.

The block dumps weren’t really necessary to confirm this but they do.  What I see is that on table test255c the row pieces are consistently 255 columns.  On test256c they are either 1 column or 255 columns and the one column row contains the first column.  On test256cplus1 there are 1 column pieces and 255 column pieces and the one column pieces contain either the first or the second column.  So, adding a column seems to split off the first column from the 255 column piece and add the new column to the end making a new 255 column piece.  But, if you notice the minimal row chaining on column 2 of my test against test256cplus1 that confirms what the block dumps show.  Column two hasn’t really moved that much after the column add but the last 255 columns must have moved a lot.

So, my suggestion that we reorganize the rows of a table with more than 255 columns that has had columns added and loaded with data is based on the results for table test256c.  I believe these results represent what a reorganized table’s performance would be.  So, not terrible row chaining and first columns have none.  Of course, if you can keep your tables under 256 columns that would be even better!

– Bobby

P.S. This is on Oracle 11.2.0.3 on HP-UX 11.31, 64-bit Itanium.  16K block size.

Posted in Uncategorized | 1 Comment