Duplicate blog to laptop VM

I started to write this blog post and then realized that in the process of making a copy of my blog on my laptop I whacked my connection to WordPress. But, it was easy to resolve it by disconnecting Jetpack from WordPress, clearing my browser cache, and then reconnecting. Whew! I worried that messing with this stuff would cause some problem but the blog still seems to work.

My idea was to bring up a copy of this blog site on a VirtualBox VM on my laptop. I have thought about doing this in the past but did not have any good reason to do so. I have mainly acted like an end user for this blog software and to iPage, the company that hosts it. I have not tried to delve into the underlying PHP code or mess with the MySQL database. But, my database team has recently started supporting MySQL and I am trying to train myself up a bit so it made sense to play with duplicating this small MySQL app.

I duplicated the blog in three main steps:

  1. Setup a VirtualBox VM
  2. Setup a MySQL database
  3. Setup the web site

STEP 1 – SETUP A VIRTUALBOX VM

I am still using the VirtualBox NAT networking that I described in an earlier post. I created a VM called Blog with this ip and port forwarding details:

Blog ip

10.0.2.17

Blog port forwarding

61014 to 22
61015 to 80

I duplicated an existing Oracle Enterprise Linux 7 VM and modified the ip and setup the port forwarding. Then I did a yum update to catch up all the existing packages. I also disabled the firewall so that it would not get in the way.

yum -y update
systemctl disable firewalld
systemctl stop firewalld

STEP 2 – SETUP MYSQL DATABASE

I downloaded mysql57-community-release-el7-11.noarch.rpm from http://dev.mysql.com/downloads/repo/yum/ and ran the following commands to install MySQL:

yum localinstall mysql57-community-release-el7-11.noarch.rpm
yum install mysql-community-server
service mysqld start
service mysqld status

Then I ran a series of SQL commands logged into MySQL to setup the user and database for the blog:

grep 'temporary password' /var/log/mysqld.log

mysql -uroot -p
use temporary root password

ALTER USER 'root'@'localhost' IDENTIFIED BY 'Bl0gC0py!';

CREATE DATABASE blogdb;

GRANT ALL PRIVILEGES ON blogdb.* TO "blogdbuser"@"localhost"
IDENTIFIED BY "Bl0gC0py!";

FLUSH PRIVILEGES;

EXIT

Next I took a backup of my blog database that I got from iPage and made the following string replacements in the SQL commands:

Replace all instances of

http://www.bobbydurrettdba.com

with 

http://localhost:61015

Also

bobby@bobbydurrettdba.com

replace with

bobby@blogvm.com

and

@bobbydurrettdba

with

@blogvm

Finally

bobbydurrettdba

with 

blogvm

I was trying to make sure that I didn’t whack anything in my real blog and that it worked with my localhost:61015 web site host name and port. I had to add two lines to the beginning of the sql script to make it work:

use blogdb
SET sql_mode = '';

I ran the script like this:

mysql -v --force -ublogdbuser -pBl0gC0py! < database.sql > database.out 2> database.err

I checked database.err and it only had a warning about using the password on the command line.

STEP3 – SETUP WEB SITE

Now that the database was setup and loaded with data I worked on the web site.

First, I installed the Linux packages for php which pulled in the web server as a dependency:

yum install php php-common php-mysql php-gd php-xml php-mbstring php-mcrypt 

Edited httpd.conf to setup web server:

vi /etc/httpd/conf/httpd.conf

Replace all instances of AllowOverride None or none
with AllowOverride all

Added VirtualHost lines at the end:

Added host to /etc/hosts

vi /etc/hosts

add this line 

127.0.0.1  blog

I forgot to mention in step 1 that when I created the VM I renamed it to blog using hostnamectl.

Next I created the web site directory and populated it with files from my ftp backup of my website.

mkdir /var/www/html/wordpress

copy my blog files to /var/www/html/wordpress

Next I edited the WordPress configuration file to work with the database:

cd /var/www/html/wordpress

vi wp-config.php

/** The name of the database for WordPress */
define('DB_NAME', 'blogdb');

/** MySQL database username */
define('DB_USER', 'blogdbuser');

/** MySQL database password */
define('DB_PASSWORD', 'Bl0gC0py!');

/** MySQL hostname */
define('DB_HOST', 'localhost');

Finally I restarted the web server and set the web server to automatically start on reboot:

systemctl restart httpd.service 

systemctl enable httpd.service

Finally, I tested the web site at http://localhost:61015 and it looked a lot like this blog site.

Bobby

Posted in Uncategorized | Leave a comment

Quick Python script to backup remote directory using ftp

I looked around for some other ways to do this but decided to just code this up in Python. It connects to a remote Linux server using ftp and recursively copies all the files and directories back to a Windows machine.

Here is the source:

This is an example of the ease of use of Python for quick scripting. It uses a low-level ftp library called ftplib.

Bobby

Posted in Uncategorized | 2 Comments

Added save and restore data function to PythonDBAGraphs

I pushed out a quick change to PythonDBAGraphs to automatically save the data for any graph that you make so that you can redraw the graph later. This is better than saving an image file because the redrawn graph lets you see details about the points on the graph when you hover the mouse over the points.

Now when you generate a graph you get a line like this:

Saving data in C:\temp\ASH active session count for MYDB database.txt

When you want to see the graph again you run show_saved.py like this:

python show_saved.py
Enter name of data file to be restored: C:\temp\ASH active session count for MYDB database.txt

Bobby

Posted in Uncategorized | Leave a comment

Python for the Oracle DBA – Connects to everything

In my introduction to my Python for the Oracle DBA topic I said that Python can connect to everything that an Oracle DBA needs. I want to use this post to expand on that and talk about the Python modules and packages that I have used.

As you might expect, Python includes built-in modules and ones that you can add. Python comes with a large built-in library called the Python Standard Library. To use these modules you just import them into your scripts. There is nothing to install. You can also install new modules from PyPI, the Python Package Index. You install a PyPI package using a tool called pip that acts a lot like Red Hat Linux’s yum utility. Here is the pip command line help:

D:\>pip

Usage:
  pip  [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
 ... etc. ...

The plotting module that I use, Matplotlib,  is part of a larger group of modules called SciPi which I assume stands for Scientific Python. I have only used Matplotlib and Numpy but SciPi includes a number of other modules. You can get the various SciPy modules from PyPI using pip as described above.

Obviously an Oracle DBA wants to connect to Oracle databases. The cx_Oracle package lets you connect to an Oracle database. I am still using version 5.3 but a newer version is out and supports a variety of features. cx_Oracle requires an Oracle client. I have been mainly using a 32 bit 11.2 Oracle Windows client with cx_Oracle 5.3 on my Windows 7 corporate laptop. There is a Python standard for database APIs so if you install modules to connect to various types of databases they will hopefully follow this standard and have similar interfaces. I have connected to Snowflake, MySQL, and Big Query databases using Python as well but have not done anything significant with them.

I have written a couple of scripts on Linux that I would have written using bash shell scripting in the past. I describe in an earlier post how you can use the built-in subprocess module to run SQL*Plus from Python. More recently I wrote a script to test all of my Unix passwords with the Paramiko package which does ssh. Here is the script:

It just tries to log into a host using ssh and prints out a useful message if the login fails. It runs the hostname command after it logs in. Note the import statements at the top of the program. I installed the paramiko module from PyPI but sys and socket are part of the standard library.

The last thing I want to mention is how you can use Python with Amazon Web Services or AWS. I have started to learn how to work with AWS for my job and have done some very basic Python scripts to just connect to AWS and run simple commands. Amazon calls their main Python module Boto 3. We have production Python scripts that other people in my company have written so Python is alive and well in Amazon’s cloud. It probably would not hurt for an Oracle DBA to learn some Python just because we might need it when working with cloud services. I have already mentioned the cloud based databases Snowflake and Big Query so those are also examples of Python working with the cloud.

So, I have mainly given an overview of the Python packages and modules that I have used. I have only played with some of them. I have used others such as cx_Oracle, Matplotlib, Paramiko, and subprocess for real work. Based on my experience so far I think Python can connect to just about anything I would need as an Oracle DBA.

Bobby

Posted in Uncategorized | 2 Comments

Python for the Oracle DBA – Easy to use

In my earlier post I said that Python is easy to use. I want to fill in some details in this post. I have two main points:

  1. Python is not easy to learn
  2. Python is easy to use after you learn it

My experience supports these points. I don’t really have time to research what other people say and somehow prove these points with scientific evidence. But, I do have my own recent experience with the language and I have thought about what I wanted to say about it to other people.

I had to work pretty hard to learn Python. I’m a reasonably technical person with experience programming in a number of languages. I mean, I have a CS degree from Harvard and I’ve been working with databases for over 20 years. But, when I started working through the Python Tutorial it did not come very easily. Python’s documentation on docs.python.org is very good but the tutorial didn’t work that well for me. Maybe it was too fast for someone who was new to the language. I thought that their explanation of list slices was hard to follow. Also, the range function in for loops seemed weird compared to other languages. When they started talking about list comprehensions and lambda expressions it was too much. I think the tutorial just covers too much ground too quickly. I have not seen a need for a lot of the features that are in the tutorial anyway. It probably makes sense to learn the language some other way. I really learned Python by taking two edX classes. The edX classes included programming assignments and a slower pace than the tutorial. There are even easier classes than the ones I took so probably someone who is new to Python should find a class that is at their pace and not bang their head against the wall trying to work through the tutorial.

When I say that Python is easy for an Oracle DBA to use I think I mean that once you learn the language it is easy to keep using it. I think that the built-in list and dictionary data structures are the strongest features of Python. It takes some effort to learn the syntax and what it means but once you know it they are great building blocks for your Python scripts. If you forget some detail the documentation is always there. Just Google Python list or Python dictionary. You will find most of the answers on Python.org’s documentation site and Stack Overflow. But, Google brings up all kinds of helpful information. I think the other thing that makes Python easy to use is its dynamic typing. You don’t have to declare a variable’s type. Just use the variable. It is nice for quick and dirty scripts. Just put a number or a string or a list in a variable and then use it. The other nice feature is that Python requires indentation. If you have an if statement or loop you have to indent the statements inside the if or loop. The style guide recommends 4 spaces. This is great for hacking together simple scripts. Python tells you right away if your spaces are off so you can make sure that you really have the things where you want them. Also, you do not have to use the more complicated features. I’ve used a bit of the object-oriented features but not much. I’ve never used a list comprehension or lambda expression. I have just used plan old ifs, loops, and functions and done everything I needed.

There is a lot more I can say about this and a lot more examples that I could give but the key point is that in my opinion and my experience Python is an easy language to come back to after you have been away from it for a while. It took work for me to learn the language at first, but once I had done so it was pretty easy to keep coming back to it. I think this is true because you can do so much with the main easy to use features of the language and because of the high quality documentation and other resources available.

Bobby

P.S. I took these two MIT edX classes:

MITx – 6.00.1x Introduction to Computer Science and Programming Using Python

MITx – 6.00.2x Introduction to Computational Thinking and Data Science

They included some computer science and a little statistics but you also learned Python and used it to write programs. These classes were too hard for people who have never written a program before but someone with programming experience would find them easier. They were easy for me and I was a CTA or course teaching assistant for each class and helped answer questions. But they were challenging enough that I learned Python.

P.P.S. A few weeks back we had a corporate event sort of like a hack-a-thon. I got to play with a Google database product that I had never seen before. A coworker uploaded some data that he had collected on his Raspberry Pi into the Google database and I wrote a quick and dirty Python script to pull in the data and graph it. I hacked the database code together from Google’s example code and wrote the plotting code based on other scripts that I have written.

Here is the output:

This is an example of being able to quickly build a useful script using Python.

Posted in Uncategorized | 9 Comments

Python for the Oracle DBA – Outline/Intro

I want to put together a talk about how useful the Python programming language is for an Oracle database administrator or DBA. I thought that I would start by putting my thoughts down in a blog post as a possible starting point for a speech. I think the best way to organize this post is around a series of questions.

First off, why does an Oracle DBA need a programming language? We are not developers. DBAs do backup and recovery, performance tuning, installations, upgrades and patching, etc. We may write the occasional small script or program to help with our work but we are not full-time heads down 40 hours a week plus programmers. A lot of what I do uses simple SQL scripts that I run through Oracle’s SQL*Plus command line utility. A handful of these scripts include Oracle’s PL/SQL programming language so I have done some programming but it is not my full-time job. Our database servers have Unix and Linux shell scripts that do things like exports and trace file cleanups. In addition I have graphical tools like Toad or Oracle Enterprise Manager that I use in my job. I can do most of my job with SQL, PL/SQL, shell scripts, and GUI database tools. Why do a need a general purpose programming language like Python in my role as an Oracle DBA?

A language like Python helps me in my DBA job because it is easy to use and connects to a lot of things.

Why would a smart Oracle database administrator need a programming language that is easy to use? We are very technical people, are we not? I think DBAs need an easy to use programming language because they are not full-time developers. I don’t know about you, but I can only hold so many details in the front of my mind. I have a lot of Oracle database syntax and details rolling around in my head. I know how to join V$SESSION and V$PROCESS without looking it up. That’s not surprising after using Oracle for 20 plus years. But, I can only remember so much. Since programming is not my primary focus I do not think I can remember a language’s details very well. So, I think it makes sense for a DBA to have an easy to use programming language like Python. I can quickly look up details that I forget if it has been a while since I wrote a Python program, so I don’t need all the details in the front of my brain.

What do I mean when I say that Python connects to a lot of things? There are all kinds of libraries or modules that you can use with Python. One of Python’s claims to fame is that you can use it as the glue to tie varies components together. I use a graphics module that helps me make plots of Oracle performance metrics. I get the data using an Oracle database module. My SQL*Plus and Unix shell scripts did not let me connect to a client side graphics library. Plus, I use Python to connect to SQL*Plus and to ssh into Unix systems to run shell scripts. So, Python can connect to pretty much any type of system or resource that I could need in my job as a database administrator. On top of all that, cloud providers such as Amazon Web Services use Python. I have used Python to connect to AWS. Also, I have tested Python with the cloud based Snowflake database. I have also connected Python to a MySQL database. It connects to a lot of stuff! Contrast Python to PL/SQL. PL/SQL is great for Oracle database programming. But it doesn’t connect to other stuff very easily. You aren’t going to connect to Amazon Web Services or to a client side graphics library through PL/SQL. It is easy to connect to these things with Python.

So, to summarize my two points for why Python is a good programming language for Oracle DBAs :

  1. Python is easy to use and DBAs are not full-time programmers.
  2. Python connects to everything a DBA uses.

In my talk I want to go into more depth on each point. What makes Python easy to use? What are some things it connects to? It would be helpful for an audience to see evidence to support each point. I could include code examples or quotes from web sites.

Maybe to keep this post from being excessively long I can make this one an intro or outline and delve into the supporting material in follow-up posts. If anyone reading this has questions or criticisms of this material I would be happy to hear it. I’m presenting my own thoughts about Python’s usefulness in my job based on my experience. If other people have good reasons why Python is not so useful to an Oracle DBA or see problems with my reasoning I would be happy to hear your opinion.

Bobby

 

Posted in Uncategorized | 7 Comments

Fix index corruption found using analyze table validate structure

We have struggled with a corrupted database and have gone through too many issues and challenges to document in a blog post. But, I thought I would document the fix to some index corruption that I found during the process. This is all on Red Hat 64 bit Linux Oracle 12.1.0.2 database.

A couple of coworkers of mine built a clone of the corrupt production database and my job was to see if there was any corruption left after they applied fixes to the known issues. I decided to work through Oracle Support’s documentation about fixing corruption. I started with this high level document:

Master Note for Handling Oracle Database Corruption Issues (Doc ID 1088018.1)

This document led me to first try using RMAN to find any remaining corruption. I followed this document:

How to identify all the Corrupted Objects in the Database with RMAN (Doc ID 472231.1)

I just ran the recommended simple commands:

rman target /

backup validate check logical database;

There was no corruption.

Next I started working through this Oracle Support document:

Identify the Corruption Extension for Block Corruption, Table/Index Inconsistency, Data Dictionary and Lost Writes (Doc ID 836658.1)

I liked the idea of looking for corruption for lost writes because we saw a number of internal errors that seemed to point to lost writes. These are some of the errors that we saw in production:

ORA-00600: internal error code, arguments: [kdifind:kcbz_objdchk]
ORA-00600: internal error code, arguments: [4137], [47.32.257993]
ORA-00600: internal error code, arguments: [kdsgrp1]
ORA-00600: internal error code, arguments: [ktprPURT_badundo]
ORA-00600: internal error code, arguments: [kturbleurec1]

So, the next check I did was with the dbv command based on the Oracle support document. I wrote this query to build all the dbv commands:

select 
'dbv file='||FILE_NAME||' blocksize='||
(select value from v$parameter where name='db_block_size')
from dba_data_files
order by FILE_NAME;

This produced commands like this:

dbv file=/u01/app/oracle/oradata/orcl/example01.dbf blocksize=8192

None of the dbv commands showed any corruption. So, I was beginning to think we had a clean system but then I tried the analyze table validate structure command from the same Oracle Support document and found corruption. I ran the command against every table. I had to run utlvalid.sql out of the $ORACLE_HOME/rdbms/admin directory to create the invalid_rows table. Then I ran these queries to build all the analyze commands:

select 
'analyze table '||owner||'."'||table_name||
'" validate structure cascade;'
from dba_tables
where PARTITIONED='NO'
order by owner,table_name;


select 
'analyze table '||owner||'."'||table_name||
'" validate structure cascade into invalid_rows;'
from dba_tables
where PARTITIONED='YES'
order by owner,table_name;

I ran the script that these queries built and got these errors:

ORA-01499: table/index cross reference failure - see trace file
ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

I ran the analyze commands again on the ones with resource busy and they ran without error. But I had three that consistently failed with ORA-01499. They were these three system tables:

SYS.WRH$_SEG_STAT_OBJ
SYS.WRH$_SQLTEXT
SYS.WRH$_SQLSTAT

This led me to yet another Oracle Support document to help diagnose the ORA-01499 errors:

ORA-1499. Table/Index row count mismatch (Doc ID 563070.1)

I needed to find the trace files that the analyze command created for each table. So, I ran the analyzes like this:

alter session set max_dump_file_size = unlimited;
ALTER SESSION SET tracefile_identifier = 'bobbydurrett';
analyze table SYS."WRH$_SQLSTAT" 
validate structure cascade into invalid_rows;

This put my name in the trace file name so I could find it easily. I found a line like this in the trace file for each command:

row not found in index tsn: 1 rdba: 0x00817bfa

I ran the script from the Oracle Support document like this:

SELECT owner, segment_name, segment_type, partition_name
FROM DBA_SEGMENTS
WHERE header_file = (SELECT file# 
                     FROM   v$datafile 
                     WHERE  rfile# =
 dbms_utility.data_block_address_file(
to_number('00817bfa','XXXXXXXX'))
                       AND  ts#= 1)
AND header_block = dbms_utility.data_block_address_block(
to_number('00817bfa','XXXXXXXX'));

This led me to the corrupt indexes:

SYS
WRH$_SQLSTAT_INDEX
INDEX PARTITION
WRH$_SQLSTA_2469445177_11544

SYS
WRH$_SEG_STAT_OBJ_INDEX
INDEX

SYS
WRH$_SQLTEXT_PK
INDEX

I ran these commands to fix the first two:

alter index SYS."WRH$_SQLSTAT_INDEX" 
modify partition WRH$_SQLSTA_2469445177_11544 unusable;

alter index SYS."WRH$_SQLSTAT_INDEX" 
rebuild partition WRH$_SQLSTA_2469445177_11544;

alter index SYS."WRH$_SEG_STAT_OBJ_INDEX" unusable;

alter index SYS."WRH$_SEG_STAT_OBJ_INDEX" rebuild;

But then I found that SYS.”WRH$_SEG_STAT_OBJ_PK was also corrupt but the rebuild failed:

SQL> alter index SYS."WRH$_SEG_STAT_OBJ_PK" rebuild;
alter index SYS."WRH$_SEG_STAT_OBJ_PK" rebuild
*
ERROR at line 1:
ORA-01452: cannot CREATE UNIQUE INDEX; duplicate keys found

This led me to yet another Oracle Support document:

ORA-01452: Cannot Create Unique Index; Duplicate Keys Found (Doc ID 332494.1)

I had to use these steps on both WRH$_SEG_STAT_OBJ_PK and WRH$_SQLTEXT_PK. I’m not sure why they had duplicate rows but I assume it was due to the index corruption.

SQL> select rowid,DBID, TS#, OBJ#, DATAOBJ#, CON_DBID 
from SYS.WRH$_SEG_STAT_OBJ
where  rowid not in (select min(rowid) from SYS.WRH$_SEG_STAT_OBJ 
group by DBID, TS#, OBJ#, DATAOBJ#, CON_DBID);  2

ROWID                    DBID        TS#       OBJ#   DATAOBJ#   CON_DBID
------------------ ---------- ---------- ---------- ---------- ----------
AAACEhAACAAA5nMAAi 2469445177         13     373044     373044 2469445177

SQL> alter session set skip_unusable_indexes=true;

Session altered.

SQL> alter table SYS.WRH$_SEG_STAT_OBJ 
disable constraint WRH$_SEG_STAT_OBJ_PK;

Table altered.

SQL> delete from SYS.WRH$_SEG_STAT_OBJ 
where rowid='AAACEhAACAAA5nMAAi';

1 row deleted.

SQL> commit;

SQL> CREATE UNIQUE INDEX SYS.WRH$_SEG_STAT_OBJ_PK 
ON SYS.WRH$_SEG_STAT_OBJ
  2  (DBID, TS#, OBJ#, DATAOBJ#, CON_DBID)
  3  LOGGING
  4  TABLESPACE SYSAUX
  5  PCTFREE    10
  6  INITRANS   2
  7  MAXTRANS   255
  8  STORAGE    (
  9              INITIAL          64K
 10              NEXT             1M
 11              MINEXTENTS       1
 12              MAXEXTENTS       UNLIMITED
 13              PCTINCREASE      0
 14              BUFFER_POOL      DEFAULT
 15             );

Index created.

SQL> alter table SYS.WRH$_SEG_STAT_OBJ 
enable  constraint WRH$_SEG_STAT_OBJ_PK;

Table altered.

I didn’t need the skip_unusable_indexes alter command so I left it off for the second PK index:

alter table SYS.WRH$_SQLTEXT disable constraint WRH$_SQLTEXT_PK;

select rowid,DBID, SQL_ID, CON_DBID from SYS.WRH$_SQLTEXT
where  rowid not in (select min(rowid) from SYS.WRH$_SQLTEXT 
group by DBID, SQL_ID, CON_DBID); 

delete from SYS.WRH$_SQLTEXT where rowid='AAACBvAACAABB6UAAE';

commit;

CREATE UNIQUE INDEX SYS.WRH$_SQLTEXT_PK ON SYS.WRH$_SQLTEXT
(DBID, SQL_ID, CON_DBID)
LOGGING
TABLESPACE SYSAUX
PCTFREE    10
INITRANS   2
MAXTRANS   255
STORAGE    (
            INITIAL          64K
            NEXT             1M
            MINEXTENTS       1
            MAXEXTENTS       UNLIMITED
            PCTINCREASE      0
            BUFFER_POOL      DEFAULT
           );
           
alter table SYS.WRH$_SQLTEXT enable constraint WRH$_SQLTEXT_PK;

Sorry that this post is so long but I thought it would be fun to document my index corruption journey. It was mostly a matter of navigating Oracle Support’s web site and following their recommendations. But, I thought it might help to document this particular situation and some of my queries.

Bobby

Posted in Uncategorized | 1 Comment

Result cache latch contention

I recently saw a dramatic example of result cache latch contention. I had just upgraded a database to 11.2.0.4 and several hours later processing pretty much came to a halt.

Of course I am telling you the end before I tell you the beginning. It all started the morning of July 22nd, 2017, a few weeks back. We had worked for a couple of months on an 11.2.0.4 upgrade. I mentioned some issues with the same upgrade in my earlier post. I spent several hours Saturday morning upgrading the database and it all went very smoothly. Then we kicked off the normal batch processing and things seemed fine. Around 1 am the next morning I get paged about an issue. All of the batch processing had ground to a halt on the upgraded database. Needless to say, I was freaking out and afraid that we would have to back out the upgrade. This would have been ugly and time-consuming.

At first I  focused on the top SQL statements in the AWR report. Here are the top few from that morning:

Elapsed Time (s) Executions Elapsed Time per Exec (s) %Total %CPU %IO SQL Id SQL Module SQL Text
17,983.20 452,555 0.04 43.98 10.64 0.00 4yw0zv7ty2t47 SQL*Plus SELECT SLS_EXCL_RSN FROM ( SEL…
3,643.96 0 8.91 13.68 0.19 2fxnrcamtbcc2 SQL*Plus DECLARE return_code number := …
3,637.60 0 8.90 3.67 0.06 18pdd22fh15dc SQL*Plus INSERT /*+ APPEND PARALLEL(TGT…

The top query dominates the others because it takes 43.98% of the total run time so it made sense that this query was the problem. I expected that some queries would change plan with the upgrade. I had used SQL Profiles to lock in other plans that I knew were problematic but I would not be surprised to see new ones. But, looking at the top SQL, sql id 4yw0zv7ty2t47, the plan was the same as before the upgrade. Here is some execution history of the query before and after the July 22 upgrade:

SQL_ID        PLAN_HASH_VALUE END_INTERVAL_TIME         EXECUTIONS_DELTA Elapsed Average ms
------------- --------------- ------------------------- ---------------- ------------------
4yw0zv7ty2t47       848116227 15-JUL-17 11.00.28.230 PM           192430         .822087045
4yw0zv7ty2t47       848116227 16-JUL-17 02.00.24.144 AM          2522505         .691503149
4yw0zv7ty2t47       848116227 16-JUL-17 03.00.42.580 AM          2787002         .793723611
4yw0zv7ty2t47       848116227 22-JUL-17 10.00.12.787 PM           220581         23.4686899
4yw0zv7ty2t47       848116227 22-JUL-17 11.00.30.251 PM           455887         20.1393904
4yw0zv7ty2t47       848116227 23-JUL-17 12.00.47.394 AM           445173         20.5407455
4yw0zv7ty2t47       848116227 23-JUL-17 01.00.05.711 AM           369333         28.3195315
4yw0zv7ty2t47       848116227 23-JUL-17 02.00.22.811 AM           465591         39.0232079
4yw0zv7ty2t47       848116227 23-JUL-17 03.00.40.758 AM           452555         39.7370594
4yw0zv7ty2t47       848116227 23-JUL-17 04.00.57.968 AM           458328         39.3421407
4yw0zv7ty2t47       848116227 23-JUL-17 04.09.32.144 AM            10055         39.1518787
4yw0zv7ty2t47       848116227 23-JUL-17 04.11.58.484 AM            18507         39.6002968
4yw0zv7ty2t47       848116227 23-JUL-17 04.15.24.661 AM             5215         39.4672715
4yw0zv7ty2t47       848116227 23-JUL-17 04.16.30.441 AM             8542         39.1123689
4yw0zv7ty2t47       848116227 23-JUL-17 05.00.15.147 AM           321635         39.9827259

So, I stared at this for a couple of hours, getting more and more stressed because my cure-all SQL Profile was not going to help in this situation. The plan had not changed. I could not think, in my sleep deprived state, of a way to resolve this issue that morning. Then, after a quick prayer for help, I noticed the %CPU and %IO columns in the query’s line in the AWR report. 10% CPU, 0% I/O. The query was spending 90% of its time waiting and doing nothing. I had noticed some latch waits on the AWR report but I thought that the CPU on the system was just busy so we had some latch waits.

These were the top three events:

Event Waits Total Wait Time (sec) Wait Avg(ms) % DB time Wait Class
latch free 3,658,537 26.9K 7 65.8 Other
DB CPU 4749 11.6
db file scattered read 71,549 142.1 2 .3 User I/O

But, now finally I looked back at the latch waits. I went down to the latch section of the AWR report and the Result Cache: RC Latch latch showed up big:

Latch Sleep Breakdown

Latch Name Get Requests Misses Sleeps Spin Gets
Result Cache: RC Latch 6,742,176 4,142,777 4,143,709 669,430
row cache objects 5,804,568 169,324 6,087 163,272
cache buffers chains 107,393,594 36,532 5,859 30,976

I had noticed the RESULT_CACHE hint in some queries on our test database but never saw the latch contention. Here is the first part of the problem query with the RESULT_CACHE hint.

SQL_ID 4yw0zv7ty2t47
--------------------
SELECT SLS_EXCL_RSN FROM ( SELECT /*+ PARALLEL RESULT_CACHE */ DISTINCT

Here is part of the plan with the result cache step:

Plan hash value: 848116227
------------------------------------------
| Id  | Operation                        |
------------------------------------------
|  11 |  COUNT STOPKEY                   |
|  12 |   VIEW                           |
|  13 |    RESULT CACHE                  |
------------------------------------------

Early in the morning I got the idea of disabling the result cache. I ran this command:

alter system set result_cache_max_size=0 scope=both;

All of the running queries immediately died with this error:

ORA-00600: internal error code, arguments: [qesrcDO_AddRO],…

But, when the jobs were rerun they quickly went to completion. I checked the query performance before and after disabling the result cache and before the upgrade as well and found that post upgrade without the result cache the query ran ten times faster than before.

SQL_ID        PLAN_HASH_VALUE END_INTERVAL_TIME         EXECUTIONS_DELTA Elapsed Average ms
------------- --------------- ------------------------- ---------------- ------------------
4yw0zv7ty2t47       848116227 15-JUL-17 11.00.28.230 PM           192430         .822087045
4yw0zv7ty2t47       848116227 16-JUL-17 02.00.24.144 AM          2522505         .691503149
4yw0zv7ty2t47       848116227 16-JUL-17 03.00.42.580 AM          2787002         .793723611
4yw0zv7ty2t47       848116227 22-JUL-17 10.00.12.787 PM           220581         23.4686899
4yw0zv7ty2t47       848116227 22-JUL-17 11.00.30.251 PM           455887         20.1393904
4yw0zv7ty2t47       848116227 23-JUL-17 12.00.47.394 AM           445173         20.5407455
4yw0zv7ty2t47       848116227 23-JUL-17 01.00.05.711 AM           369333         28.3195315
4yw0zv7ty2t47       848116227 23-JUL-17 02.00.22.811 AM           465591         39.0232079
4yw0zv7ty2t47       848116227 23-JUL-17 03.00.40.758 AM           452555         39.7370594
4yw0zv7ty2t47       848116227 23-JUL-17 04.00.57.968 AM           458328         39.3421407
4yw0zv7ty2t47       848116227 23-JUL-17 04.09.32.144 AM            10055         39.1518787
4yw0zv7ty2t47       848116227 23-JUL-17 04.11.58.484 AM            18507         39.6002968
4yw0zv7ty2t47       848116227 23-JUL-17 04.15.24.661 AM             5215         39.4672715
4yw0zv7ty2t47       848116227 23-JUL-17 04.16.30.441 AM             8542         39.1123689
4yw0zv7ty2t47       848116227 23-JUL-17 05.00.15.147 AM           321635         39.9827259
4yw0zv7ty2t47       848116227 23-JUL-17 05.20.37.524 AM           329457          17.895581
4yw0zv7ty2t47       848116227 23-JUL-17 05.21.15.363 AM           205154         .050141323
4yw0zv7ty2t47       848116227 23-JUL-17 05.25.07.159 AM          1023657         .049949389

If you look through the output you will see that pre-upgrade on July 16th the query averaged about .69 to .79 milliseconds. During the latch contention on July 23rd it averaged about 39 milliseconds, a lot worse. But, after disabling the result cache it averaged .05 milliseconds which is at least 10 times faster than with the result cache before the upgrade.

So, it seems that the result cache hint on this query has always slowed it down. But, the upgrade and the load afterward caused some intense latch contention that we had not seen before. But, it is very cool that disabling the result cache actually made the query faster than it has been in the past.

I don’t think that it makes sense to put a RESULT_CACHE hint in a query that will run in .05 milliseconds without it. The overhead of the result cache made the query run 10 times slower at least. Something about the upgrade resulted in latch contention that caused a system wide problem, but disabling the result cache made the query run faster than it ever had. We could have avoided this problem by leaving off the RESULT_CACHE hint, but it was nice that I could resolve the problem quickly by disabling the result cache using a parameter change.

Bobby

Posted in Uncategorized | Leave a comment

Two configuration changes with 11.2.0.4 upgrade

A couple of weeks ago I upgraded a major production database from 11.1.0.7 to 11.2.0.4. Our developers kept hitting one ugly bug after another on the minimally patched 11.1.0.7 so I wanted to get them on the most patched up version of Oracle 11. This is on HP-UX Itanium 11.31. I made two key configuration changes that I want to discuss here. I changed the database to use direct I/O and I set a parameter so that the database would not use direct path reads for most table and partition scans. The main point of this blog post is that one change required the other. I moved to direct I/O to improve checkpoint performance but that slowed down queries that repeatedly scanned segments using direct path reads. I set an underscore parameter that prevented direct path reads except on very large table scans and that sped the problem queries back up to normal.

In our testing a coworker ran a massive number of updating jobs at the same time. A truncate took about an hour during this excessive load. I ran a simple test script to truncate a small table during this load and it took over a minute. The truncate spent almost all of its time on these two waits:

  1. local write wait – 61%
  2. enq: RO – fast object reuse – 38%

This was on a Delphix clone of production so the NFS file systems were already forcing the database to use direct I/O. But, I had to max out the database writer processes to get the needed checkpoint performance. I based this decision on my experience with checkpoint performance on another database which I documented in this post: url. I set these parameters:

  1. filesystemio_options=directIO
  2. db_writer_processes=36
  3. dbwr_io_slaves=0

We use these settings on a large data warehouse staging database on 11.2.0.4 and HP-UX 11.31 to tune checkpoint performance there. So, we have tested this configuration in production for several years.

After changing these parameters the truncates ran fast under the same heavy updating load.

We remounted the database file systems with mincache=direct, convosync=direct options on our non-Delphix physical databases when we moved them to direct I/O.

For a long time I have seen issues with slowness of full partition scans on our Delphix clones of the production database that this post is about. Here is a post about this issue: url. I remember hearing about an undocumented underscore parameter that you could use to prevent direct path reads and always meant to look into it but kept holding back. I didn’t want to set the underscore parameter on my Delphix copies of production and not set it on my physical production database. That would make our test database different from production and that could lead to invalid testing. But, an upgrade was a great time to put the parameter in both on Delphix and in production. This was the parameter that I put in:

_small_table_threshold=1000000

I read several good blog posts about this parameter and other ways to deal with direct path reads in Oracle 11. Evidently some behavior changed in Oracle 11 that caused full scans to bypass the buffer cache in more situations, using direct path reads. Some post talked about DBA’s just bumping up the value for _small_table_threshold when they upgraded to Oracle 11 so I tried it and testing proved it out. Here is an earlier post about my work with this parameter: url.

So, the upgrade has given me a chance to not only move our database to a more patched up, and hopefully stable, release but it also was a chance to make two key configuration changes. Without direct I/O our production database was using the Unix file system cache to cache the blocks that were being repeatedly scanned using direct path reads. But, to switch to direct I/O I had to end the direct path reads so that the database buffer cache would be used to cache the scanned blocks. Direct I/O and the 36 database writer processes gave us great checkpoint performance. Disabling direct path read kept the queries that had depended on the Unix filesystem cache running just as fast by allowing them to use the database buffer cache.

Bobby

Posted in Uncategorized | Leave a comment

September AZORA meeting

The September AZORA meeting has a great lineup of speakers. AZORA is the Arizona Oracle User Group. Republic Services is providing us with an excellent place to meet off of the 101. I’m looking forward to it. Check it out if you are in the Phoenix area.

Bobby

Starting the fall with a bang! Rich Niemiec, Jerry Ward, Benoit Chaffanjon

Thursday, Sep 21, 2017, 10:00 AM

Republic Services – 2nd Floor Conference Rooms
14400 N 87th St (AZ101 & Raintree) Scottsdale, AZ

6 AZORAS Attending

Wow! We’re coming off of the summer break with a bang! We’ll have three speakers: Rich Niemiec and Jerry Ward from Viscosity, followed by Benoit Chaffanjon, Oracle VP of Enterprise Solutions and Chief Architect.We’ll also be starting earlier in the day at 10:00 am to change things up!Save the date! Thursday, September 21st, 10am-2:30pmFor the la…

Check out this Meetup →

 

Posted in Uncategorized | Leave a comment