SQL_TRACE, TKProf and Explain Plan

Oracle provides extensive tracing and interpretation tools to assist analysts diagnose and resolve sub-optimal performance issues. This article covers the most important of these and points to detailed reference articles.

Tracing Related Initialization Parameters

Prior to tracing, there are a number of parameters that need to be set so that the trace information is complete. These parameter should be set up in the "init.ora" file for the particular instance (SID) where you wish to use SQL Trace although they can also be set individually at the session level.

Parameter:TIMED_STATISTICS

Enable/Disable the collection of timed statistics, such as CPU and elapsed times.

• TRUE - Enable timing
• FALSE - Disable timing (Default value).

Note that much of the tracing information that can be gathered is rendered useless if TIMED_STATISTICS is set to False and timings are not collected.

Parameter:MAX_DUMP_FILE_SIZE

Specifies the maximum size of trace files in operating system blocks. The default value for this was 10000 OS blocks in 8i version, and limited only by the space available in 9i version. If your trace file is truncated then you will see a message similar to:

*** DUMP FILE SIZE IS LIMITED TO 12345 BYTES***

and the size of this parameter should be increased.

Parameter:USER_DUMP_DEST

Specifies the destination for the trace file. The default value for this parameter is the default destination for oracle dumps on your operating system.

These parameters can be dynamically altered using alter system/alter session commands (Note that USER_DUMP_DEST can only be modified at the system level).

For example TIMED_STATISTICS can be enabled/disabled dynamically by using the following SQL statement:

ALTER SYSTEM/SESSION SET TIMED_STATISTICS = TRUE/FALSE;

SQL_TRACE

SQL_TRACE is the main method for collecting SQL Execution information in Oracle collecting a wide range of information and statistics that can be used to tune SQL operations.

Enabling SQL_TRACE

The SQL Trace facility can be enabled/disabled for an individual session or at the instance level. If the initialisation Parameter SQL_TRACE is set to TRUE in the init.ora of an instance, then all sessions will be traced.

SQL_TRACE can be set at the instance level by using the initialisation parameter SQL_TRACE:

Parameter:SQL_TRACE

Enable/Disable SQL Trace instance wide.

• TRUE - Enable statistics to be collected for all sessions.
• FALSE - Disable statistics to be collected for all sessions.

SQL_TRACE can also be enabled/disabled at the system/session by issuing the following SQL statement:

ALTER SYSTEM/SESSION SET SQL_TRACE = TRUE/FALSE;

Trace can also be enabled/disabled on other sessions (as well as your own) using the DBMS_SUPPORT package.

See Note:62160.1 Tracing Sessions in Oracle7/8 for details.

For more information on how to setup tracing, refer to:

Note:15160.1 Setting SQL Trace in the Oracle Tools.

There is also extensive information in the

Oracle9i Database Performance Tuning Guide and Reference Chapter 10: Using SQL Trace and TKProf

Trace Files

Oracle will generate trace (.trc) files for every session where the value of SQL_TRACE=TRUE and write them to the USER_DUMP_DEST destination. If tracing has been enabled for the instance then individual trace files will be generated for each session, unless otherwise disabled. Note that the generated files may be owned by an operating system user other than your own so the necessary privileges will need to be put in place before they can be formatted with TKProf.

Explain Plan

The Explain Plan command generates information that details the execution plan that will be used on a particular query. It uses a precreated table (PLAN_TABLE) in the current shema to store information about the execution plan chosen by the optimizer.

The plan table is created using the script utlxplan.sql. This script is typically found under the Oracle Home in the rdbms/admin directory.

On Unix its location will be:

$ORACLE_HOME/rdbms/admin On WindowsNT/2000:

%ORACLE_HOME%\rdbms\admin

This script creates the output table, called PLAN_TABLE, for holding the output of the Explain plan Command. Note that the exact structure of the plan table can change with different release as new features are introduced.

Populating the Plan Table

The plan table is populated using the explain plan command:

SQL> EXPLAIN PLAN for select * from emp where empno=1000;

This command inserts the execution plan of the SQL statement into the plan table. It is also possible to adds the name tag to the explain information by using the set statement_id clause.

Displaying the Execution Plan

Once the table has been populated, the explain information needs to be retrieved and formatted. There are a large number of scripts available to format the plan table data. Some of the most popular are noted below:

Supplied Scripts:

$ORACLE_HOME/rdbms/admin/utlxpls.sql: script to format serial explain plans $ORACLE_HOME/rdbms/admin/utlxplp.sql: script to format parallel explain plans

Articles:

<Note:31101.1> Obtaining Formatted Explain Plan Output

<Note:39294.1> Formatted Select of PLAN_TABLE for EXPLAIN PLAN command

<Note:39341.1> Automatic Explain Plan

<Note:1019631.6> SCRIPT: SCRIPT TO SIMPLIFY THE USE OF EXPLAIN PLAN

Interpretation of Explain Plan

This is a complex topic and is covered in detail in the following article:

<Note:46234.1> Interpreting Explain Plan

There is also extensive information in the

Oracle9i Database Performance Tuning Guide and Reference Chapter 9: Using EXPLAIN PLAN

AUTOTRACE

The autotrace facility in SQL*Plus allows analysts to view the execution plan and some useful statistics for a SQL statement within a SQL*Plus session. This option was introduced with 7.3 version of Oracle.

Autotrace needs to be initiated in the SQL*Plus session prior to executing the statement. The Autotrace command is:

SET AUTOTRACE [OPTIONS] [EXPLAIN/STATISTICS]

For a detailed explanation of AUTOTRACE functions see:

<Note:43214.1> AUTOTRACE option in 7.3As

with the EXPLAIN PLAN command, to obtain an execution plan the PLAN_TABLE must be created in the user's schema prior to autotracing.

Example

SQL> set autotrace traceonly explain

SQL> select * from dual;Execution Plan
----------------------------------------------------------
0 SELECT STATEMENT Optimizer=CHOOSE
1 0 TABLE ACCESS (FULL) OF 'DUAL'

To enable viewing of STATISTICS data, the autotracing user must have access to dynamic performance tables. To achieve this, grant PLUSTRACE role to the user.

The PLUSTRACE role is created by the plustrce.sql script.

On Unix the location is:
$ORACLE_HOME/sqlplus/admin
On WindowsNT/2000:
%ORACLE_HOME%\sqlplus\admin

This script must be run by the SYS user. A DBA user can then grant the role to the users who wish to use the the AUTOTRACE option.

Refer to:

<Note:1055431.6> ORA-01919 usingAUTOTRACE in SQL*Plus

Extensive reference information regarding the autotrace facility can be found in the

SQL*Plus Users Guide and Reference Release 9.0.1

TKProf

The TKProf facility accepts as input a SQL trace file and produces a formatted output file. For the full syntax of TKProf see the Oracle Server Tuning Manual. If TKProf is invoked with no arguments, an online help is displayed.

Basic Syntax of TKProf

TKPROF filename_source filename_output EXPLAIN=[username/password] SYS=[YES/No] TABLE=[tablename]

For more information on using TKProf see:

<Note:32951.1> TKPROF Interpretation

<Note:29012.1> Quick Reference TKPROF usage

<Note:41634.1> TKPROF and Problem Solving

There is also extensive information in the

Oracle9i Database Performance Tuning Guide and Reference Chapter 10: Using SQL Trace and TKProf

Concurrent Managers

The concurrent managers in the Oracle e-Business suite serve several important administrative functions. Foremost, the concurrent managers ensure that the applications are not overwhelmed with requests, and the second areas of functions are the management of batch processing and report generation.

This article will explore tools that are used by experienced administrators to gain insight and improved control over the concurrent management functions. We will explore how the concurrent managers can be configured via the GUI, and also explore scripts and dictionary queries that are used to improve the functionality of concurrent management.

The Master Concurrent Managers

There is a lot of talk about "the" concurrent manager in Oracle Applications. Actually, there are many Concurrent Managers, each governing flow within each Oracle Apps areas. In addition there are "super" Concurrent Managers whose job is to govern the behavior of the slave Concurrent Managers. The Oracle e-Business suite has three important master Concurrent Managers:

• Internal Concurrent Manager — The master manager is called the Internal Concurrent Manager (ICM) because it controls the behavior of all of the other managers, and because the ICM is the boss, it must be running before any other managers can be activated. The main functions of the ICM are to start up and shutdown the individual concurrent managers, and reset the other managers after one them has a failure.
• Standard Manager — Another important master Concurrent Manager is called the Standard Manager (SM). The SM functions to run any reports and batch jobs that have not been defined to run in any specific product manager. Examples of specific concurrent managers include the Inventory Manager, CRP Inquiry Manager, and the Receivables Tax Manager.
• Conflict Resolution Manager — The Conflict Resolution Manager (CRM) functions to check concurrent program definitions for incompatibility rules. However, the ICM can be configured to take over the CRM's job to resolve incompatibilities.

Now that we understand the functions of the master Concurrent Managers, let's take a quick look at techniques that are used by Oracle Apps DBAs to monitor the tune the behavior of the Concurrent Managers.

Tuning the Concurrent Manager

All successful Oracle Apps DBAs must understand how to monitor and tune each of the Concurrent Managers. This article will explore some of the important techniques for monitoring and tuning the Oracle Apps Concurrent Manager processes. The topics will include:

• Tuning the Concurrent Manager
  - Tuning the Internal Concurrent Manager
  - Purging Concurrent Requests
  - Troubleshooting Oracle Apps performance problems
  - Adjusting the Concurrent Manager Cache Size
  - Analyzing the Oracle Apps Dictionary Tables
• Monitoring Pending Requests in the Concurrent Manager
• Changing the dispatching priority within the Concurrent Manager

Tuning the Internal Concurrent Manager (ICM)

The ICM performance is affected by the three important Oracle parameters PMON cycle, queue size, and sleep time.

• PMON cycle — This is the number of sleep cycles that the ICM waits between the time it checks for concurrent managers failures, which defaults to 20. You should change the PMON cycle to a number lower than 20 if your concurrent managers are having problems with abnormal terminations.
• Queue Size — The queue size is the number of PMON cycles that the ICM waits between checking for disabled or new concurrent managers. The default for queue size of 1 PMON cycle should be used.
• Sleep Time — The sleep time parameter indicates the seconds that the ICM should wait between checking for requests that are waiting to run. The default sleep time is 60, but you can lower this number if you see you have a lot of request waiting (Pending/Normal). However, reducing this number to a very low value many cause excessive cpu utilization.

All of the concurrent managers, with the exception of the ICM and CRM, can be configured to run as many processes as needed, as well as the time and days a manager can process requests. However, the number of processes needed is dependent on each organization's environment.

An Applications DBA must monitor the concurrent processing in order to decide how to configure each manager. For a fresh install of the applications, initially configure the standard manager to run with five processes, and all the other managers with two processes. After the applications have been in operation for a while, the concurrent managers should be monitored to determine is more operating system process should be allocated.

Purging Concurrent Requests

One important area of Concurrent Manager tuning is monitoring the space usage for the subsets within each concurrent manager. When the space in FND_CONCURRENT_PROCESSES and FND_CONCURRENT_REQUESTS exceed 50K, you can start to experience serious performance problems within your Oracle Applications.

When you experience these space problems, a specific request called "Purge Concurrent Requests And/Or Manager Data" should be scheduled to run on a regular basis. This request can be configured to purge the request data from the FND tables as well as the log files and output files on accumulate on disk.

Adjusting the Concurrent Manager Cache Size

Concurrent manager performance can also be enhanced by increasing the manager cache size to be at lease twice the number of target processes. The cache size specifies the number of requests that will be cached each time the concurrent manager reads from the FND_CONCURRENT_REQUESTS table. Increasing the cache size will boost the throughput of the managers by attempting to avoid sleep time.

Analyzing Oracle Apps Dictionary Tables for High Performance

It is also very important to run the request Gather Table Statistics on these tables:

• FND_CONCURRENT_PROCESSES
• FND_CONCURRENT_PROGRAMS
• FND_CONCURRENT_REQUESTS
• FND_CONCURRENT_QUEUES.

Run the request "Analyze All Index Column Statistics" on the indexes of these tables. Since the APPLSYS user is the owner of these tables, so you can also just run the request Analyze Schema Statistics for APPLSYS.

To troubleshoot performance, a DBA can use three types of trace.

A module trace, such as PO or AR, can be set by enabling the module's profile option Debug Trace from within the applications.

Second, most concurrent requests can be set to generate a trace file by changing the request parameters. To enable trace for a specific request, log in as a user with the System Administrator responsibility. Navigate to Concurrent -> Program -> Define. Query for the request that you want to enable trace.

Another popular way to troubleshoot the Concurrent Managers is to generate a trace file. This is done by setting the OS environment variable FNDSQLCHK to FULL, and running the request from the command line.

Monitoring Pending Requests in the Concurrent Managers

Occasionally, you may find that requests are stacking up in the concurrent managers with a status of "pending". This can be caused by any of these conditions:

1. The concurrent managers were brought down will a request was running.
2. The database was shutdown before shutting down the concurrent managers.
3. There is a shortage of RAM memory or CPU resources.

When you get a backlog of pending requests, you can first allocate more processes to the manager that is having the problem in order to allow most of the requests to process, and then make a list of the requests that will not complete so they can be resubmitted, and cancel them.

To allocate more processes to a manager, log in as a user with the System Administrator responsibility. Navigate to Concurrent -> Manager -> Define. Increase the number in the Processes column. Also, you may not need all the concurrent managers that Oracle supplies with an Oracle Applications install, so you can save resources by identifying the unneeded managers and disabling them.

However, you can still have problems. If the request remains in a phase of RUNNING and a status of TERMINATING after allocating more processes to the manager, then shutdown the concurrent managers, kill any processes from the operating system that won't terminate, and execute the following sqlplus statement as the APPLSYS user to reset the managers in the FND_CONCURRENT_REQUESTS table:

update fnd_concurrent_requestsset status_code='X', phase_code='C'where status_code='T';

Changing Dispatching Priority within the Concurrent Manager

If there are requests that have a higher priority to run over other requests, you can navigate to Concurrent --> Program --> Define to change the priority of a request. If a priority is not set for a request, it will have the same priority as all other requests, or it will be set to the value specified in the user's profile option Concurrent:Priority. Also, you can specify that a request run using an SQL optimizer mode of FIRST_ROWS, ALL_ROWS, RULE, or CHOOSE, and this can radically effect the performance of the SQL inside the Concurrent request.

If several long running requests are submitted together, they can cause fast running requests to have to wait unnecessarily. If this is occurring, try to schedule as many long running requests to run after peak business hours. Additionally, a concurrent manager can be created to run only fast running requests.

Oracle supplies several useful scripts, (located in $FND_TOP/sql directory), for monitoring the concurrent managers:

afcmstat.sql : Displays all the defined managers, their maximum capacity, pids, and their status.

afimchk.sql : Displays the status of ICM and PMON method in effect, the ICM's log file, and determines if the concurrent manger monitor is running.

afcmcreq.sql : Displays the concurrent manager and the name of its log file that processed a request.

afrqwait.sql : Displays the requests that are pending, held, and scheduled.

afrqstat.sql : Displays of summary of concurrent request execution time and status since a particular date.

afqpmrid.sql : Displays the operating system process id of the FNDLIBR process based on a concurrent request id. The process id can then be used with the ORADEBUG utility.

afimlock.sql : Displays the process id, terminal, and process id that may be causing locks that the ICM and CRM are waiting to get. You should run this script if there are long delays when submitting jobs, or if you suspect the ICM is in a gridlock with another oracle process.