Showing posts with label Datapump. Show all posts
Showing posts with label Datapump. Show all posts

Thursday, May 26, 2022

Script to gather details while using Datapump for DB migration

 Here is a script that can be used to gather source database details when we are attempting to upgrade from one version of DB to another or migrate Oracle DB from one server to another. 

The purpose is to gather as much details so that the target DB can be prepared in such a way we don't have much work post import of the required schemas. 

/* -----------------------------------------------------------------------------------------
Script name 	: export_details.sql
Usage			: sqlplus / as sysdba @export_details.sql

Instructions	: 
Replace all SCHEMA1 and SCHEMA2 with the schemas you are intended to export 
and save the file as export_details.sql, use as many users needed

Important Note	: 
Once details gathered, export NLS_LANG as specified below to set env variable
and start export by using the options mentioned in the end of this file

export NLS_LANG=AMERICAN_AMERICA.AL32UTF8 - Check what is the NLS_LANG needed as per app

This script will create 3 o/p files, 1 html and 2 sql files.
------------------------------------------------------------------------------------------ */

set echo on pages 999 long 90000
col OWNER for a30
col USERNAME for a30
col PROFILE for a30
col GRANTED_ROLE for a30
col GRANTEE for a30
col PROPERTY_NAME for a40
col PROPERTY_VALUE for a40
col DISPLAY_VALUE for a50
col DEFAULT_VALUE for a50
set markup html on spool on pre off
spool source_details.html

Prompt Parameter details:
------ ------------------
select name, DISPLAY_VALUE from v$parameter where name in ('service_names', 'compatible','sga_max_size','sga_target','log_buffer','db_cache_size','pga_aggregate_target', 'pga_aggregate_limit','cpu_count','session_cached_cursors','open_cursors','processes') order by name;

Prompt User details:
------ -------------
select count(*) "Total Schemas" from dba_users;
select sum(bytes)/1024/1024/1024 Total_DB_used_Size_In_GB from dba_segments;
select sum(bytes)/1024/1024/1024 Total_DB_allocated_Size_In_GB from dba_data_files;
SELECT owner,sum(bytes)/1024/1024/1024 Size_In_GB from dba_segments WHERE owner IN ('SCHEMA1','SCHEMA2') group by owner order by owner;

Prompt Info about Schemas:
------ -------------------
select USERNAME,ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE,DEFAULT_TABLESPACE,TEMPORARY_TABLESPACE,PROFILE from dba_users where USERNAME in ('SCHEMA1','SCHEMA2') order by 1;

set markup html on spool on pre on
Prompt Profile details for user:
------ -------------------------
SELECT dbms_metadata.get_ddl('PROFILE', p.profile) || ';' from dba_profiles p where profile not in ('DEFAULT') and resource_name='COMPOSITE_LIMIT';

set markup html on spool on pre off
Prompt Tablespace Size for particular schema:
------ --------------------------------------
select owner, TABLESPACE_NAME, sum(BYTES)/1024/1024/1024 Size_In_GB from dba_segments where OWNER in ('SCHEMA1','SCHEMA2') group by owner,TABLESPACE_NAME order by owner,TABLESPACE_NAME;

set markup html on spool on pre on
Prompt DDL for Schema, Non default Roles and Tablespace:
------ -------------------------------------------------
SELECT dbms_metadata.get_ddl('USER',u.username) || ';' as USERS from dba_users u where username in ('SCHEMA1','SCHEMA2');
select dbms_metadata.get_ddl('TABLESPACE',tb.tablespace_name) || ';' as Tablespaces from dba_tablespaces tb where tablespace_name not in ('SYSTEM','SYSAUX','UNDOTBS','TEMP');
select dbms_metadata.get_ddl('ROLE', r.role) || ';' as Roles from dba_roles r where ORACLE_MAINTAINED<>'Y';

set markup html on spool on pre off
Prompt Temp tablespace size:
------ ---------------------
select tablespace_name, sum(BYTES)/1024/1024/1024 from dba_temp_files group by tablespace_name;

Prompt Objects counts:
------ ---------------
select owner, object_type, status, count(*) from dba_objects where owner in ('SCHEMA1','SCHEMA2') group by object_type,owner, status order by 1,2,3;

Prompt Roles:
------ ------
select GRANTEE, GRANTED_ROLE, ADMIN_OPTION from dba_role_privs where grantee in ('SCHEMA1','SCHEMA2') order by 1,2;
select GRANTEE, PRIVILEGE, ADMIN_OPTION from dba_sys_privs where grantee in ('SCHEMA1','SCHEMA2') order by 1,2;


Prompt System privileges having a property value = 1
Prompt Use specific DBMS packages such as DBMS_RESOURCE_MANAGER_PRIVS to provide grants to these objects. 
------ --------------------------------------------------------------------------------------------------
select p.grantee, m.name from system_privilege_map m, dba_sys_privs p where m.name=p.privilege and m.property=1 and p.grantee in ('SCHEMA1','SCHEMA2') order by p.grantee, m.name;

Prompt DB Time Zone Details:
------ ---------------------
select sysdate,systimestamp,current_timestamp,sessiontimezone,dbtimezone from dual;

Prompt DB Characterset:
------ ----------------
select PROPERTY_NAME,PROPERTY_VALUE from database_properties where PROPERTY_NAME in ('NLS_CHARACTERSET','NLS_NCHAR_CHARACTERSET');

spool off

Prompt Generating grants script:
------ -------------------------
set markup html off
set lines 200 pages 0 head off feed off
spool source_all_privs.sql

select 'grant ' || privilege ||' on '|| owner || '.' || table_name ||' to '||grantee||';' from dba_tab_privs where grantee in ('SCHEMA1','SCHEMA2') and table_name NOT like 'BIN%';
spool off

spool source_all_synonyms.sql

SELECT 'CREATE OR REPLACE PUBLIC SYNONYM '||SYNONYM_NAME||' FOR '||TABLE_OWNER||'.'||TABLE_NAME||';' FROM ALL_SYNONYMS WHERE OWNER='PUBLIC' AND TABLE_OWNER IN ('SCHEMA1','SCHEMA2');

spool off

exit
Once the details are gathered, we can use the generated html file to prepare the target DB like creating required tablespaces, roles, etc. 
Once the import is completed in the target DB, both the sql scripts can be run against target DB to make sure all the grants are provided and public synonyms are created. 

Let me know in comments on whether we can further add any details which might be needed for collecting details on source DB. 

Also check, Datapump & system privileges with property value = 1 for details on this part. 

Happy Datapumping...!!!

Saturday, May 7, 2022

Datapump & system privileges with property value = 1

Many automation tools like Autoupgrade (a sample demo is provided in this link), Zero Downtime Migration (ZDM), etc has been released by Oracle to make DBA life easier. Though we have these tools, we always run towards the most preferred and simple method of Datapump whether to jump the DBs from one server to another or one version to another if it can be done within the allowed downtime. It is such a powerful utility used by almost every DBA in their work life.

In one of my previous post, I have explained regarding few tips to make datapump job perform better. They still stand true for current version of Oracle as well with many advancements included.


Today we see regarding the privileges with property value =1 and its effect on datapump. 
In my recent migration of DB from version 11.2.0.4 to 19c, post migration using datapump, application complained stating few privileges are missing in the 19c DB. This can't be possible as I have taken a completed schema export and imported in the target database without any errors reported in the impdp logfile.
Upon investigating, ADMINISTER RESOURCE MANAGER privilege has not granted in the 19c DB though there are no errors in both expdp and impdp logfile. 
Taking a look at this Doc ID 1163383.1, a few privileges are not granted normally via grant statements but they have to be granted through specific plsql packages. 

How do we find those privileges? - The below query can provide and the result differs w.r.to DB version. Example from my source system 11g. 
SQL> select name from system_privilege_map where property=1 order by name;

NAME
----------------------------------------
ADMINISTER RESOURCE MANAGER
ALTER ANY EVALUATION CONTEXT
ALTER ANY RULE
ALTER ANY RULE SET
CREATE ANY EVALUATION CONTEXT
CREATE ANY RULE
CREATE ANY RULE SET
CREATE EVALUATION CONTEXT
CREATE RULE
CREATE RULE SET
DEQUEUE ANY QUEUE

NAME
----------------------------------------
DROP ANY EVALUATION CONTEXT
DROP ANY RULE
DROP ANY RULE SET
ENQUEUE ANY QUEUE
EXECUTE ANY EVALUATION CONTEXT
EXECUTE ANY RULE
EXECUTE ANY RULE SET
MANAGE ANY FILE GROUP
MANAGE ANY QUEUE
MANAGE FILE GROUP
READ ANY FILE GROUP

22 rows selected.

SQL>
All the above are granted via specific packages or via explicit grants on the system. 

How do we find which privileges does our source schema has from the above list which needs to be granted explicitly via packages? - The below query is an example
SQL> select p.grantee, m.name from system_privilege_map m, dba_sys_privs p
  2  where m.name=p.privilege and m.property=1 and p.grantee in ('DUMMY')
  3  order by p.grantee, m.name;

GRANTEE                        NAME
------------------------------ ----------------------------------------
DUMMY                          ADMINISTER RESOURCE MANAGER
DUMMY                          READ ANY FILE GROUP

SQL>
I have got 2 of such privileges that needs explicit grants on the target DB. 

Now, on the 19c DB we need to grant the privileges as below. A few would require explicit grant (like the READ ANY FILE GOUP) and  few would require grants via packages as below (like the ADMINISTER RESOURCE MANAGER). 
SQL> @check_user_privs.sql

GRANTEE                        TYP PRIVILEGE OR ROLE
------------------------------ --- ---------------------------------------------------------------------------
DUMMY                          PRV ALTER SESSION
DUMMY                              CREATE ANY DIRECTORY
DUMMY                              CREATE DATABASE LINK
DUMMY                              CREATE JOB
DUMMY                              CREATE PROCEDURE
DUMMY                              CREATE SEQUENCE
DUMMY                              CREATE SESSION
DUMMY                              CREATE SYNONYM
DUMMY                              CREATE TABLE
DUMMY                              CREATE TYPE
DUMMY                              CREATE VIEW
DUMMY                              SELECT ANY DICTIONARY
DUMMY                              UNLIMITED TABLESPACE


13 rows selected.

SQL> BEGIN
  DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SYSTEM_PRIVILEGE(
   GRANTEE_NAME   => 'DUMMY',
   PRIVILEGE_NAME => 'ADMINISTER_RESOURCE_MANAGER',
   ADMIN_OPTION   => FALSE);
END;
/  2    3    4    5    6    7

PL/SQL procedure successfully completed.

SQL> grant READ ANY FILE GROUP to DUMMY;

Grant succeeded.

SQL> @check_user_privs.sql

GRANTEE                        TYP PRIVILEGE OR ROLE
------------------------------ --- ---------------------------------------------------------------------------
DUMMY                          PRV ADMINISTER RESOURCE MANAGER
DUMMY                              ALTER SESSION
DUMMY                              CREATE ANY DIRECTORY
DUMMY                              CREATE DATABASE LINK
DUMMY                              CREATE JOB
DUMMY                              CREATE PROCEDURE
DUMMY                              CREATE SEQUENCE
DUMMY                              CREATE SESSION
DUMMY                              CREATE SYNONYM
DUMMY                              CREATE TABLE
DUMMY                              CREATE TYPE
DUMMY                              CREATE VIEW
DUMMY                              READ ANY FILE GROUP
DUMMY                              SELECT ANY DICTIONARY
DUMMY                              UNLIMITED TABLESPACE


15 rows selected.

SQL>
Now, we are all set. So while performing export and import we need to make sure to take care of these privilege grants without fail for the application to run without any issues. 

Note: Similarly, grants on sys owned objects will also be not transferred via datapump to target database. They have to be explicitly granted on the target database. 

Query used: 

check_user_privs.sql
select grantee, 'PRV' type, privilege pv 
from dba_sys_privs where grantee = 'DUMMY' union
select username grantee, '---' type, 'empty user ---' pv from dba_users 
where not username in (select distinct grantee from dba_role_privs) and
not username in (select distinct grantee from dba_sys_privs) and 
not username in (select distinct grantee from dba_tab_privs) and username like 'DUMMY'
group by username
order by grantee, type, pv;

References: 
Primary Note For Privileges And Roles (Doc ID 1347470.1)

Monday, June 21, 2021

IMPDP, ORA-39083, ORA-02304 and the fix

 I was trying some PoC today with an impdp job involving sample schemas that I had in my lab database. The objective I was testing was to check on LONG column support for impdp via a network link (more of network link import in this link). 

I am aware LONG column is not supported in Oracle version 12.1 if we use network_link and is already documented in this official link but the same is mentioned as supported and as a new feature in the Oracle 12.2 official link here.

Well, testing makes us to be confident on what we are going to present to our clients, isn't it? :)

So in the due process of testing this, I just stumbled up on this error below. 

DB env: Oracle 12.2

Issue:

* Output truncated

[oracle@linux75-2 ~]$ impdp system REMAP_SCHEMA=pm:nwimp DIRECTORY=data_pump_dir NETWORK_LINK=nw_import_demo remap_tablespace=users:tt1 schemas=pm

Import: Release 12.2.0.1.0 - Production on Sun Jun 20 20:30:26 2021

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.
Password:

Connected to: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Starting "SYSTEM"."SYS_IMPORT_SCHEMA_01":  system/******** REMAP_SCHEMA=pm:nwimp DIRECTORY=data_pump_dir NETWORK_LINK=nw_import_demo remap_tablespace=users:tt1 schemas=pm
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 18.81 MB
Processing object type SCHEMA_EXPORT/USER
ORA-31684: Object type USER:"NWIMP" already exists

Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/TYPE/TYPE_SPEC
ORA-39083: Object type TYPE:"NWIMP"."ADHEADER_TYP" failed to create with error:
ORA-02304: invalid object identifier literal

Failing sql is:
CREATE EDITIONABLE TYPE "NWIMP"."ADHEADER_TYP"   OID '82A4AF6A4CCE656DE034080020E0EE3D'

  AS OBJECT
    ( header_name        VARCHAR2(256)
    , creation_date      DATE
    , header_text        VARCHAR2(1024)
    , logo               BLOB
    );

ORA-39083: Object type TYPE:"NWIMP"."TEXTDOC_TYP" failed to create with error:
ORA-02304: invalid object identifier literal
 ...
 ...
 ...

The IMPDP operation failed during the import of TYPE_SPEC and says invalid object identifier literal. 
This is a scenario which would occur when we try to duplicate a schema or duplicate objects and their dependent objects within the same database. What I'm trying here is importing a sample schema from source database via network_link which I already have in the target database into another schema and so in short, this also is sort of duplicating the schema.
When the types are exported, we also export the object_identifier (OID) of the types. Within the current architecture, the object-identifier needs to be unique in the database and the OID of the types already exist in target, the types can't be created and hence the error.

Solution
We can overcome this by precreating the type and tables in the target database under the schema nwimp and then import the schema with table_exists_action parameter set to append. 

or

Starting Oracle version 10.2, we have a parameter TRANSFORM which is used to alter object creation DDL for the objects being imported. So we will now use this parameter to tell Oracle not to perform OID checking when looking for an existing matching type on the target database and also to assign a new OID to the object. 

TRANSFORM=OID:N

* Output truncated
[oracle@linux75-2 ~]$ impdp system transform=oid:n REMAP_SCHEMA=pm:nwimp DIRECTORY=data_pump_dir NETWORK_LINK=nw_import_demo remap_tablespace=example:tt1 schemas=pm

Import: Release 12.2.0.1.0 - Production on Sun Jun 20 20:37:45 2021

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.
Password:

Connected to: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Starting "SYSTEM"."SYS_IMPORT_SCHEMA_01":  system/******** transform=oid:n REMAP_SCHEMA=pm:nwimp DIRECTORY=data_pump_dir NETWORK_LINK=nw_import_demo remap_tablespace=example:tt1 schemas=pm
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 18.81 MB
Processing object type SCHEMA_EXPORT/USER
ORA-31684: Object type USER:"NWIMP" already exists

Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/TYPE/TYPE_SPEC
ORA-31684: Object type TYPE:"NWIMP"."ADHEADER_TYP" already exists

ORA-31684: Object type TYPE:"NWIMP"."TEXTDOC_TYP" already exists

ORA-31684: Object type TYPE:"NWIMP"."TEXTDOC_TAB" already exists

Processing object type SCHEMA_EXPORT/TABLE/TABLE

. . imported "NWIMP"."PRINT_MEDIA"                            4 rows
. . imported "NWIMP"."TEXTDOCS_NESTEDTAB"                    12 rows
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX
...
...
...
 

Now, we can see the import says the objects are already existing and just continued with the import of table object and data without issues. 

By the way, I'm still pursuing my testing on long columns. That's all for today... :)

References

DataPump Import Of Object Types Fails With Errors ORA-39083/ORA-39082 ORA-2304 Or ORA-39117 ORA-39779 (Doc ID 351519.1)

Happy importing...!!!

Saturday, July 11, 2015

Datapump Export using Datapump API

The Data Pump API, DBMS_DATAPUMP, provides a high-speed mechanism to move all or part of the data and metadata for a site from one database to another. The Data Pump Export and Data Pump Import utilities are based on the Data Pump API.

In this post lets see how to perform a schema level export using the datapump API.
There are a few reasons (as I'm thinking as of now) to perform export/import through API rather then command mode interface is as follows.
1. When you don't have access to database server to perform an export but you only have access to the database and you know the pre created directory object.
2. When you perform a remote operation on a database from another server (say an application server) so you would like to perform export from the same connection
3. When you would like to schedule a database job from within the database itself without the necessity to create OS scripts to invoke export.

As a prerequisite, to perform export using Datapump API, exp_full_database role has to be granted to the user performing expdp directly rather then through a role. (citation needed here as I tried granted through a role to fail)
I've also modified the API code to accept schema name to be exported as input so that we can store it as a procedure and invoke as and when required. The code is as below.
CREATE OR REPLACE PROCEDURE Proc_bkp_schema 
-- orabliss.blogspot.com
(v_schema IN VARCHAR2)
IS
   dp_handle    NUMBER;
   job_status   VARCHAR2 (30);
   v_dt         NUMBER;
   v_sch_name   VARCHAR2(30);
   v_filename   VARCHAR2(30);
   v_logname    VARCHAR2(30);
BEGIN
   SELECT TO_NUMBER (TO_CHAR (SYSDATE, 'yyyymmdd')) INTO v_dt FROM DUAL;
   v_sch_name := 'IN ('''||v_schema||''')';
   v_filename := ''||v_schema||'_'||v_dt||'.dmp'; -- dumpfile name
   v_logname  := ''||v_schema||'_'||v_dt||'.log'; -- logfile name

   -- schema export mode
   dp_handle := DBMS_DATAPUMP.open (operation => 'EXPORT', job_mode => 'SCHEMA');

   -- dump file
   DBMS_DATAPUMP.add_file (
      handle      => dp_handle,
      filename    => v_filename,
      directory   => 'DATAPUMP',
      filetype    => SYS.DBMS_DATAPUMP.KU$_FILE_TYPE_DUMP_FILE);

   -- log file
   DBMS_DATAPUMP.add_file (
      handle      => dp_handle,
      filename    => v_logname,
      directory   => 'DATAPUMP',
      filetype    => SYS.DBMS_DATAPUMP.KU$_FILE_TYPE_LOG_FILE);

   -- specify schema name
   DBMS_DATAPUMP.metadata_filter (handle   => dp_handle,
                                  name     => 'SCHEMA_EXPR',
                                  VALUE    => v_sch_name);

   DBMS_DATAPUMP.start_job (dp_handle);

   DBMS_DATAPUMP.wait_for_job (handle => dp_handle, job_state => job_status);

   DBMS_OUTPUT.put_line (
         'DataPump Export - '
      || TO_CHAR (SYSDATE, 'DD/MM/YYYY HH24:MI:SS')
      || ' Status '
      || job_status);

   DBMS_DATAPUMP.detach (handle => dp_handle);
END;
/

The procedure can be called as below.
begin PROC_BKP_SCHEMA ('ORACLE'); end;
/

This procedure above accepts a single schema as input. The same procedure can be altered to accept many users through a 'for loop' and export several schemas.
You might also make use of metadata_filter with 'NAME_EXPR' to filter out only required tables and data_filter to filter out partitions of table using the datapump API.

Happy working!

Monday, May 21, 2012

Datapump - some tips


Data Pump is a utility for unloading/loading data and metadata into a set of operating system files called a dump file set. The dump file set can be imported only by the Data Pump Import utility. The dump file set can be imported on the same system or it can be moved to another system and loaded there.
In this post, let us see some tips and tricks that can done with Datapump. 

Tip #1 : Using PARALLEL parameter
PARALLEL parameter is used to improve the speed of the export. But this will be more effective when you split the dumpfiles with DUMPFILE parameter across the filesystem.
Create 2 or 3 directories in different filesystems and use the commands effectively.

expdp / dumpfile=dir1:test_1.dmp, dir1:test_2.dmp, dir2:test_3.dmp, dir3:test_4.dmp logfile=dir1:test.log full=y parallel=4
where dir1, dir2 and dir3 are directory names created in the database.

Tip #2 : Using FILESIZE parameter
FILESIZE parameter is used to limit the dumpfile size. For eg., if you want to limit your dumpfiles to 5gb, you can issue command as below

expdp / directory=dir1 dumpfile=test1.dmp,test2.dmp,test3.dmp logfile=test.log filesize=5120m
or 
expdp / directory=dir1 dumpfile=test_%U.dmp logfile=test.log filesize=5120m full=y
where %U will assign numbers automatically from 1 to 99. 

Note: If you use %U, dumpfile number 100 can't be created and export fails with "dumpfile exhausted" error.

Update (23 July 2013): If you want to create more than 99 files, you can use this work around. 
expdp / directory=dir1 dumpfile=test_%U.dmp dumpfile=test_1%U.dmp logfile=test.log filesize=5120m full=y
This will create files in a round robin method like below.
test_01.dmp
test_101.dmp
test_02.dmp
test_102.dmp

Tip #3 : Usage of VERSION parameter
VERSION parameter is used while taking export if you want to create a dumpfile which should be imported into a DB which is lower than the source DB. 
For eg., if your source DB is 11g and target DB is 10g, you can't use the dumpfile taken from 11g expdp utility to import into 10g DB. 
This throws the below error.
ORA-39142: incompatible version number 3.1 in dump file "/u02/dpump/test.dmp"
To overcome this we can use the VERSION parameter.
VERSION={COMPATIBLE | LATEST | version_string}
For example
expdp / directory=dir1 dumpfile=test_1.dmp logfile=test.log VERSION=10.2.0

Tip #4 : PARALLEL with single DUMPFILE
When you use PARALLEL parameter and use only one dumpfile to unload datas from the DB, you may get the below error.
expdp / directory=dir1 dumpfile=test_1.dmp logfile=test.log parallel=4

ORA-39095: Dump file space has been exhausted: Unable to allocate 8192 bytes 
Job "USER"."TABLE_UNLOAD" stopped due to fatal error at 00:37:29
Now a simple work around is to remove the PARALLEL parameter or add dumpfiles. This will over come the error.

expdp / directory=dir1 dumpfile=test_1.dmp logfile=test.log 
or
expdp / directory=dir1 dumpfile=test_1.dmp,test_2.dmp,test_3.dmp, test_4.dmp logfile=test.log parallel=4
or
expdp / directory=dir1 dumpfile=test_%U.dmp logfile=test.log parallel=4

Tip #5 : Drop dba_datapump_job rows
Sometimes before the export completes or when the export encounters a resumable wait or you would have stopped the export job in between. Now you start the DataPump job that stopped. Then the dump file has been removed from the directory location. You are not able to attach to the job. 
You will get an error like this.

ORA-39000: bad dump file specification
ORA-31640: unable to open dump file "/oracle/product/10.2.0/db_2/rdbms/log/test.dmp" for read
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
But you will see the row updated in view dba_datapump_jobs
SQL> select * from dba_datapump_jobs;
OWNER JOB_NAME                       OPERATI JOB_M STATE                    DEGREE ATTACHED_SESSIONS DATAPUMP_SESSIONS
----- ------------------------------ ------- ----- -------------------- ---------- ----------------- -----------------
SYS   SYS_EXPORT_FULL_01             EXPORT  FULL  NOT RUNNING                   0        ##########                 0
You are not able to remove the row from dba_datapump_jobs as you are not able to attach to the export job with expdp client to kill the job.
In this case you can remove the row by dropping the master table created by the datapump export.
SQL> drop table SYS_EXPORT_FULL_01 purge;
Table dropped.
SQL> select * from dba_datapump_jobs;
no rows selected
Now you can see the row is deleted from the dba_datapump_jobs view.

Tip #6 : FLASHBACK_SCN and FLASHBACK_TIME 
Do not use FLASHBACK_SCN and FLASHBACK_TIME as these parameters slow down the performace of export.

Tip #7 : Effective EXCLUDE
Import of full database should be split as tables first and indexes next. Use the parameter exclude effectively to improve the speed of import.
EXCLUDE = INDEX,STATISTICS 
This will not import the indexes and statistics which in turn only import the tables, hence improving the performance.

Tip #8 : INDEXFILE=<filename> usage
After the import of tables has been completed, you can create the indexes and collect statistics of the tables. To get the indexes creation ddl, you can use the INDEXFILE = <filename> parameter to get all the indexes creation statements which were involved in the import operation.

Example of effective import 
impdp / directory=dir1,dir2,dir3 dumpfile=test_%U.dmp logfile=test.log EXCLUDE=STATISTICS Full=Y INDEXFILE=index_ddl.sql
The above will turn on the legacy mode import of datapump as the  parameter indexfile is present instead of SQLFILE parameter.
Indexfile parameter is available in imp and sqlfile parameter with impdp. However you can use indexfile parameter in impdp which will turn on legacy mode import which is as below. 
;;; Legacy Mode Active due to the following parameters:
;;; Legacy Mode Parameter: "indexfile=testindex.sql" Location: Command Line, Replaced with: "sqlfile=index_ddl.sql include=index"
Hence to extract only the indexes the statement should be as below.
impdp / directory=dir1,dir2,dir3 dumpfile=test_%U.dmp logfile=test.log EXCLUDE=STATISTICS Full=Y SQLFILE=index_ddl.sql INCLUDE=INDEX
Note: Tip #8 edited as per comment from Eric below.

Tip #9 : Contents of Dump file
If you are not sure about the schemas that were present in the dumpfile or tablespaces present inside the dumpfile, etc., you can easily check the dumpfile for those information using the below command

grep -a "CREATE USER" test_1.dmp
grep -a "CREATE TABLESPACE" test_1.dmp
-a is not a recognised flag in some OS and hence command works without the flag. Mind, the dumpfile created is a binary file.

The above command gives all the CREATE USER statements and CREATE TABLESPACE statements which will be useful in many cases. You can also get the INDEXES and TABLES creation ddl from the dumpfile as well.

Tip #10 : init.ora parameter cursor_sharing
Always set init.ora parameter cursor_sharing to exact which has a good effect on import's performance.

Tip #11 : STATUS parameter usage
You can check the on going datapump export/import operation with the use of STATUS parameter and track the progress by yourself. You can attach to a export/import session and check the status. 

For example:
[oracle@ini8115l3aa2ba-136018207027 ~]$ expdp attach=SYS_EXPORT_FULL_01
Export: Release 11.2.0.1.0 - Production on Mon May 21 10:56:28 2012
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
Username: sys as sysdba
Password:
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Job: SYS_EXPORT_FULL_01
  Owner: SYS
  Operation: EXPORT
  Creator Privs: TRUE
  GUID: C08622D4FB5571E4E04012881BCF4C92
  Start Time: Monday, 21 May, 2012 10:55:55
  Mode: FULL
  Instance: newdb
  Max Parallelism: 1
  EXPORT Job Parameters:
  Parameter Name      Parameter Value:
     CLIENT_COMMAND        sys/******** AS SYSDBA directory=dmpdir full=y
  State: EXECUTING
  Bytes Processed: 0
  Current Parallelism: 1
  Job Error Count: 0
  Dump File: /u02/dpump/expdat.dmp
    bytes written: 4,096
Worker 1 Status:
  Process Name: DW00
  State: EXECUTING
  Object Type: DATABASE_EXPORT/SCHEMA/TABLE/COMMENT
  Completed Objects: 400
  Worker Parallelism: 1
Export> status
Job: SYS_EXPORT_FULL_01
  Operation: EXPORT
  Mode: FULL
  State: COMPLETING
  Bytes Processed: 37,121
  Percent Done: 100
  Current Parallelism: 1
  Job Error Count: 0
  Dump File: /u02/dpump/expdat.dmp
    bytes written: 561,152
Worker 1 Status:
  Process Name: DW00
  State: WORK WAITING
Here you can see the bytes written which will be progressing and you can track the export/import job easily.
Note: The parameter ATTACH when used, it cannot be combined with any other parameter other than the USERID parameter.

$ expdp ATTACH= JOB_NAME
I’ll be updating the post whenever I come across things that can help improving the performance of datapump. J

Tuesday, March 27, 2012

Datapump export with error ORA-28112


Today I was implementing datapump export to one of my database where I came across this error below.
. . exported "SYSMAN"."MGMT_TARGET_TYPE_VERSIONS"        54.52 KB     395 rows
. . exported "SYSMAN"."OCS_TARGET_ASSOC_DEFS"            70.78 KB     537 rows
. . exported "SYSMAN"."OCS_TEMPLATES_DEFS"               59.57 KB     316 rows
. . exported "SYSMAN"."AQ$_MGMT_ADMINMSG_BUS_S"          7.820 KB       3 rows
ORA-31693: Table data object "SYSMAN"."MGMT_IP_REPORT_DEF" failed to load/unload and is being skipped due to error:
ORA-28112: failed to execute policy function
. . exported "SYSMAN"."MGMT_TASK_QTABLE"                 19.21 KB      32 rows
. . exported "SYSMAN"."AQ$_MGMT_HOST_PING_QTABLE_S"      7.789 KB       2 rows
ORA-31693: Table data object "SYSMAN"."MGMT_JOB" failed to load/unload and is being skipped due to error:
ORA-28112: failed to execute policy function
. . exported "SYSMAN"."AQ$_MGMT_LOADER_QTABLE_S"         7.781 KB       2 rows
. . exported "SYSMAN"."AQ$_MGMT_NOTIFY_QTABLE_S"         7.781 KB       2 rows
. . exported "SYSMAN"."BHV_TARGET_ASSOC_DEFS"            11.68 KB      34 rows
. . exported "SYSMAN"."DB_USER_PREFERENCES"              7.671 KB      14 rows

This error happens when we do an export of grid control OMS database.
Running export as SYS or SYSTEM  may not be a problem and other exports may run without error.

User running the export might have the required privileges to run the export such as EXP_FULL_DATABASE, CONNECT, DBA, etc., but still we face the above error.

Users Running Export should have EXEMPT ACCESS POLICY privilege to export all rows as that user is then exempt from VPD policy enforcement.  SYS is always exempted from VPD or Oracle Label Security policy enforcement, regardless of the export mode, application, or utility that is used to extract data from the database.

So the workaround would be to grant the exempt access policy to the user running the export.
SQL> grant exempt access policy to USERNAME; -- replace with desired username

Grant succeeded.

Now the export comletes without any error. :-)