Looking for something? Try here..

Thursday, May 26, 2022

Script to gather details while using Datapump for DB migration

 Here is a script that can be used to gather source database details when we are attempting to upgrade from one version of DB to another or migrate Oracle DB from one server to another. 

The purpose is to gather as much details so that the target DB can be prepared in such a way we don't have much work post import of the required schemas. 

/* -----------------------------------------------------------------------------------------
Script name 	: export_details.sql
Usage			: sqlplus / as sysdba @export_details.sql

Instructions	: 
Replace all SCHEMA1 and SCHEMA2 with the schemas you are intended to export 
and save the file as export_details.sql, use as many users needed

Important Note	: 
Once details gathered, export NLS_LANG as specified below to set env variable
and start export by using the options mentioned in the end of this file

export NLS_LANG=AMERICAN_AMERICA.AL32UTF8 - Check what is the NLS_LANG needed as per app

This script will create 3 o/p files, 1 html and 2 sql files.
------------------------------------------------------------------------------------------ */

set echo on pages 999 long 90000
col OWNER for a30
col USERNAME for a30
col PROFILE for a30
col GRANTED_ROLE for a30
col GRANTEE for a30
col PROPERTY_NAME for a40
col PROPERTY_VALUE for a40
col DISPLAY_VALUE for a50
col DEFAULT_VALUE for a50
set markup html on spool on pre off
spool source_details.html

Prompt Parameter details:
------ ------------------
select name, DISPLAY_VALUE from v$parameter where name in ('service_names', 'compatible','sga_max_size','sga_target','log_buffer','db_cache_size','pga_aggregate_target', 'pga_aggregate_limit','cpu_count','session_cached_cursors','open_cursors','processes') order by name;

Prompt User details:
------ -------------
select count(*) "Total Schemas" from dba_users;
select sum(bytes)/1024/1024/1024 Total_DB_used_Size_In_GB from dba_segments;
select sum(bytes)/1024/1024/1024 Total_DB_allocated_Size_In_GB from dba_data_files;
SELECT owner,sum(bytes)/1024/1024/1024 Size_In_GB from dba_segments WHERE owner IN ('SCHEMA1','SCHEMA2') group by owner order by owner;

Prompt Info about Schemas:
------ -------------------
select USERNAME,ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE,DEFAULT_TABLESPACE,TEMPORARY_TABLESPACE,PROFILE from dba_users where USERNAME in ('SCHEMA1','SCHEMA2') order by 1;

set markup html on spool on pre on
Prompt Profile details for user:
------ -------------------------
SELECT dbms_metadata.get_ddl('PROFILE', p.profile) || ';' from dba_profiles p where profile not in ('DEFAULT') and resource_name='COMPOSITE_LIMIT';

set markup html on spool on pre off
Prompt Tablespace Size for particular schema:
------ --------------------------------------
select owner, TABLESPACE_NAME, sum(BYTES)/1024/1024/1024 Size_In_GB from dba_segments where OWNER in ('SCHEMA1','SCHEMA2') group by owner,TABLESPACE_NAME order by owner,TABLESPACE_NAME;

set markup html on spool on pre on
Prompt DDL for Schema, Non default Roles and Tablespace:
------ -------------------------------------------------
SELECT dbms_metadata.get_ddl('USER',u.username) || ';' as USERS from dba_users u where username in ('SCHEMA1','SCHEMA2');
select dbms_metadata.get_ddl('TABLESPACE',tb.tablespace_name) || ';' as Tablespaces from dba_tablespaces tb where tablespace_name not in ('SYSTEM','SYSAUX','UNDOTBS','TEMP');
select dbms_metadata.get_ddl('ROLE', r.role) || ';' as Roles from dba_roles r where ORACLE_MAINTAINED<>'Y';

set markup html on spool on pre off
Prompt Temp tablespace size:
------ ---------------------
select tablespace_name, sum(BYTES)/1024/1024/1024 from dba_temp_files group by tablespace_name;

Prompt Objects counts:
------ ---------------
select owner, object_type, status, count(*) from dba_objects where owner in ('SCHEMA1','SCHEMA2') group by object_type,owner, status order by 1,2,3;

Prompt Roles:
------ ------
select GRANTEE, GRANTED_ROLE, ADMIN_OPTION from dba_role_privs where grantee in ('SCHEMA1','SCHEMA2') order by 1,2;
select GRANTEE, PRIVILEGE, ADMIN_OPTION from dba_sys_privs where grantee in ('SCHEMA1','SCHEMA2') order by 1,2;


Prompt System privileges having a property value = 1
Prompt Use specific DBMS packages such as DBMS_RESOURCE_MANAGER_PRIVS to provide grants to these objects. 
------ --------------------------------------------------------------------------------------------------
select p.grantee, m.name from system_privilege_map m, dba_sys_privs p where m.name=p.privilege and m.property=1 and p.grantee in ('SCHEMA1','SCHEMA2') order by p.grantee, m.name;

Prompt DB Time Zone Details:
------ ---------------------
select sysdate,systimestamp,current_timestamp,sessiontimezone,dbtimezone from dual;

Prompt DB Characterset:
------ ----------------
select PROPERTY_NAME,PROPERTY_VALUE from database_properties where PROPERTY_NAME in ('NLS_CHARACTERSET','NLS_NCHAR_CHARACTERSET');

spool off

Prompt Generating grants script:
------ -------------------------
set markup html off
set lines 200 pages 0 head off feed off
spool source_all_privs.sql

select 'grant ' || privilege ||' on '|| owner || '.' || table_name ||' to '||grantee||';' from dba_tab_privs where grantee in ('SCHEMA1','SCHEMA2') and table_name NOT like 'BIN%';
spool off

spool source_all_synonyms.sql

SELECT 'CREATE OR REPLACE PUBLIC SYNONYM '||SYNONYM_NAME||' FOR '||TABLE_OWNER||'.'||TABLE_NAME||';' FROM ALL_SYNONYMS WHERE OWNER='PUBLIC' AND TABLE_OWNER IN ('SCHEMA1','SCHEMA2');

spool off

exit
Once the details are gathered, we can use the generated html file to prepare the target DB like creating required tablespaces, roles, etc. 
Once the import is completed in the target DB, both the sql scripts can be run against target DB to make sure all the grants are provided and public synonyms are created. 

Let me know in comments on whether we can further add any details which might be needed for collecting details on source DB. 

Also check, Datapump & system privileges with property value = 1 for details on this part. 

Happy Datapumping...!!!

Saturday, May 7, 2022

Datapump & system privileges with property value = 1

Many automation tools like Autoupgrade (a sample demo is provided in this link), Zero Downtime Migration (ZDM), etc has been released by Oracle to make DBA life easier. Though we have these tools, we always run towards the most preferred and simple method of Datapump whether to jump the DBs from one server to another or one version to another if it can be done within the allowed downtime. It is such a powerful utility used by almost every DBA in their work life.

In one of my previous post, I have explained regarding few tips to make datapump job perform better. They still stand true for current version of Oracle as well with many advancements included.


Today we see regarding the privileges with property value =1 and its effect on datapump. 
In my recent migration of DB from version 11.2.0.4 to 19c, post migration using datapump, application complained stating few privileges are missing in the 19c DB. This can't be possible as I have taken a completed schema export and imported in the target database without any errors reported in the impdp logfile.
Upon investigating, ADMINISTER RESOURCE MANAGER privilege has not granted in the 19c DB though there are no errors in both expdp and impdp logfile. 
Taking a look at this Doc ID 1163383.1, a few privileges are not granted normally via grant statements but they have to be granted through specific plsql packages. 

How do we find those privileges? - The below query can provide and the result differs w.r.to DB version. Example from my source system 11g. 
SQL> select name from system_privilege_map where property=1 order by name;

NAME
----------------------------------------
ADMINISTER RESOURCE MANAGER
ALTER ANY EVALUATION CONTEXT
ALTER ANY RULE
ALTER ANY RULE SET
CREATE ANY EVALUATION CONTEXT
CREATE ANY RULE
CREATE ANY RULE SET
CREATE EVALUATION CONTEXT
CREATE RULE
CREATE RULE SET
DEQUEUE ANY QUEUE

NAME
----------------------------------------
DROP ANY EVALUATION CONTEXT
DROP ANY RULE
DROP ANY RULE SET
ENQUEUE ANY QUEUE
EXECUTE ANY EVALUATION CONTEXT
EXECUTE ANY RULE
EXECUTE ANY RULE SET
MANAGE ANY FILE GROUP
MANAGE ANY QUEUE
MANAGE FILE GROUP
READ ANY FILE GROUP

22 rows selected.

SQL>
All the above are granted via specific packages or via explicit grants on the system. 

How do we find which privileges does our source schema has from the above list which needs to be granted explicitly via packages? - The below query is an example
SQL> select p.grantee, m.name from system_privilege_map m, dba_sys_privs p
  2  where m.name=p.privilege and m.property=1 and p.grantee in ('DUMMY')
  3  order by p.grantee, m.name;

GRANTEE                        NAME
------------------------------ ----------------------------------------
DUMMY                          ADMINISTER RESOURCE MANAGER
DUMMY                          READ ANY FILE GROUP

SQL>
I have got 2 of such privileges that needs explicit grants on the target DB. 

Now, on the 19c DB we need to grant the privileges as below. A few would require explicit grant (like the READ ANY FILE GOUP) and  few would require grants via packages as below (like the ADMINISTER RESOURCE MANAGER). 
SQL> @check_user_privs.sql

GRANTEE                        TYP PRIVILEGE OR ROLE
------------------------------ --- ---------------------------------------------------------------------------
DUMMY                          PRV ALTER SESSION
DUMMY                              CREATE ANY DIRECTORY
DUMMY                              CREATE DATABASE LINK
DUMMY                              CREATE JOB
DUMMY                              CREATE PROCEDURE
DUMMY                              CREATE SEQUENCE
DUMMY                              CREATE SESSION
DUMMY                              CREATE SYNONYM
DUMMY                              CREATE TABLE
DUMMY                              CREATE TYPE
DUMMY                              CREATE VIEW
DUMMY                              SELECT ANY DICTIONARY
DUMMY                              UNLIMITED TABLESPACE


13 rows selected.

SQL> BEGIN
  DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SYSTEM_PRIVILEGE(
   GRANTEE_NAME   => 'DUMMY',
   PRIVILEGE_NAME => 'ADMINISTER_RESOURCE_MANAGER',
   ADMIN_OPTION   => FALSE);
END;
/  2    3    4    5    6    7

PL/SQL procedure successfully completed.

SQL> grant READ ANY FILE GROUP to DUMMY;

Grant succeeded.

SQL> @check_user_privs.sql

GRANTEE                        TYP PRIVILEGE OR ROLE
------------------------------ --- ---------------------------------------------------------------------------
DUMMY                          PRV ADMINISTER RESOURCE MANAGER
DUMMY                              ALTER SESSION
DUMMY                              CREATE ANY DIRECTORY
DUMMY                              CREATE DATABASE LINK
DUMMY                              CREATE JOB
DUMMY                              CREATE PROCEDURE
DUMMY                              CREATE SEQUENCE
DUMMY                              CREATE SESSION
DUMMY                              CREATE SYNONYM
DUMMY                              CREATE TABLE
DUMMY                              CREATE TYPE
DUMMY                              CREATE VIEW
DUMMY                              READ ANY FILE GROUP
DUMMY                              SELECT ANY DICTIONARY
DUMMY                              UNLIMITED TABLESPACE


15 rows selected.

SQL>
Now, we are all set. So while performing export and import we need to make sure to take care of these privilege grants without fail for the application to run without any issues. 

Note: Similarly, grants on sys owned objects will also be not transferred via datapump to target database. They have to be explicitly granted on the target database. 

Query used: 

check_user_privs.sql
select grantee, 'PRV' type, privilege pv 
from dba_sys_privs where grantee = 'DUMMY' union
select username grantee, '---' type, 'empty user ---' pv from dba_users 
where not username in (select distinct grantee from dba_role_privs) and
not username in (select distinct grantee from dba_sys_privs) and 
not username in (select distinct grantee from dba_tab_privs) and username like 'DUMMY'
group by username
order by grantee, type, pv;

References: 
Primary Note For Privileges And Roles (Doc ID 1347470.1)