An interesting option introduced with this release is the possibility to restore even a single table, without impacting other objects, tablespace or database when that table is no more in the recycle bin.
The current release I'm using is Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit based on a Linux platform.
Let's start with an example.
I'm going to create into a pluggable database a new user (MARCOV) and two tables with few rows.
Before dropping your tables be sure to have a backup containing the missing tables.
I was not able to find any recommendation on Oracle documentation ("Backup and Recovery User's Guide" and "Backup and Recovery Reference") about having a backup of the container database if you want to restore a table previously created into a pluggable database !
In the following scenario I had indeed a full backup of the container database and I was able to recover the "lost" table;
then I tried to perform the same steps having only the backup of the pluggable database and it was not possible to complete the recovery task.
Here is my test case. I'm currently connected to a CDB named CDB001.
In this container I've a pluggable database named PDB0101. The new user MARCOV on PDB0101 is created with few grants and the possibility to run the dbms_flashback package to obtain the current System Change Number (SCN).
[oracle@vsi08devpom ~]$ sqlplus / as sysdba SQL*Plus: Release 12.1.0.1.0 Production on Wed Feb 12 10:15:04 2014 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options SQL> select name, open_mode from V$PDBS; NAME OPEN_MODE ------------------------------ ---------- PDB$SEED READ ONLY PDB0101 READ WRITE SQL> show con_name CON_NAME ------------------------------ CDB$ROOT SQL> alter session set container=PDB0101; Session altered. SQL> show con_name CON_NAME ------------------------------ PDB0101 SQL> create user marcov 2 identified by "marcov" 3 default tablespace users 4 temporary tablespace temp 5 profile default 6 account unlock; User created. SQL> grant create session, create table to marcov; Grant succeeded. SQL> alter user marcov quota 50M on users; User altered. SQL> grant execute on dbms_flashback to marcov; Grant succeeded.Once the new user is created, I can connect to the pluggable database with that user, creating one table with one row.
The current SCN after the table has been created is 8009717.
SQL> conn marcov/marcov@PDB0101_CDB001 Connected. SQL> show con_name CON_NAME ------------------------------ PDB0101 SQL> create table t1 (a number); Table created. SQL> insert into t1 values (1); 1 row created. SQL> commit; Commit complete. SQL> select dbms_flashback.get_system_change_number scn from dual; SCN ---------- 8009717I want to take a backup of the container database. I'm going to use the 'report schema' and 'list backup' to show you what datafile and backups are available.
[oracle@vsi08devpom ~]$ export NLS_DATE_FORMAT='DD-MM-YYYY HH24:MI:SS' [oracle@vsi08devpom ~]$ rman target / Recovery Manager: Release 12.1.0.1.0 - Production on Wed Feb 12 10:39:12 2014 Copyright (c) 1982, 2013, Oracle and/or its affiliates. All rights reserved. connected to target database: CDB001 (DBID=4134963396) RMAN> report schema; using target database control file instead of recovery catalog Report of database schema for database with db_unique_name CDB001 List of Permanent Datafiles =========================== File Size(MB) Tablespace RB segs Datafile Name ---- -------- -------------------- ------- ------------------------ 1 790 SYSTEM *** /opt/app/oracle/oradata/CDB001/system01.dbf 3 2070 SYSAUX *** /opt/app/oracle/oradata/CDB001/sysaux01.dbf 4 150 UNDOTBS1 *** /opt/app/oracle/oradata/CDB001/undotbs01.dbf 5 250 PDB$SEED:SYSTEM *** /opt/app/oracle/oradata/CDB001/pdbseed/system01.dbf 6 5 USERS *** /opt/app/oracle/oradata/CDB001/users01.dbf 7 590 PDB$SEED:SYSAUX *** /opt/app/oracle/oradata/CDB001/pdbseed/sysaux01.dbf 8 270 PDB0101:SYSTEM *** /opt/app/oracle/oradata/CDB001/PDB0101/system01.dbf 9 640 PDB0101:SYSAUX *** /opt/app/oracle/oradata/CDB001/PDB0101/sysaux01.dbf 10 5 PDB0101:USERS *** /opt/app/oracle/oradata/CDB001/PDB0101/PDB0101_users01.dbf List of Temporary Files ======================= File Size(MB) Tablespace Maxsize(MB) Tempfile Name ---- -------- -------------------- ----------- -------------------- 1 60 TEMP 32767 /opt/app/oracle/oradata/CDB001/temp01.dbf 2 20 PDB$SEED:TEMP 32767 /opt/app/oracle/oradata/CDB001/pdbseed/pdbseed_temp01.dbf 3 20 PDB0101:TEMP 32767 /opt/app/oracle/oradata/CDB001/PDB0101/temp01.dbf RMAN> list backup of database; specification does not match any backup in the repositoryNo backup is available so I need to take at least one backup. I'm going to take a full database backup of the CDB and its PDBs.
RMAN> backup database plus archivelog; Starting backup at 12-02-2014 10:39:47 current log archived allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=274 device type=DISK channel ORA_DISK_1: starting archived log backup set channel ORA_DISK_1: specifying archived log(s) in backup set input archived log thread=1 sequence=659 RECID=10 STAMP=838910191 input archived log thread=1 sequence=660 RECID=11 STAMP=838910385 input archived log thread=1 sequence=661 RECID=12 STAMP=838911248 input archived log thread=1 sequence=662 RECID=13 STAMP=838911290 input archived log thread=1 sequence=663 RECID=14 STAMP=838936876 input archived log thread=1 sequence=664 RECID=15 STAMP=838954825 input archived log thread=1 sequence=665 RECID=16 STAMP=838969262 input archived log thread=1 sequence=666 RECID=17 STAMP=838990856 input archived log thread=1 sequence=667 RECID=18 STAMP=839010107 input archived log thread=1 sequence=668 RECID=19 STAMP=839025149 input archived log thread=1 sequence=669 RECID=20 STAMP=839052023 input archived log thread=1 sequence=670 RECID=21 STAMP=839066666 input archived log thread=1 sequence=671 RECID=22 STAMP=839084422 input archived log thread=1 sequence=672 RECID=23 STAMP=839106050 input archived log thread=1 sequence=673 RECID=24 STAMP=839134863 input archived log thread=1 sequence=674 RECID=25 STAMP=839159419 input archived log thread=1 sequence=675 RECID=26 STAMP=839159631 input archived log thread=1 sequence=676 RECID=27 STAMP=839160939 input archived log thread=1 sequence=677 RECID=28 STAMP=839160961 input archived log thread=1 sequence=678 RECID=29 STAMP=839162174 input archived log thread=1 sequence=679 RECID=30 STAMP=839162192 input archived log thread=1 sequence=680 RECID=31 STAMP=839196037 input archived log thread=1 sequence=681 RECID=32 STAMP=839199610 input archived log thread=1 sequence=682 RECID=33 STAMP=839232033 input archived log thread=1 sequence=683 RECID=34 STAMP=839275218 input archived log thread=1 sequence=684 RECID=35 STAMP=839289656 input archived log thread=1 sequence=685 RECID=36 STAMP=839322019 input archived log thread=1 sequence=686 RECID=37 STAMP=839327987 channel ORA_DISK_1: starting piece 1 at 12-02-2014 10:39:47 channel ORA_DISK_1: finished piece 1 at 12-02-2014 10:40:22 piece handle=/opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_annnn_TAG20140212T103947_9hpjc3vp_.bkp tag=TAG20140212T103947 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:35 Finished backup at 12-02-2014 10:40:22 Starting backup at 12-02-2014 10:40:22 using channel ORA_DISK_1 channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00003 name=/opt/app/oracle/oradata/CDB001/sysaux01.dbf input datafile file number=00001 name=/opt/app/oracle/oradata/CDB001/system01.dbf input datafile file number=00004 name=/opt/app/oracle/oradata/CDB001/undotbs01.dbf input datafile file number=00006 name=/opt/app/oracle/oradata/CDB001/users01.dbf channel ORA_DISK_1: starting piece 1 at 12-02-2014 10:40:23 channel ORA_DISK_1: finished piece 1 at 12-02-2014 10:42:08 piece handle=/opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjd76g_.bkp tag=TAG20140212T104022 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:01:45 channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00009 name=/opt/app/oracle/oradata/CDB001/PDB0101/sysaux01.dbf input datafile file number=00008 name=/opt/app/oracle/oradata/CDB001/PDB0101/system01.dbf input datafile file number=00010 name=/opt/app/oracle/oradata/CDB001/PDB0101/PDB0101_users01.dbf channel ORA_DISK_1: starting piece 1 at 12-02-2014 10:42:08 channel ORA_DISK_1: finished piece 1 at 12-02-2014 10:42:53 piece handle=/opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp tag=TAG20140212T104022 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:45 channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00007 name=/opt/app/oracle/oradata/CDB001/pdbseed/sysaux01.dbf input datafile file number=00005 name=/opt/app/oracle/oradata/CDB001/pdbseed/system01.dbf channel ORA_DISK_1: starting piece 1 at 12-02-2014 10:42:53 channel ORA_DISK_1: finished piece 1 at 12-02-2014 10:43:28 piece handle=/opt/app/oracle/fast_recovery_area/CDB001/E18E1CE36B940C96E0430100007FE168/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjjxhq_.bkp tag=TAG20140212T104022 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:35 Finished backup at 12-02-2014 10:43:28 Starting backup at 12-02-2014 10:43:28 current log archived using channel ORA_DISK_1 channel ORA_DISK_1: starting archived log backup set channel ORA_DISK_1: specifying archived log(s) in backup set input archived log thread=1 sequence=687 RECID=38 STAMP=839328208 channel ORA_DISK_1: starting piece 1 at 12-02-2014 10:43:28 channel ORA_DISK_1: finished piece 1 at 12-02-2014 10:43:29 piece handle=/opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_annnn_TAG20140212T104328_9hpjl0x9_.bkp tag=TAG20140212T104328 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01 Finished backup at 12-02-2014 10:43:29 Starting Control File and SPFILE Autobackup at 12-02-2014 10:43:30 piece handle=/opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839328210_9hpjl2xk_.bkp comment=NONE Finished Control File and SPFILE Autobackup at 12-02-2014 10:43:33Once the backup is completed I know I have a valid copy of MARCOV.T1 table in one RMAN backup set.
Let's check the current SCN after the backup is taken.
SQL> show user USER is "MARCOV" SQL> select dbms_flashback.get_system_change_number scn from dual; SCN ---------- 8010280I want to create another table, T2, on MARCOV schema inside the pluggable database PDB0101.
SQL> create table t2 (a number); Table created. SQL> insert into t2 values (1); 1 row created. SQL> commit; Commit complete. SQL> select dbms_flashback.get_system_change_number scn from dual; SCN ---------- 8010338At SCN number 8010338 I have a valid backup containing the MARCOV.T1 table but not the MARCOV.T2 and still the two available tables on the MARCOV schema.
I'm dropping "inadvertently" MARCOV.T1 table, but without the PURGE option. Only one table is still owned by MARCOV user querying the USER_TABLES view. Anyway something is present into the so called recyclebin: it's original name is T1.
SQL> drop table t1; Table dropped. SQL> select table_name from user_tables; TABLE_NAME -------------------------------------------------- T2 SQL> select * from tab; TNAME TABTYPE CLUSTERID -------------------------------------------------- ------- ---------- T2 TABLE BIN$8b6luRpCd8ngQwEAAH9AuA==$0 TABLE SQL> show recyclebin ORIGINAL NAME RECYCLEBIN NAME OBJECT TYPE DROP TIME ---------------- ------------------------------ ------------ ------------------- T1 BIN$8b6luRpCd8ngQwEAAH9AuA==$0 TABLE 2014-02-12:10:48:38Using the flashback table command I can "extract" from the recyclebin the T1 table so that its rows could be selected again.
SQL> flashback table "BIN$8b6luRpCd8ngQwEAAH9AuA==$0" to before drop; Flashback complete. SQL> select table_name from user_tables; TABLE_NAME -------------------------------------------------- T1 T2 SQL> select count(*) from t1; COUNT(*) ---------- 1Now I want to finally test the RMAN recover table command so I need to use again the drop command with the purge option to avoid the recyclebin and definitively lose my tables.
SQL> drop table t1 purge; Table dropped. SQL> drop table t2 purge; Table dropped. SQL> show recyclebinI've lost both tables and I'm wondering if the recover table command will be able to recover them.
During the recovery process RMAN will create and start a new "temporary" instance: the instance needs a valid location where to write its datafiles taken from the previous backup. I want to create this location just under the default location of the pluggable database PDB0101.
[oracle@vsi08devpom ~]$ mkdir /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLEAs first attempt I want to try a recover of MARCOV.T1 table using a SCN before the available backup was taken.
I'm expecting that the recover operation is going to fail.
RMAN> recover table marcov.t1 of pluggable database PDB0101 until scn 8009717 auxiliary destination '/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE'; Starting recover at 12-02-2014 11:07:32 using channel ORA_DISK_1 RMAN-05026: WARNING: presuming following set of tablespaces applies to specified Point-in-Time List of tablespaces expected to have UNDO segments Tablespace SYSTEM Tablespace UNDOTBS1 Creating automatic instance, with SID='FBzE' initialization parameters used for automatic instance: db_name=CDB001 db_unique_name=FBzE_pitr_PDB0101_CDB001 compatible=12.1.0.0.0 db_block_size=8192 db_files=200 sga_target=1G processes=80 diagnostic_dest=/opt/app/oracle db_create_file_dest=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE log_archive_dest_1='location=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE' enable_pluggable_database=true _clone_one_pdb_recovery=true #No auxiliary parameter file used starting up automatic instance CDB001 Oracle instance started Total System Global Area 1068937216 bytes Fixed Size 2296576 bytes Variable Size 281019648 bytes Database Buffers 780140544 bytes Redo Buffers 5480448 bytes Automatic instance created contents of Memory Script: { # set requested point in time set until scn 8009717; # restore the controlfile restore clone controlfile; # mount the controlfile sql clone 'alter database mount clone database'; # archive current online log sql 'alter system archive log current'; } executing Memory Script executing command: SET until clause Starting restore at 12-02-2014 11:07:46 allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=11 device type=DISK Removing automatic instance shutting down automatic instance Oracle instance shut down Automatic instance removed RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 02/12/2014 11:07:48 RMAN-03015: error occurred in stored script Memory Script RMAN-06026: some targets not found - aborting restore RMAN-06024: no backup or copy of the control file found to restoreThe recover command failed because no available backup was created before the SCN proposed. Let's see what happens if I try to recover both tables specifying a SCN after I completed the backup and just after the creation of the second table.
RMAN> recover table marcov.t1, marcov.t2 of pluggable database PDB0101 until scn 8010338 auxiliary destination '/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE'; Starting recover at 12-02-2014 11:15:55 current log archived using channel ORA_DISK_1 RMAN-05026: WARNING: presuming following set of tablespaces applies to specified Point-in-Time List of tablespaces expected to have UNDO segments Tablespace SYSTEM Tablespace UNDOTBS1 Creating automatic instance, with SID='aBwj' initialization parameters used for automatic instance: db_name=CDB001 db_unique_name=aBwj_pitr_PDB0101_CDB001 compatible=12.1.0.0.0 db_block_size=8192 db_files=200 sga_target=1G processes=80 diagnostic_dest=/opt/app/oracle db_create_file_dest=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE log_archive_dest_1='location=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE' enable_pluggable_database=true _clone_one_pdb_recovery=true #No auxiliary parameter file used starting up automatic instance CDB001 Oracle instance started Total System Global Area 1068937216 bytes Fixed Size 2296576 bytes Variable Size 281019648 bytes Database Buffers 780140544 bytes Redo Buffers 5480448 bytes Automatic instance created contents of Memory Script: { # set requested point in time set until scn 8010338; # restore the controlfile restore clone controlfile; # mount the controlfile sql clone 'alter database mount clone database'; # archive current online log sql 'alter system archive log current'; } executing Memory Script executing command: SET until clause Starting restore at 12-02-2014 11:16:08 allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=82 device type=DISK channel ORA_AUX_DISK_1: starting datafile backup set restore channel ORA_AUX_DISK_1: restoring control file channel ORA_AUX_DISK_1: reading from backup piece /opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839328210_9hpjl2xk_.bkp channel ORA_AUX_DISK_1: piece handle=/opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839328210_9hpjl2xk_.bkp tag=TAG20140212T104330 channel ORA_AUX_DISK_1: restored backup piece 1 channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01 output file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hplh8ry_.ctl Finished restore at 12-02-2014 11:16:09 sql statement: alter database mount clone database sql statement: alter system archive log current contents of Memory Script: { # set requested point in time set until scn 8010338; # set destinations for recovery set and auxiliary set datafiles set newname for clone datafile 1 to new; set newname for clone datafile 4 to new; set newname for clone datafile 3 to new; set newname for clone datafile 8 to new; set newname for clone datafile 9 to new; set newname for clone tempfile 1 to new; set newname for clone tempfile 3 to new; # switch all tempfiles switch clone tempfile all; # restore the tablespaces in the recovery set and the auxiliary set restore clone datafile 1, 4, 3, 8, 9; switch clone datafile all; } executing Memory Script executing command: SET until clause executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME renamed tempfile 1 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_%u_.tmp in control file renamed tempfile 3 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_%u_.tmp in control file Starting restore at 12-02-2014 11:16:14 using channel ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: starting datafile backup set restore channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set channel ORA_AUX_DISK_1: restoring datafile 00001 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_%u_.dbf channel ORA_AUX_DISK_1: restoring datafile 00004 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_undotbs1_%u_.dbf channel ORA_AUX_DISK_1: restoring datafile 00003 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_%u_.dbf channel ORA_AUX_DISK_1: reading from backup piece /opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjd76g_.bkp channel ORA_AUX_DISK_1: piece handle=/opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjd76g_.bkp tag=TAG20140212T104022 channel ORA_AUX_DISK_1: restored backup piece 1 channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:01:35 channel ORA_AUX_DISK_1: starting datafile backup set restore channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set channel ORA_AUX_DISK_1: restoring datafile 00008 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_%u_.dbf channel ORA_AUX_DISK_1: restoring datafile 00009 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_%u_.dbf channel ORA_AUX_DISK_1: reading from backup piece /opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp channel ORA_AUX_DISK_1: piece handle=/opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp tag=TAG20140212T104022 channel ORA_AUX_DISK_1: restored backup piece 1 channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:35 Finished restore at 12-02-2014 11:18:25 datafile 1 switched to datafile copy input datafile copy RECID=8 STAMP=839330305 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_9hplhh9p_.dbf datafile 4 switched to datafile copy input datafile copy RECID=9 STAMP=839330305 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_undotbs1_9hplhhbk_.dbf datafile 3 switched to datafile copy input datafile copy RECID=10 STAMP=839330305 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_9hplhh8t_.dbf datafile 8 switched to datafile copy input datafile copy RECID=11 STAMP=839330305 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_9hpllgjp_.dbf datafile 9 switched to datafile copy input datafile copy RECID=12 STAMP=839330305 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_9hpllg7d_.dbf contents of Memory Script: { # set requested point in time set until scn 8010338; # online the datafiles restored or switched sql clone "alter database datafile 1 online"; sql clone "alter database datafile 4 online"; sql clone "alter database datafile 3 online"; sql clone 'PDB0101' "alter database datafile 8 online"; sql clone 'PDB0101' "alter database datafile 9 online"; # recover and open database read only recover clone database tablespace "SYSTEM", "UNDOTBS1", "SYSAUX", "PDB0101":"SYSTEM", "PDB0101":"SYSAUX"; sql clone 'alter database open read only'; } executing Memory Script executing command: SET until clause sql statement: alter database datafile 1 online sql statement: alter database datafile 4 online sql statement: alter database datafile 3 online sql statement: alter database datafile 8 online sql statement: alter database datafile 9 online Starting recover at 12-02-2014 11:18:25 using channel ORA_AUX_DISK_1 starting media recovery archived log for thread 1 with sequence 687 is already on disk as file /opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_687_9hpjl0pr_.arc archived log for thread 1 with sequence 688 is already on disk as file /opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_688_9hplgw0k_.arc archived log file name=/opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_687_9hpjl0pr_.arc thread=1 sequence=687 archived log file name=/opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_688_9hplgw0k_.arc thread=1 sequence=688 media recovery complete, elapsed time: 00:00:02 Finished recover at 12-02-2014 11:18:29 sql statement: alter database open read only contents of Memory Script: { sql clone 'alter pluggable database PDB0101 open read only'; } executing Memory Script sql statement: alter pluggable database PDB0101 open read only contents of Memory Script: { sql clone "create spfile from memory"; shutdown clone immediate; startup clone nomount; sql clone "alter system set control_files = ''/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hplh8ry_.ctl'' comment= ''RMAN set'' scope=spfile"; shutdown clone immediate; startup clone nomount; # mount database sql clone 'alter database mount clone database'; } executing Memory Script sql statement: create spfile from memory database closed database dismounted Oracle instance shut down connected to auxiliary database (not started) Oracle instance started Total System Global Area 1068937216 bytes Fixed Size 2296576 bytes Variable Size 285213952 bytes Database Buffers 775946240 bytes Redo Buffers 5480448 bytes sql statement: alter system set control_files = ''/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hplh8ry_.ctl'' comment= ''RMAN set'' scope=spfile Oracle instance shut down connected to auxiliary database (not started) Oracle instance started Total System Global Area 1068937216 bytes Fixed Size 2296576 bytes Variable Size 285213952 bytes Database Buffers 775946240 bytes Redo Buffers 5480448 bytes sql statement: alter database mount clone database contents of Memory Script: { # set requested point in time set until scn 8010338; # set destinations for recovery set and auxiliary set datafiles set newname for datafile 10 to new; # restore the tablespaces in the recovery set and the auxiliary set restore clone datafile 10; switch clone datafile all; } executing Memory Script executing command: SET until clause executing command: SET NEWNAME Starting restore at 12-02-2014 11:19:08 allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=11 device type=DISK channel ORA_AUX_DISK_1: starting datafile backup set restore channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set channel ORA_AUX_DISK_1: restoring datafile 00010 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/ABWJ_PITR_PDB0101_CDB001/datafile/o1_mf_users_%u_.dbf channel ORA_AUX_DISK_1: reading from backup piece /opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp channel ORA_AUX_DISK_1: piece handle=/opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp tag=TAG20140212T104022 channel ORA_AUX_DISK_1: restored backup piece 1 channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01 Finished restore at 12-02-2014 11:19:09 datafile 10 switched to datafile copy input datafile copy RECID=14 STAMP=839330350 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/ABWJ_PITR_PDB0101_CDB001/datafile/o1_mf_users_9hplnwvb_.dbf contents of Memory Script: { # set requested point in time set until scn 8010338; # online the datafiles restored or switched sql clone 'PDB0101' "alter database datafile 10 online"; # recover and open resetlogs recover clone database tablespace "PDB0101":"USERS", "SYSTEM", "UNDOTBS1", "SYSAUX", "PDB0101":"SYSTEM", "PDB0101":"SYSAUX" delete archivelog; alter clone database open resetlogs; } executing Memory Script executing command: SET until clause sql statement: alter database datafile 10 online Starting recover at 12-02-2014 11:19:09 using channel ORA_AUX_DISK_1 starting media recovery archived log for thread 1 with sequence 687 is already on disk as file /opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_687_9hpjl0pr_.arc archived log for thread 1 with sequence 688 is already on disk as file /opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_688_9hplgw0k_.arc archived log file name=/opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_687_9hpjl0pr_.arc thread=1 sequence=687 archived log file name=/opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_688_9hplgw0k_.arc thread=1 sequence=688 media recovery complete, elapsed time: 00:00:00 Finished recover at 12-02-2014 11:19:10 database opened contents of Memory Script: { sql clone 'alter pluggable database PDB0101 open'; } executing Memory Script sql statement: alter pluggable database PDB0101 open contents of Memory Script: { # create directory for datapump import sql 'PDB0101' "create or replace directory TSPITR_DIROBJ_DPDIR as '' /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE''"; # create directory for datapump export sql clone 'PDB0101' "create or replace directory TSPITR_DIROBJ_DPDIR as '' /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE''"; } executing Memory Script sql statement: create or replace directory TSPITR_DIROBJ_DPDIR as ''/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE'' sql statement: create or replace directory TSPITR_DIROBJ_DPDIR as ''/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE'' Performing export of tables... EXPDP> Starting "SYS"."TSPITR_EXP_aBwj_wdxk": EXPDP> Estimate in progress using BLOCKS method... EXPDP> Processing object type TABLE_EXPORT/TABLE/TABLE_DATA EXPDP> Total estimation using BLOCKS method: 128 KB EXPDP> Processing object type TABLE_EXPORT/TABLE/TABLE EXPDP> Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS EXPDP> Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER EXPDP> . . exported "MARCOV"."T1" 5.031 KB 1 rows EXPDP> . . exported "MARCOV"."T2" 5.031 KB 1 rows EXPDP> Master table "SYS"."TSPITR_EXP_aBwj_wdxk" successfully loaded/unloaded EXPDP> ****************************************************************************** EXPDP> Dump file set for SYS.TSPITR_EXP_aBwj_wdxk is: EXPDP> /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/tspitr_aBwj_36569.dmp EXPDP> Job "SYS"."TSPITR_EXP_aBwj_wdxk" successfully completed at Wed Feb 12 11:20:04 2014 elapsed 0 00:00:26 Export completed contents of Memory Script: { # shutdown clone before import shutdown clone abort } executing Memory Script Oracle instance shut down Performing import of tables... IMPDP> Master table "SYS"."TSPITR_IMP_aBwj_woyn" successfully loaded/unloaded IMPDP> Starting "SYS"."TSPITR_IMP_aBwj_woyn": IMPDP> Processing object type TABLE_EXPORT/TABLE/TABLE IMPDP> Processing object type TABLE_EXPORT/TABLE/TABLE_DATA IMPDP> . . imported "MARCOV"."T1" 5.031 KB 1 rows IMPDP> . . imported "MARCOV"."T2" 5.031 KB 1 rows IMPDP> Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS IMPDP> Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER IMPDP> Job "SYS"."TSPITR_IMP_aBwj_woyn" successfully completed at Wed Feb 12 11:21:16 2014 elapsed 0 00:01:00 Import completed Removing automatic instance Automatic instance removed auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_9hplmz8c_.tmp deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_9hplmph9_.tmp deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/ABWJ_PITR_PDB0101_CDB001/onlinelog/o1_mf_3_9hplo18l_.log deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/ABWJ_PITR_PDB0101_CDB001/onlinelog/o1_mf_2_9hplo04t_.log deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/ABWJ_PITR_PDB0101_CDB001/onlinelog/o1_mf_1_9hplnz1q_.log deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/ABWJ_PITR_PDB0101_CDB001/datafile/o1_mf_users_9hplnwvb_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_9hpllg7d_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_9hpllgjp_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_9hplhh8t_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_undotbs1_9hplhhbk_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_9hplhh9p_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hplh8ry_.ctl deleted auxiliary instance file tspitr_aBwj_36569.dmp deleted Finished recover at 12-02-2014 11:21:17The recover command successfully completed its steps.
It was able to create an auxiliary database named aBwj, use it to restore all the backup and the archivelog for the two tables, export and import their rows into the target pluggable database PDB0101. As you can see during the recovery steps a new aBwj instance is created...
[root@vsi08devpom ~]# ps -ef|grep smon oracle 20271 1 0 11:19 ? 00:00:00 ora_smon_aBwj root 20592 19983 0 11:19 pts/2 00:00:00 grep smon oracle 29331 1 0 Feb06 ? 00:00:26 ora_smon_CDB001... and then removed.
[root@vsi08devpom ~]# ps -ef|grep smon root 20873 19983 0 11:38 pts/2 00:00:00 grep smon oracle 29331 1 0 Feb06 ? 00:00:26 ora_smon_CDB001Inside MARCOV schema of the pluggable database PDB0101 the two tables are available again.
SQL> show user USER is "MARCOV" SQL> select dbms_flashback.get_system_change_number scn from dual; SCN ---------- 8013207 SQL> select * from t1; A ---------- 1 SQL> select * from t2; A ---------- 1Let's drop again MARCOV.T1 table and try its recover at a SCN after the backup but prior the creation of MARCOV.T2 table.
Will be MARCOV.T2 table still available ?
SQL> drop table t1 purge; Table dropped.Here is the recover command to a different specific SCN.
Find 8010280 inside this web page to see when this SCN was calculated.
RMAN> recover table marcov.t1 of pluggable database PDB0101 until scn 8010280 auxiliary destination '/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE'; Starting recover at 12-02-2014 11:43:54 using channel ORA_DISK_1 RMAN-05026: WARNING: presuming following set of tablespaces applies to specified Point-in-Time List of tablespaces expected to have UNDO segments Tablespace SYSTEM Tablespace UNDOTBS1 Creating automatic instance, with SID='plxu' initialization parameters used for automatic instance: db_name=CDB001 db_unique_name=plxu_pitr_PDB0101_CDB001 compatible=12.1.0.0.0 db_block_size=8192 db_files=200 sga_target=1G processes=80 diagnostic_dest=/opt/app/oracle db_create_file_dest=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE log_archive_dest_1='location=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE' enable_pluggable_database=true _clone_one_pdb_recovery=true #No auxiliary parameter file used starting up automatic instance CDB001 Oracle instance started Total System Global Area 1068937216 bytes Fixed Size 2296576 bytes Variable Size 281019648 bytes Database Buffers 780140544 bytes Redo Buffers 5480448 bytes Automatic instance created contents of Memory Script: { # set requested point in time set until scn 8010280; # restore the controlfile restore clone controlfile; # mount the controlfile sql clone 'alter database mount clone database'; # archive current online log sql 'alter system archive log current'; } executing Memory Script executing command: SET until clause Starting restore at 12-02-2014 11:44:06 allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=82 device type=DISK channel ORA_AUX_DISK_1: starting datafile backup set restore channel ORA_AUX_DISK_1: restoring control file channel ORA_AUX_DISK_1: reading from backup piece /opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839328210_9hpjl2xk_.bkp channel ORA_AUX_DISK_1: piece handle=/opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839328210_9hpjl2xk_.bkp tag=TAG20140212T104330 channel ORA_AUX_DISK_1: restored backup piece 1 channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01 output file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hpn3pjf_.ctl Finished restore at 12-02-2014 11:44:07 sql statement: alter database mount clone database sql statement: alter system archive log current contents of Memory Script: { # set requested point in time set until scn 8010280; # set destinations for recovery set and auxiliary set datafiles set newname for clone datafile 1 to new; set newname for clone datafile 4 to new; set newname for clone datafile 3 to new; set newname for clone datafile 8 to new; set newname for clone datafile 9 to new; set newname for clone tempfile 1 to new; set newname for clone tempfile 3 to new; # switch all tempfiles switch clone tempfile all; # restore the tablespaces in the recovery set and the auxiliary set restore clone datafile 1, 4, 3, 8, 9; switch clone datafile all; } executing Memory Script executing command: SET until clause executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME renamed tempfile 1 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_%u_.tmp in control file renamed tempfile 3 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_%u_.tmp in control file Starting restore at 12-02-2014 11:44:12 using channel ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: starting datafile backup set restore channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set channel ORA_AUX_DISK_1: restoring datafile 00001 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_%u_.dbf channel ORA_AUX_DISK_1: restoring datafile 00004 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_undotbs1_%u_.dbf channel ORA_AUX_DISK_1: restoring datafile 00003 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_%u_.dbf channel ORA_AUX_DISK_1: reading from backup piece /opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjd76g_.bkp channel ORA_AUX_DISK_1: piece handle=/opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjd76g_.bkp tag=TAG20140212T104022 channel ORA_AUX_DISK_1: restored backup piece 1 channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:01:35 channel ORA_AUX_DISK_1: starting datafile backup set restore channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set channel ORA_AUX_DISK_1: restoring datafile 00008 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_%u_.dbf channel ORA_AUX_DISK_1: restoring datafile 00009 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_%u_.dbf channel ORA_AUX_DISK_1: reading from backup piece /opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp channel ORA_AUX_DISK_1: piece handle=/opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp tag=TAG20140212T104022 channel ORA_AUX_DISK_1: restored backup piece 1 channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:35 Finished restore at 12-02-2014 11:46:22 datafile 1 switched to datafile copy input datafile copy RECID=8 STAMP=839331983 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_9hpn3x6z_.dbf datafile 4 switched to datafile copy input datafile copy RECID=9 STAMP=839331983 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_undotbs1_9hpn3x7s_.dbf datafile 3 switched to datafile copy input datafile copy RECID=10 STAMP=839331983 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_9hpn3x60_.dbf datafile 8 switched to datafile copy input datafile copy RECID=11 STAMP=839331983 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_9hpn6w4d_.dbf datafile 9 switched to datafile copy input datafile copy RECID=12 STAMP=839331983 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_9hpn6w3t_.dbf contents of Memory Script: { # set requested point in time set until scn 8010280; # online the datafiles restored or switched sql clone "alter database datafile 1 online"; sql clone "alter database datafile 4 online"; sql clone "alter database datafile 3 online"; sql clone 'PDB0101' "alter database datafile 8 online"; sql clone 'PDB0101' "alter database datafile 9 online"; # recover and open database read only recover clone database tablespace "SYSTEM", "UNDOTBS1", "SYSAUX", "PDB0101":"SYSTEM", "PDB0101":"SYSAUX"; sql clone 'alter database open read only'; } executing Memory Script executing command: SET until clause sql statement: alter database datafile 1 online sql statement: alter database datafile 4 online sql statement: alter database datafile 3 online sql statement: alter database datafile 8 online sql statement: alter database datafile 9 online Starting recover at 12-02-2014 11:46:23 using channel ORA_AUX_DISK_1 starting media recovery archived log for thread 1 with sequence 687 is already on disk as file /opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_687_9hpjl0pr_.arc archived log for thread 1 with sequence 688 is already on disk as file /opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_688_9hplgw0k_.arc archived log file name=/opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_687_9hpjl0pr_.arc thread=1 sequence=687 archived log file name=/opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_688_9hplgw0k_.arc thread=1 sequence=688 media recovery complete, elapsed time: 00:00:01 Finished recover at 12-02-2014 11:46:25 sql statement: alter database open read only contents of Memory Script: { sql clone 'alter pluggable database PDB0101 open read only'; } executing Memory Script sql statement: alter pluggable database PDB0101 open read only contents of Memory Script: { sql clone "create spfile from memory"; shutdown clone immediate; startup clone nomount; sql clone "alter system set control_files = ''/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hpn3pjf_.ctl'' comment= ''RMAN set'' scope=spfile"; shutdown clone immediate; startup clone nomount; # mount database sql clone 'alter database mount clone database'; } executing Memory Script sql statement: create spfile from memory database closed database dismounted Oracle instance shut down connected to auxiliary database (not started) Oracle instance started Total System Global Area 1068937216 bytes Fixed Size 2296576 bytes Variable Size 285213952 bytes Database Buffers 775946240 bytes Redo Buffers 5480448 bytes sql statement: alter system set control_files = ''/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hpn3pjf_.ctl'' comment= ''RMAN set'' scope=spfile Oracle instance shut down connected to auxiliary database (not started) Oracle instance started Total System Global Area 1068937216 bytes Fixed Size 2296576 bytes Variable Size 285213952 bytes Database Buffers 775946240 bytes Redo Buffers 5480448 bytes sql statement: alter database mount clone database contents of Memory Script: { # set requested point in time set until scn 8010280; # set destinations for recovery set and auxiliary set datafiles set newname for datafile 10 to new; # restore the tablespaces in the recovery set and the auxiliary set restore clone datafile 10; switch clone datafile all; } executing Memory Script executing command: SET until clause executing command: SET NEWNAME Starting restore at 12-02-2014 11:46:57 allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=11 device type=DISK channel ORA_AUX_DISK_1: starting datafile backup set restore channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set channel ORA_AUX_DISK_1: restoring datafile 00010 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/PLXU_PITR_PDB0101_CDB001/datafile/o1_mf_users_%u_.dbf channel ORA_AUX_DISK_1: reading from backup piece /opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp channel ORA_AUX_DISK_1: piece handle=/opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp tag=TAG20140212T104022 channel ORA_AUX_DISK_1: restored backup piece 1 channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01 Finished restore at 12-02-2014 11:46:59 datafile 10 switched to datafile copy input datafile copy RECID=14 STAMP=839332019 file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/PLXU_PITR_PDB0101_CDB001/datafile/o1_mf_users_9hpn92hl_.dbf contents of Memory Script: { # set requested point in time set until scn 8010280; # online the datafiles restored or switched sql clone 'PDB0101' "alter database datafile 10 online"; # recover and open resetlogs recover clone database tablespace "PDB0101":"USERS", "SYSTEM", "UNDOTBS1", "SYSAUX", "PDB0101":"SYSTEM", "PDB0101":"SYSAUX" delete archivelog; alter clone database open resetlogs; } executing Memory Script executing command: SET until clause sql statement: alter database datafile 10 online Starting recover at 12-02-2014 11:46:59 using channel ORA_AUX_DISK_1 starting media recovery archived log for thread 1 with sequence 687 is already on disk as file /opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_687_9hpjl0pr_.arc archived log for thread 1 with sequence 688 is already on disk as file /opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_688_9hplgw0k_.arc archived log file name=/opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_687_9hpjl0pr_.arc thread=1 sequence=687 archived log file name=/opt/app/oracle/fast_recovery_area/CDB001/archivelog/2014_02_12/o1_mf_1_688_9hplgw0k_.arc thread=1 sequence=688 media recovery complete, elapsed time: 00:00:00 Finished recover at 12-02-2014 11:47:00 database opened contents of Memory Script: { sql clone 'alter pluggable database PDB0101 open'; } executing Memory Script sql statement: alter pluggable database PDB0101 open contents of Memory Script: { # create directory for datapump import sql 'PDB0101' "create or replace directory TSPITR_DIROBJ_DPDIR as '' /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE''"; # create directory for datapump export sql clone 'PDB0101' "create or replace directory TSPITR_DIROBJ_DPDIR as '' /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE''"; } executing Memory Script sql statement: create or replace directory TSPITR_DIROBJ_DPDIR as ''/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE'' sql statement: create or replace directory TSPITR_DIROBJ_DPDIR as ''/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE'' Performing export of tables... EXPDP> Starting "SYS"."TSPITR_EXP_plxu_rzrl": EXPDP> Estimate in progress using BLOCKS method... EXPDP> Processing object type TABLE_EXPORT/TABLE/TABLE_DATA EXPDP> Total estimation using BLOCKS method: 64 KB EXPDP> Processing object type TABLE_EXPORT/TABLE/TABLE EXPDP> Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS EXPDP> Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER EXPDP> . . exported "MARCOV"."T1" 5.031 KB 1 rows EXPDP> Master table "SYS"."TSPITR_EXP_plxu_rzrl" successfully loaded/unloaded EXPDP> ****************************************************************************** EXPDP> Dump file set for SYS.TSPITR_EXP_plxu_rzrl is: EXPDP> /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/tspitr_plxu_39412.dmp EXPDP> Job "SYS"."TSPITR_EXP_plxu_rzrl" successfully completed at Wed Feb 12 11:47:49 2014 elapsed 0 00:00:23 Export completed contents of Memory Script: { # shutdown clone before import shutdown clone abort } executing Memory Script Oracle instance shut down Performing import of tables... IMPDP> Master table "SYS"."TSPITR_IMP_plxu_ixEA" successfully loaded/unloaded IMPDP> Starting "SYS"."TSPITR_IMP_plxu_ixEA": IMPDP> Processing object type TABLE_EXPORT/TABLE/TABLE IMPDP> Processing object type TABLE_EXPORT/TABLE/TABLE_DATA IMPDP> . . imported "MARCOV"."T1" 5.031 KB 1 rows IMPDP> Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS IMPDP> Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER IMPDP> Job "SYS"."TSPITR_IMP_plxu_ixEA" successfully completed at Wed Feb 12 11:48:53 2014 elapsed 0 00:00:58 Import completed Removing automatic instance Automatic instance removed auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_9hpn88ho_.tmp deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_9hpn82js_.tmp deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/PLXU_PITR_PDB0101_CDB001/onlinelog/o1_mf_3_9hpn96tp_.log deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/PLXU_PITR_PDB0101_CDB001/onlinelog/o1_mf_2_9hpn95pb_.log deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/PLXU_PITR_PDB0101_CDB001/onlinelog/o1_mf_1_9hpn94lp_.log deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/PLXU_PITR_PDB0101_CDB001/datafile/o1_mf_users_9hpn92hl_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_9hpn6w3t_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_9hpn6w4d_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_sysaux_9hpn3x60_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_undotbs1_9hpn3x7s_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_system_9hpn3x6z_.dbf deleted auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hpn3pjf_.ctl deleted auxiliary instance file tspitr_plxu_39412.dmp deleted Finished recover at 12-02-2014 11:48:54The recover command is completed successfully again.
During its recovery operations RMAN created and removed the auxiliary instance named this time plxu.
[root@vsi08devpom ~]# ps -ef|grep smon oracle 20997 1 0 11:44 ? 00:00:00 ora_smon_plxu root 21051 19983 0 11:44 pts/2 00:00:00 grep smon oracle 29331 1 0 Feb06 ? 00:00:26 ora_smon_CDB001
[root@vsi08devpom ~]# ps -ef|grep smon root 21796 19983 0 11:58 pts/2 00:00:00 grep smon oracle 29331 1 0 Feb06 ? 00:00:26 ora_smon_CDB001Naturally the recover table process is not a flashback of the database so the MARCOV.T2 table is still there and always available to be queried.
SQL> select * from t1; A ---------- 1 SQL> select * from t2; A ---------- 1Now the interesting scenario... I want to remove the previous full container database backup...
RMAN> delete noprompt backup; using channel ORA_DISK_1 List of Backup Pieces BP Key BS Key Pc# Cp# Status Device Type Piece Name ------- ------- --- --- ----------- ----------- ---------- 24 24 1 1 AVAILABLE DISK /opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_annnn_TAG20140212T103947_9hpjc3vp_.bkp 25 25 1 1 AVAILABLE DISK /opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjd76g_.bkp 26 26 1 1 AVAILABLE DISK /opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp 27 27 1 1 AVAILABLE DISK /opt/app/oracle/fast_recovery_area/CDB001/E18E1CE36B940C96E0430100007FE168/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjjxhq_.bkp 28 28 1 1 AVAILABLE DISK /opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_annnn_TAG20140212T104328_9hpjl0x9_.bkp 29 29 1 1 AVAILABLE DISK /opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839328210_9hpjl2xk_.bkp deleted backup piece backup piece handle=/opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_annnn_TAG20140212T103947_9hpjc3vp_.bkp RECID=24 STAMP=839327987 deleted backup piece backup piece handle=/opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjd76g_.bkp RECID=25 STAMP=839328023 deleted backup piece backup piece handle=/opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjhjbw_.bkp RECID=26 STAMP=839328128 deleted backup piece backup piece handle=/opt/app/oracle/fast_recovery_area/CDB001/E18E1CE36B940C96E0430100007FE168/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T104022_9hpjjxhq_.bkp RECID=27 STAMP=839328173 deleted backup piece backup piece handle=/opt/app/oracle/fast_recovery_area/CDB001/backupset/2014_02_12/o1_mf_annnn_TAG20140212T104328_9hpjl0x9_.bkp RECID=28 STAMP=839328208 deleted backup piece backup piece handle=/opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839328210_9hpjl2xk_.bkp RECID=29 STAMP=839328210 Deleted 6 objects RMAN> list backup; specification does not match any backup in the repository... and execute a backup of just the pluggable database PDB0101.
RMAN> backup pluggable database PDB0101; Starting backup at 12-02-2014 12:00:29 using channel ORA_DISK_1 channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00009 name=/opt/app/oracle/oradata/CDB001/PDB0101/sysaux01.dbf input datafile file number=00008 name=/opt/app/oracle/oradata/CDB001/PDB0101/system01.dbf input datafile file number=00010 name=/opt/app/oracle/oradata/CDB001/PDB0101/PDB0101_users01.dbf channel ORA_DISK_1: starting piece 1 at 12-02-2014 12:00:29 channel ORA_DISK_1: finished piece 1 at 12-02-2014 12:01:24 piece handle=/opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T120029_9hpo2fnm_.bkp tag=TAG20140212T120029 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:55 Finished backup at 12-02-2014 12:01:24 Starting Control File and SPFILE Autobackup at 12-02-2014 12:01:24 piece handle=/opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839332884_9hpo458m_.bkp comment=NONE Finished Control File and SPFILE Autobackup at 12-02-2014 12:01:27 RMAN> list backup of pluggable database PDB0101; List of Backup Sets =================== BS Key Type LV Size Device Type Elapsed Time Completion Time ------- ---- -- ---------- ----------- ------------ ------------------- 30 Full 718.37M DISK 00:00:49 12-02-2014 12:01:18 BP Key: 30 Status: AVAILABLE Compressed: NO Tag: TAG20140212T120029 Piece Name: /opt/app/oracle/fast_recovery_area/CDB001/E18E282148FD10A5E0430100007FC94D/backupset/2014_02_12/o1_mf_nnndf_TAG20140212T120029_9hpo2fnm_.bkp List of Datafiles in backup set 30 Container ID: 3, PDB Name: PDB0101 File LV Type Ckp SCN Ckp Time Name ---- -- ---- ---------- ------------------- ---- 8 Full 8014530 12-02-2014 12:00:29 /opt/app/oracle/oradata/CDB001/PDB0101/system01.dbf 9 Full 8014530 12-02-2014 12:00:29 /opt/app/oracle/oradata/CDB001/PDB0101/sysaux01.dbf 10 Full 8014530 12-02-2014 12:00:29 /opt/app/oracle/oradata/CDB001/PDB0101/PDB0101_users01.dbfThe current SCN after that backup is 8015786.
SQL> select dbms_flashback.get_system_change_number scn from dual; SCN ---------- 8015786I'm going to simulate the lost of MARCOV.T2: I'm confident to recover it because I have the full pluggable database backup containing all the information of that table.
SQL> drop table t2 purge; Table dropped.By now it's just the time to execute the usual recover table command and wait for the creation of the auxiliary instance.
RMAN> recover table marcov.t2 of pluggable database PDB0101 until scn 8015786 auxiliary destination '/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE'; Starting recover at 12-02-2014 12:22:00 current log archived using channel ORA_DISK_1 RMAN-05026: WARNING: presuming following set of tablespaces applies to specified Point-in-Time List of tablespaces expected to have UNDO segments Tablespace SYSTEM Tablespace UNDOTBS1 Creating automatic instance, with SID='pknF' initialization parameters used for automatic instance: db_name=CDB001 db_unique_name=pknF_pitr_PDB0101_CDB001 compatible=12.1.0.0.0 db_block_size=8192 db_files=200 sga_target=1G processes=80 diagnostic_dest=/opt/app/oracle db_create_file_dest=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE log_archive_dest_1='location=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE' enable_pluggable_database=true _clone_one_pdb_recovery=true #No auxiliary parameter file used starting up automatic instance CDB001 Oracle instance started Total System Global Area 1068937216 bytes Fixed Size 2296576 bytes Variable Size 281019648 bytes Database Buffers 780140544 bytes Redo Buffers 5480448 bytes Automatic instance created contents of Memory Script: { # set requested point in time set until scn 8015786; # restore the controlfile restore clone controlfile; # mount the controlfile sql clone 'alter database mount clone database'; # archive current online log sql 'alter system archive log current'; } executing Memory Script executing command: SET until clause Starting restore at 12-02-2014 12:22:11 allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=11 device type=DISK channel ORA_AUX_DISK_1: starting datafile backup set restore channel ORA_AUX_DISK_1: restoring control file channel ORA_AUX_DISK_1: reading from backup piece /opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839332884_9hpo458m_.bkp channel ORA_AUX_DISK_1: piece handle=/opt/app/oracle/fast_recovery_area/CDB001/autobackup/2014_02_12/o1_mf_s_839332884_9hpo458m_.bkp tag=TAG20140212T120124 channel ORA_AUX_DISK_1: restored backup piece 1 channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01 output file name=/opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hppc4b3_.ctl Finished restore at 12-02-2014 12:22:13 sql statement: alter database mount clone database sql statement: alter system archive log current contents of Memory Script: { # set requested point in time set until scn 8015786; # set destinations for recovery set and auxiliary set datafiles set newname for clone datafile 1 to new; set newname for clone datafile 4 to new; set newname for clone datafile 3 to new; set newname for clone datafile 8 to new; set newname for clone datafile 9 to new; set newname for clone tempfile 1 to new; set newname for clone tempfile 3 to new; # switch all tempfiles switch clone tempfile all; # restore the tablespaces in the recovery set and the auxiliary set restore clone datafile 1, 4, 3, 8, 9; switch clone datafile all; } executing Memory Script executing command: SET until clause executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME renamed tempfile 1 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_%u_.tmp in control file renamed tempfile 3 to /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/datafile/o1_mf_temp_%u_.tmp in control file Starting restore at 12-02-2014 12:22:18 using channel ORA_AUX_DISK_1 One or more auxiliary set of datafiles could not be removed Removing automatic instance shutting down automatic instance Oracle instance shut down Automatic instance removed auxiliary instance file /opt/app/oracle/oradata/CDB001/PDB0101/RECOVERY_PURGED_TABLE/CDB001/controlfile/o1_mf_9hppc4b3_.ctl deleted RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 02/12/2014 12:22:18 RMAN-03015: error occurred in stored script Memory Script RMAN-06026: some targets not found - aborting restore RMAN-06023: no backup or copy of datafile 3 found to restore RMAN-06023: no backup or copy of datafile 4 found to restore RMAN-06023: no backup or copy of datafile 1 found to restoreWhat's happening ? What's wrong ? Why the auxiliary instance named pknF doesn't start ?
[root@vsi08devpom ~]# ps -ef|grep smon oracle 22255 1 0 12:22 ? 00:00:00 ora_smon_pknF root 22307 19983 0 12:22 pts/2 00:00:00 grep smon oracle 29331 1 0 Feb06 ? 00:00:26 ora_smon_CDB001 [root@vsi08devpom ~]# ps -ef|grep smon root 22309 19983 0 12:22 pts/2 00:00:00 grep smon oracle 29331 1 0 Feb06 ? 00:00:26 ora_smon_CDB001
The problem is that to use the recover table command you need a full backup of the container database: having only a backup of the pluggable database is not sufficient because it doesn't have enough information to build an auxiliary instance (just think about at the "shared" UNDO tablespace not backed up for the pluggable database).
It's not mentioned (at least I was not able to find this information) into the Oracle documentation that you need a valid backup of the container database if you want to recover a table "lost" in a schema of a pluggable database.
I'm going to suggest an integration to contributors of the "Backup and Recovery User's Guide" and "Backup and Recovery Reference" so it could be more clear that you cannot recover a table if you have only a backup of the pluggable database.
That's all.
10 comments:
Great job. Very interesting tests.
oakley sunglasses, prada handbags, oakley sunglasses, longchamp handbags, longchamp handbags, louboutin shoes, louis vuitton handbags, coach factory outlet, tiffany and co, coach purses, louis vuitton outlet, polo ralph lauren outlet, air max, prada outlet, longchamp outlet, oakley sunglasses cheap, ray ban sunglasses, louboutin outlet, michael kors outlet, michael kors outlet, tiffany and co, burberry outlet, christian louboutin shoes, coach outlet store online, jordan shoes, polo ralph lauren outlet, louboutin, kate spade handbags, michael kors outlet, coach outlet, air max, gucci outlet, michael kors outlet, ray ban sunglasses, chanel handbags, michael kors outlet, tory burch outlet, nike free, kate spade outlet, louis vuitton outlet, burberry outlet, louis vuitton outlet stores, louis vuitton, nike shoes, michael kors outlet
excelent article. i got 2 questipn
can we take backup of a pluggable db in clos mode and restore to different cdb
2. i like take close backup od=f pdb and ewstore in many time at different run of ugade for next 3-4 monts. this will be my golden BACKUP.
THNAKS
SYED
jordan shoes, christian louboutin, uggs outlet, michael kors outlet online, uggs on sale, louis vuitton outlet, louis vuitton outlet, louis vuitton, ray ban sunglasses, replica watches, christian louboutin uk, chanel handbags, michael kors outlet online, uggs outlet, longchamp outlet, nike air max, michael kors outlet, burberry handbags, tiffany and co, polo outlet, nike free, nike air max, ugg boots, oakley sunglasses, ray ban sunglasses, michael kors outlet online, oakley sunglasses, christian louboutin outlet, longchamp outlet, prada handbags, gucci handbags, prada outlet, oakley sunglasses wholesale, michael kors outlet, oakley sunglasses, kate spade outlet, christian louboutin shoes, louis vuitton outlet, tory burch outlet, ugg boots, michael kors outlet online, burberry outlet, cheap oakley sunglasses, louis vuitton, ray ban sunglasses, nike outlet, longchamp outlet
jordan shoes
louboutin shoes
birkin bag
air yeezy
cheap mlb jerseys
fila
golden goose
curry 4
yeezy boost 350
off white nike
I definitely love this site.
http://prokr2020.inube.com/
https://lamissibrahim11.wixsite.com/prokr2020
https://prokr2020.hatenablog.com/
https://www.prokr.net/ksa/jeddah-water-leaks-detection-isolate-companies/
شركة عزل فوم بجدة
شركة عزل فوم بخميس مشيط
شركة عزل فوم براس تنورة
شركة عزل فوم بمكة
شركة عوازل الرياض
شركة غسيل الخزانات بالرياض
شركة غسيل خزانات الرياض
شركة كشف تسربات المياه بالرياض
شركة كشف تسربات المياه بالرياض عماله فلبينيه
شركه تنظيف خزانات بالرياض
عزل الاسطح بالرياض
Nice Blog!
Facing error while using QuickBooks get instant solution with our QuickBooks experts.Dial +1-(855)533-6333 Quickbooks Enterprise Support Phone Number
Nice Post !
QuickBooks Error 80029c4a 1-855-662-2O4O. It is a common error but a complex one. The error is reported by many users. If it occurs to you.
Need instant help in QuickBooks? Don’t worry!! Just get in touch with our QuickBooks Support Phone Number California +1(844)233-3033, and get all your queries settled at once. Our technicians are skilled enough to handle any sort of user’s complexities.
https://tinyurl.com/yyavgpto
https://tinyurl.com/y68z6hd4
https://tinyurl.com/yyavgpto
https://tinyurl.com/y3arnu38
Post a Comment