Flash Disk Replacement due to poor performance in Exadata X6-2 Environment

We have Exadata X6-2 Environment, where one of our Flashdisk was showing,

To identify a poor performance flash disk, use the following command:

This flash disk is in poor performance status.

Recommended Action  

The flash disk has entered poor performance status. A white cell locator LED has been lit to help locate the affected cell. Please replace the flash disk.

If the flash disk is used for flash cache, then flash cache will be disabled on this disk thus reducing the effective flash cache size. If the flash disk is used for flash log, then flash log will be disabled on this disk thus reducing the effective flash log size. If the flash disk is used for grid disks, then Oracle ASM rebalance will automatically restore the data redundancy.

Sun Oracle Exadata Storage Server is equipped with four PCIe cards. Each card has four flash disks (FDOMs) for a total of 16 flash disks. The 4 PCIe cards are present on PCI slot numbers 1, 2, 4, and 5. The PCIe cards are not hot-pluggable such that Exadata Cell must be powered down before replacing the flash disks or cards.

Hence DataCenter Team replaced a flash disk in co-ordination with us (DBA) because the flash disk was in poor performance status.

1. Shut down the cell.

The following procedure describes how to power down Exadata Cell.Run the following command to check if there are offline disks on other cells that are mirrored with disks on this cell:

CellCLI > LIST GRIDDISK ATTRIBUTES name WHERE asmdeactivationoutcome != 'Yes'

If any grid disks are returned, then it is not safe to take the storage server offline because proper Oracle ASM disk group redundancy will not be intact. Taking the storage server offline when one or more grid disks are in this state will cause Oracle ASM to dismount the affected disk group, causing the databases to shut down abruptly.

Inactivate all the grid disks when Oracle Exadata Storage Server is safe to take offline using the following command:

CellCLI> ALTER GRIDDISK ALL INACTIVE

The preceding command will complete once all disks are inactive and offline. Depending on the storage server activity, it may take several minutes for this command to complete.

Verify all grid disks areINACTIVEto allow safe storage server shut down by running the following command.

CellCLI> LIST GRIDDISK

If all grid disks areINACTIVE, then the storage server can be shutdown without affecting database availability.

Stop the cell services using the following command:

CellCLI> ALTER CELL SHUTDOWN SERVICES ALL

Shut down the cell.

2. Replace the failed flash disk based on the PCI number and FDOM number.

3. Power up the cell. The cell services will be started automatically.

4.Bring all grid disks are online using the following command:

CellCLI> ALTER GRIDDISK ALL ACTIVE

5. Verify that all grid disks have been successfully put online using the following command:

CellCLI> LIST GRIDDISK ATTRIBUTES name, asmmodestatus

        Wait until asmmodestatus from SYNCING to ONLINE for all grid disks.
        The following is an example of the output:

CellCLI> list physicaldisk where disktype=flashdisk
         FLASH_1_1       S2T7NA0J304430  normal
         FLASH_2_1       S2T7NAAH409309  normal
         FLASH_4_1       S2T7NA0J304420  normal
         FLASH_5_1       S2T7NA0JB00348  normal
CellCLI> list griddisk attributes name, asmmodestatus
         DATAC1_CD_00_ABC   ONLINE
         DATAC1_CD_01_ABC   ONLINE
         DATAC1_CD_02_ABC   ONLINE
         DATAC1_CD_03_ABC   ONLINE
         DATAC1_CD_04_ABC   ONLINE
         DATAC1_CD_05_ABC   ONLINE
         DATAC1_CD_06_ABC   ONLINE
         DATAC1_CD_07_ABC   ONLINE
         DATAC1_CD_08_ABC   ONLINE
         DATAC1_CD_09_ABC   ONLINE
         DATAC1_CD_10_ABC   ONLINE
         DATAC1_CD_11_ABC   ONLINE
         DBFS_DG_CD_02_ABC  ONLINE
         DBFS_DG_CD_03_ABC  ONLINE
         DBFS_DG_CD_04_ABC  ONLINE
         DBFS_DG_CD_05_ABC  ONLINE
         DBFS_DG_CD_06_ABC  ONLINE
         DBFS_DG_CD_07_ABC  ONLINE
         DBFS_DG_CD_08_ABC  ONLINE
         DBFS_DG_CD_09_ABC  ONLINE
         DBFS_DG_CD_10_ABC  ONLINE
         DBFS_DG_CD_11_ABC  ONLINE
         RECOC1_CD_00_ABC   ONLINE
         RECOC1_CD_01_ABC   ONLINE
         RECOC1_CD_02_ABC   ONLINE
         RECOC1_CD_03_ABC   ONLINE
         RECOC1_CD_04_ABC   ONLINE
         RECOC1_CD_05_ABC   ONLINE
         RECOC1_CD_06_ABC   ONLINE
         RECOC1_CD_07_ABC   ONLINE
         RECOC1_CD_08_ABC   ONLINE
         RECOC1_CD_09_ABC   ONLINE
         RECOC1_CD_10_ABC   ONLINE
         RECOC1_CD_11_ABC   ONLINE

Oracle ASM synchronization is only complete when all grid disks show attribute asmmodestatus=ONLINE. Before taking another storage server offline, Oracle ASM synchronization must complete on the restarted Oracle Exadata Storage Server. If synchronization is not complete, then the check performed on another storage server will fail.

The new flash disk will be automatically used by the system. If the flash disk is used for flash cache, then the effective cache size will increase. If the flash disk is used for grid disks, then the grid disks will be recreated on the new flash disk. If those gird disks were part of an Oracle ASM disk group, then they will be added back to the disk group and the data will be rebalanced on them based on the disk group redundancy and asm_power_limit parameter.

Oracle ASM rebalance occurs when dropping or adding a disk. To check the status of the rebalance, do the following:

• • The rebalance operation may have been successfully run. Check the Oracle ASM alert logs to confirm

• • The rebalance operation may be currently running. Check the GV$ASM_OPERATION view to determine if the rebalance operation is still running.

• • The rebalance operation may have failed. Check the GV$ASM_OPERATION.ERROR view to determine if the rebalance operation failed.

• • Rebalance operations from multiple disk groups can be done on different Oracle ASM instances in the same cluster if the physical disk being replaced contains ASM disks from multiple disk groups. One Oracle ASM instance can run one rebalance operation at a time. If all Oracle ASM instances are busy, then rebalance operations will be queued.

Doc ID Referred :

HALRT-02011: Flash disk poor performance status (Doc ID 1206015.1)

Steps to shut down or reboot an Exadata storage cell without affecting ASM (Doc ID 1188080.1)

Recover loss of the SYSTEM tablespace on the original location

In particular this example will restore the lost tablespace to its original location.
When you lose the SYSTEM tablespace the instance could crash or hang: anyway the instance will alert you as soon as possible.

It’s not rare the case when you cannot even shutdown the instance and proceed with a kill or a shutdown abort command.

If you have a good backup you can, of course, restore it, but the database could not be open until the recovery process finishes.
No problem for your committed transactions because your production database is always running in ARCHIVELOG mode and they will be available as soon as the database open.

Let’s begin simulating the loss of SYSTEM tablespace.
In my case the instance was not running as you can see when I deleted the file.

Here are my datafiles.

and removed system tablespace

Let's try to shut down the instance

I’m not able to login and a clear error message is showed on screen. Let’s use then RMAN…

RMAN shows the same error message.

The instance tried to start, shared memory were successfully attached by the Oracle processes, but the instance doesn’t work as expected: the SYSTEM tablespace is missing for RMAN too… I need to kill the instance.

Lets mount database using RMAN

You can see system tablespace size is '0'

Now we can issue restore command to for SYSTEM tablespace

The previous restore command try to restore the datafile of SYSTEM tablespace to its original location. Then it’s time to issue the recover command

Now OPEN the database

Now the database is available again to all the users and the SYSTEM tablespace is fully recovered.

How to recover from a loss of a non-system tablespace

The following scenario will describe how to proceed when you lose a non-system tablespace, I mean not the SYSTEM and UNDO tablespace. In particular this example will restore the lost tablespace to its original location. When you lose a non-system tablespace you cannot access and query only objects that were created on their datafiles;
meanwhile users can continue to query and use all the others objects in the database and you can restore it while the database is open.
Moreover because your database is running in ARCHIVELOG mode any committed transactions don't need to be inserted again.

Let's simulate a loss of the EXAMPLE tablespace, in my case formed by only one datafile: 

[oracle@alpha datafile]$ ll /u01/app/oracle/oradata/TEST/datafile/example.dbf
-rw-r----- 1 oracle oinstall 10493952 Mar 13 13:46 /u01/app/oracle/oradata/TEST/datafile/example.dbf
[oracle@alpha datafile]$ mv /u01/app/oracle/oradata/TEST/datafile/example.dbf /u01/app/oracle/oradata/TEST/datafile/example.dbf.back

The database is still open and I query for the very first time an object located on the EXAMPLE tablespace. I receive an error stating the instance was not able to open the example01.dbf (data)file



As you can see the objects located on EXAMPLE tablespace are no more available. So it's time to recover our tablespace: I'd like to remind you that all the following steps are executed while the database is OPEN as you can see even from the screen log of rman console (connected to target database: TEST (DBID=2300947395)
) 



Put the tablespace offline



Restore tablespace

RMAN> restore tablespace example;
Starting restore at 13-MAR-19
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=21 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00005 to /u01/app/oracle/oradata/TEST/datafile/example.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/fast_recovery_area/TEST/backupset/2019_03_13/o1_mf_nnndf_TAG20190313T134524_g8lmy49o_.bkp
channel ORA_DISK_1: piece handle=/u01/app/oracle/fast_recovery_area/TEST/backupset/2019_03_13/o1_mf_nnndf_TAG20190313T134524_g8lmy49o_.bkp tag=TAG20190313T134524
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
Finished restore at 13-MAR-19
Recover tablespace



Put tablespace online.



Tablespace got recovered and we can see that in original location



Content in alert log

How to recover a corrupted/damaged/lost NONSYSTEM datafile (Database in ARCHIVELOG mode)

The following scenario describes what happens when a nonsystem datfile is lost. Before proceeding be sure to have a complete backup of your database and be sure your database in ARCHIVELOG mode.

SQL> show parameter instance_name
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      test

SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     11
Next log sequence to archive   13
Current log sequence           13
SQL>

Connect to RMAN to check information about datafiles

[oracle@alpha datafile]$ rman target /
Recovery Manager: Release 12.1.0.2.0 - Production on Wed Mar 13 21:02:30 2019
Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.
connected to target database: TEST (DBID=2300947395)
RMAN> report schema;
using target database control file instead of recovery catalog
Report of database schema for database with db_unique_name TEST
List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    780      SYSTEM               YES     /u01/app/oracle/oradata/TEST/datafile/o1_mf_system_g8lmhryv_.dbf
3    610      SYSAUX               NO      /u01/app/oracle/oradata/TEST/datafile/o1_mf_sysaux_g8lf08dm_.dbf
4    60       UNDOTBS1             YES     /u01/app/oracle/oradata/TEST/datafile/o1_mf_undotbs1_g8lf34lb_.dbf
5    10       EXAMPLE              NO      /u01/app/oracle/oradata/TEST/datafile/example.dbf
6    5        USERS                NO      /u01/app/oracle/oradata/TEST/datafile/o1_mf_users_g8lf33c1_.dbf
List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    60       TEMP                 32767       /u01/app/oracle/oradata/TEST/datafile/o1_mf_temp_g8lf4w5d_.tmp
RMAN>

Query the dictionary tables and see in which datafiles some tables are located.

SQL> col owner format a15
SQL> col table_name format a40
SQL> /
OWNER           TABLE_NAME                               TABLESPACE_NAME
--------------- ---------------------------------------- ------------------------------
SCOTT           DEPT                                     USERS
SCOTT           EMP                                      USERS
SCOTT           BONUS                                    USERS
SCOTT           SALGRADE                                 USERS

SQL> select count(*) from scott.dept;
  COUNT(*)
----------
         4
SQL> select count(*) from scott.emp;
  COUNT(*)
----------
        14

What does it happens when I delete the datafile where USERS tablespace is based on.

[oracle@alpha datafile]$ ls -lrt
total 1523444
-rw-r----- 1 oracle oinstall   5251072 Mar 13 13:46 o1_mf_users_g8lf33c1_.dbf
-rw-r----- 1 oracle oinstall  10493952 Mar 13 13:46 example.dbf.back
-rw-r----- 1 oracle oinstall  10493952 Mar 13 15:31 example.dbf
-rw-r----- 1 oracle oinstall  62922752 Mar 13 19:31 o1_mf_temp_g8lf4w5d_.tmp
-rw-r----- 1 oracle oinstall  62922752 Mar 13 21:05 o1_mf_undotbs1_g8lf34lb_.dbf
-rw-r----- 1 oracle oinstall 817897472 Mar 13 21:05 o1_mf_system_g8lmhryv_.dbf
-rw-r----- 1 oracle oinstall 639639552 Mar 13 21:06 o1_mf_sysaux_g8lf08dm_.dbf
[oracle@alpha datafile]$ mv o1_mf_users_g8lf33c1_.dbf o1_mf_users_g8lf33c1_.dbf.corrupt

I'm still able to query the dictionary tables for example but...

SQL> select count(*) from dba_tables;
  COUNT(*)
----------

      2343

I obtain an error when I try to select some rows from the SCOTT.EMP table

select count(*) from scott.emp;

ORA-01116: error in opening database file 6
ORA-01110: data file 6: '/u01/app/oracle/oradata/TEST/datafile/o1_mf_users_g8lf33c1_.dbf'
ORA-27041: unable to open file
Linux-x86_64 Error: 2: No such file or directory

Additional information: 3

If you try to connect with RMAN the REPORT SCHEMA command is now unable to correctly size the USERS tablespace. RMAN says its size is 0.

RMAN> report schema;

using target database control file instead of recovery catalog
Report of database schema for database with db_unique_name TEST

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    780      SYSTEM               YES     /u01/app/oracle/oradata/TEST/datafile/o1_mf_system_g8lmhryv_.dbf
3    610      SYSAUX               NO      /u01/app/oracle/oradata/TEST/datafile/o1_mf_sysaux_g8lf08dm_.dbf
4    60       UNDOTBS1             YES     /u01/app/oracle/oradata/TEST/datafile/o1_mf_undotbs1_g8lf34lb_.dbf
5    10       EXAMPLE              NO      /u01/app/oracle/oradata/TEST/datafile/example.dbf
6    0        USERS                NO      /u01/app/oracle/oradata/TEST/datafile/o1_mf_users_g8lf33c1_.dbf

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    60       TEMP                 32767       /u01/app/oracle/oradata/TEST/datafile/o1_mf_temp_g8lf4w5d_.tmp

Because the USERS tablespace is not a system tablespace, we can recover it just putting it in offline mode.

RMAN> sql 'alter database datafile 6 offline';

sql statement: alter database datafile 6 offline

RMAN> restore datafile 6;

Starting restore at 13-MAR-19
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=20 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00006 to /u01/app/oracle/oradata/TEST/datafile/o1_mf_users_g8lf33c1_.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/fast_recovery_area/TEST/backupset/2019_03_13/o1_mf_nnndf_TAG20190313T134524_g8lmy49o_.bkp
channel ORA_DISK_1: piece handle=/u01/app/oracle/fast_recovery_area/TEST/backupset/2019_03_13/o1_mf_nnndf_TAG20190313T134524_g8lmy49o_.bkp tag=TAG20190313T134524
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
Finished restore at 13-MAR-19

RMAN> recover datafile 6;

Starting recover at 13-MAR-19
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 13-MAR-19

RMAN> sql 'alter database datafile 6 online';

sql statement: alter database datafile 6 online

You can see in alert log how the restore and recovery process proceeds

Wed Mar 13 21:19:51 2019
alter database datafile 6 offline
Completed: alter database datafile 6 offline
Wed Mar 13 21:20:06 2019
Full restore complete of datafile 6 /u01/app/oracle/oradata/TEST/datafile/o1_mf_users_g8mglp3n_.dbf.  Elapsed time: 0:00:00
  checkpoint is 2214185
  last deallocation scn is 3
alter database recover datafile list clear
Completed: alter database recover datafile list clear
alter database recover if needed
 datafile 4
Wed Mar 13 21:20:22 2019
Media Recovery Start
Wed Mar 13 21:20:22 2019
Serial Media Recovery started
Wed Mar 13 21:20:22 2019
Media Recovery failed with error 1124
ORA-283 signalled during: alter database recover if needed
 datafile 4
...
alter database recover datafile list clear
Completed: alter database recover datafile list clear
alter database recover if needed
 datafile 6
Wed Mar 13 21:20:33 2019
Media Recovery Start
Wed Mar 13 21:20:33 2019
Serial Media Recovery started
Wed Mar 13 21:20:33 2019
Recovery of Online Redo Log: Thread 1 Group 3 Seq 12 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/TEST/onlinelog/o1_mf_3_g8lf4pjj_.log
  Mem# 1: /u01/app/oracle/fast_recovery_area/TEST/onlinelog/o1_mf_3_g8lf4plq_.log
Wed Mar 13 21:20:33 2019
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/TEST/onlinelog/o1_mf_1_g8lf4oc2_.log
  Mem# 1: /u01/app/oracle/fast_recovery_area/TEST/onlinelog/o1_mf_1_g8lf4ofn_.log
Wed Mar 13 21:20:33 2019
Media Recovery Complete (test)
Completed: alter database recover if needed
 datafile 6
alter database datafile 6 online
Completed: alter database datafile 6 online
Now I am able to query SCOTT.EMP table

SQL> select count(*) from scott.emp;

  COUNT(*)
----------
        14

We can see recovered file

[oracle@alpha datafile]$ ls -lrt *user*.dbf
-rw-r----- 1 oracle oinstall 5251072 Mar 13 21:21 o1_mf_users_g8mglp3n_.dbf
Check the size of USERS tablesspace in RMAN

RMAN> report schema;

Report of database schema for database with db_unique_name TEST

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    780      SYSTEM               YES     /u01/app/oracle/oradata/TEST/datafile/o1_mf_system_g8lmhryv_.dbf
3    610      SYSAUX               NO      /u01/app/oracle/oradata/TEST/datafile/o1_mf_sysaux_g8lf08dm_.dbf
4    60       UNDOTBS1             YES     /u01/app/oracle/oradata/TEST/datafile/o1_mf_undotbs1_g8lf34lb_.dbf
5    10       EXAMPLE              NO      /u01/app/oracle/oradata/TEST/datafile/example.dbf
6    5        USERS                NO      /u01/app/oracle/oradata/TEST/datafile/o1_mf_users_g8mglp3n_.dbf

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    60       TEMP                 32767       /u01/app/oracle/oradata/TEST/datafile/o1_mf_temp_g8lf4w5d_.tmp

How to recover loss of SYSTEM tablespace on different location

This is another post on a scenario describing how to proceed when you lose the SYSTEM tablespace: as already said this tablespace always contains the data dictionary tables for the entire database.

In particular this example will restore the lost tablespace to another location, just as you have to restore it because a disk controller is no more working and you have to recreate it (recovering) to a different location.

When you lose the SYSTEM tablespace the instance could crash or hang: anyway the instance will alert you as soon as possible.

It's not rare the case when you cannot even shutdown the instance and proceed with a kill or a shutdown abort command.

Let's begin simulating the loss of the SYSTEM tablespace. In my case the instance was running, so I shutted it down 

[oracle@alpha datafile]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Thu Mar 14 16:38:15 2019
Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
[oracle@alpha datafile]$ ls -lrt
total 1555972
-rw-r----- 1 oracle oinstall  10493952 Mar 13 13:46 example.dbf.back
-rw-r----- 1 oracle oinstall  62922752 Mar 13 22:01 o1_mf_temp_g8lf4w5d_.tmp
-rw-r----- 1 oracle oinstall   5251072 Mar 14 16:38 o1_mf_users_g8mglp3n_.dbf
-rw-r----- 1 oracle oinstall  62922752 Mar 14 16:38 o1_mf_undotbs1_g8lf34lb_.dbf
-rw-r----- 1 oracle oinstall 828383232 Mar 14 16:38 o1_mf_system_g8lmhryv_.dbf
-rw-r----- 1 oracle oinstall 660611072 Mar 14 16:38 o1_mf_sysaux_g8lf08dm_.dbf
-rw-r----- 1 oracle oinstall  10493952 Mar 14 16:38 example.dbf

Remove SYSTEM datafile.

[oracle@alpha datafile]$ pwd
/u01/app/oracle/oradata/TEST/datafile
[oracle@alpha datafile]$ rm o1_mf_system_g8lmhryv_.dbf

Lets connect to RMAN and start database in MOUNT to restore/recover process

[oracle@alpha datafile]$ rman target /
Recovery Manager: Release 12.1.0.2.0 - Production on Thu Mar 14 16:41:31 2019
Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.
connected to target database (not started)

RMAN> startup mount
Oracle instance started
database mounted
Total System Global Area     419430400 bytes
Fixed Size                     2925120 bytes
Variable Size                335547840 bytes
Database Buffers              75497472 bytes
Redo Buffers                   5459968 bytes

Using the report schema command, you can see RMAN is not able to correctly know the size of SYSTEM datafile. 

RMAN> report schema;
using target database control file instead of recovery catalog
Report of database schema for database with db_unique_name TEST
List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    0        SYSTEM               ***     /u01/app/oracle/oradata/TEST/datafile/o1_mf_system_g8lmhryv_.dbf3    630      SYSAUX               ***     /u01/app/oracle/oradata/TEST/datafile/o1_mf_sysaux_g8lf08dm_.dbf
4    60       UNDOTBS1             ***     /u01/app/oracle/oradata/TEST/datafile/o1_mf_undotbs1_g8lf34lb_.dbf
5    10       EXAMPLE              ***     /u01/app/oracle/oradata/TEST/datafile/example.dbf
6    5        USERS                ***     /u01/app/oracle/oradata/TEST/datafile/o1_mf_users_g8mglp3n_.dbf
List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    60       TEMP                 32767       /u01/app/oracle/oradata/TEST/datafile/o1_mf_temp_g8lf4w5d_.tmp

Now restore the datafile to different location.

[oracle@alpha datafile]$ mkdir non_default_location

[oracle@alpha datafile]$ cd non_default_location/

[oracle@alpha non_default_location]$ pwd/u01/app/oracle/oradata/TEST/datafile/non_default_location

To switch a datafile to another location you have to user SET NEWNAME.
And its important to include SWITCH DATAFILE ALL command before recovery

What does it mean and why we have to execute that command ? From Oracle documentation it "specifies that all data files for which a SET NEWNAME FOR DATAFILE command has been issued in this job are switched to their new name": control file will be so updated with the new location of SYSTEM datafile. 

RMAN> run {
set newname for datafile 1 to '/u01/app/oracle/oradata/TEST/datafile/non_default_location/system.dbf';
restore tablespace system;
switch datafile all;
recover tablespace system;
alter database open;
}2> 3> 4> 5> 6> 7>

executing command: SET NEWNAME
Starting restore at 14-MAR-19
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=15 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /u01/app/oracle/oradata/TEST/datafile/non_default_location/system.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/fast_recovery_area/TEST/backupset/2019_03_13/o1_mf_nnndf_TAG20190313T134524_g8lmy49o_.bkp
channel ORA_DISK_1: piece handle=/u01/app/oracle/fast_recovery_area/TEST/backupset/2019_03_13/o1_mf_nnndf_TAG20190313T134524_g8lmy49o_.bkp tag=TAG20190313T134524
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:15
Finished restore at 14-MAR-19
datafile 1 switched to datafile copy
input datafile copy RECID=4 STAMP=1002906030 file name=/u01/app/oracle/oradata/TEST/datafile/non_default_location/system.dbf
Starting recover at 14-MAR-19
using channel ORA_DISK_1
starting media recovery
archived log for thread 1 with sequence 12 is already on disk as file /u01/app/oracle/fast_recovery_area/TEST/archivelog/2019_03_13/o1_mf_1_12_g8lmz859_.arc
archived log for thread 1 with sequence 13 is already on disk as file /u01/app/oracle/fast_recovery_area/TEST/archivelog/2019_03_13/o1_mf_1_13_g8mjyhs7_.arc
archived log for thread 1 with sequence 14 is already on disk as file /u01/app/oracle/fast_recovery_area/TEST/archivelog/2019_03_13/o1_mf_1_14_g8mjzwsm_.arc
archived log for thread 1 with sequence 15 is already on disk as file /u01/app/oracle/fast_recovery_area/TEST/archivelog/2019_03_13/o1_mf_1_15_g8mk1hr7_.arc
archived log for thread 1 with sequence 16 is already on disk as file /u01/app/oracle/fast_recovery_area/TEST/archivelog/2019_03_14/o1_mf_1_16_g8nqnvxr_.arc
archived log file name=/u01/app/oracle/fast_recovery_area/TEST/archivelog/2019_03_13/o1_mf_1_12_g8lmz859_.arc thread=1 sequence=12
archived log file name=/u01/app/oracle/fast_recovery_area/TEST/archivelog/2019_03_13/o1_mf_1_13_g8mjyhs7_.arc thread=1 sequence=13
archived log file name=/u01/app/oracle/fast_recovery_area/TEST/archivelog/2019_03_13/o1_mf_1_14_g8mjzwsm_.arc thread=1 sequence=14
media recovery complete, elapsed time: 00:00:08
Finished recover at 14-MAR-19
Statement processed

After the end of recovery process, you can report the schema info again. The size of SYSTEM tablespace is again well known and the report schema command shows also a new location of the datafile number 1. 

RMAN> report schema;
Report of database schema for database with db_unique_name TEST
List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    790      SYSTEM               YES     /u01/app/oracle/oradata/TEST/datafile/non_default_location/system.dbf
3    640      SYSAUX               NO      /u01/app/oracle/oradata/TEST/datafile/o1_mf_sysaux_g8lf08dm_.dbf
4    60       UNDOTBS1             YES     /u01/app/oracle/oradata/TEST/datafile/o1_mf_undotbs1_g8lf34lb_.dbf
5    10       EXAMPLE              NO      /u01/app/oracle/oradata/TEST/datafile/example.dbf
6    5        USERS                NO      /u01/app/oracle/oradata/TEST/datafile/o1_mf_users_g8mglp3n_.dbf
List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    60       TEMP                 32767       /u01/app/oracle/oradata/TEST/datafile/o1_mf_temp_g8lf4w5d_.tmp

We can see the system.dbf file in new location.

[oracle@alpha non_default_location]$ ls -rlt
total 809768
-rw-r----- 1 oracle oinstall 828383232 Mar 14 17:01 system.dbf