Using ZFS Storage Appliance with Exadata: Match Made in Heaven

    By: Umair Mansoob on Apr 20, 2018

    By Umair Mansoob | Editor Jim Czuprynski

    Many Oracle shops that use Exadata Database Machines (DBMs ) face a common dilemma when it comes to additional storage options for their Exadata environments, especially for backup and recovery of their largest databases. Oracle DBAs need a cost-effective storage option that can be used not only for backups, but also aid in reducing database administration work. While local Exadata storage can certainly be used to host backup storage, if you have multiple Exadata DBMs hosting databases in the terabyte range, it can be an expensive solution and a waste of valuable on-platform storage.

    So for Oracle DBAs who are looking to fully optimize Oracle Exadata local storage by complementing it with an enterprise level multipurpose storage, the Oracle ZFS Storage Appliance is often an excellent choice. The Oracle ZFS Storage Appliance provide cost effective, extreme performance and high-bandwidth storage option to complement your Exadata storage. The Oracle ZFS Storage Appliance is a multiple purpose storage solution which can be used for more than just database backups while providing a high level of data protection for your Exadata DBMs.

    This article will deep dive into the use cases mentioned previously, including configuration details and best practices to help protect mission-critical data that resides on your Exadata DBMs. There are many use cases of using Oracle ZFS Storage Appliance with Exadata but this article focuses on achieving a fast and reliable backup and restore solution only.


    Umair One.png


    Oracle ZFS is an easy-to-deploy unified storage system which is uniquely suited for Exadata DBMs because of following characteristics:


    • It provides extreme network bandwidth – either 10Gbe or InfiniBand connections - with built-in network redundancy.
    • It’s extremely easy to manage using native web management interface and provides an integration option with Oracle Enterprise Manager.  
    • It supports a full range to compression options and is tightly integrated with Oracle databases to provide Hybrid Columnar Compression (HCC) which is only available on Oracle storage.
    • Its tight integration with Oracle Database helps it achieve extreme backup rates of up to 42 TB/hr and restore rates of up 55 TB/hr.
    • It supports highly redundant and scalable InfiniBand architecture which can be seamlessly integrated with Oracle Exadata Machine to provide cost effective storage option.
    • It’s integrated with Oracle Recovery Manager (RMAN) to provide up to 2000 concurrent threads evenly distributed across many channels spread across multiple controllers.
    • Finally, Oracle ZFS uses Oracle Direct NFS to reduce CPU and memory overhead which bypasses the operating system and cache data to to userspace only without maintaining a second copy in kernel space.

    The network established between your Exadata DBMs and the ZFS Storage Appliance is one of the most important components for achieving extreme backup performance, so it’s important to carefully map out the many network components in this setup, including distance, number of cables and multipathing.  There are two major methods through which a ZFS Storage Appliance can be connected to an Exadata DBM:  InfiniBand and 10Gbe. It is strongly recommended to use InfiniBand for Exadata connectivity because it provides better bandwidth, low latency and reduced CPU utilization. However, there are a few cases when it makes sense to use 10Gbe connectivity; for example, if you want to connect multiple Exadata Machines to a single ZFS Storage Appliance and they reside at extreme distances from each other, it’s possible that you might run into distance limitation (approximately 50 feet) of the InfiniBand network. Additionally, if you are planning to use a ZFS Storage Appliance as general-purpose storage in addition to a purely backup solution, 10Gbe connectivity would be a better option because you will be able to use ZFS storage with  non-Exadata hosts using NFS mounts as long as they are on same 10Gbe network. 



    Umair Two.png


    If you have chosen InfiniBand as your preferred network for Exadata connectivity with ZFS Appliance, the next step will be to come up with the best network configuration to achieve the highest availability and performance. It is recommended that each ZFS Storage Appliance controller is configured with two dual-port InfiniBand Cards providing four ports per controller for maximum availability and throughput. If you are planning to connect two Exadata DBMs to the ZFS Storage Appliance, you will want to use four dual port InfiniBand PCIe cards providing 8 ports per controller. The Oracle ZFS Storage Appliance will be connected to each Exadata DBM using four cables between each head of the ZFS Storage Appliance and the Exadata DBM’s InfiniBand leaf switches. It is also a best practice to group Exadata DBMs and ZFS Storage Appliances based on their functionality – for example, production vs. test. This way you will be able to have a separate maintenance window for each group and will be able to perform upgrades like patching without impacting other group of applications and databases. The same goes for IB subnet: You have the option to use just one IB subnet for both Exadata DBMs, or multiple subnets so each Exadata machine has its own IB subnet. If you choose to have a separate subnet for each Exadata DBM, you will be able isolate maintenance work performed on IB switches.

    Exadata DBM Backup Use Cases with ZFS Storage Appliance 
    Leveraging ZFS Storage Appliance for database protection is probably the most widely-used case for using ZFS appliance with Exadata DBM. It’s important to choose a backup strategy for your Oracle databases that meets your recovery time objectives (RTO), recovery point objectives (RPO) and version retention objectives (VRO). Additionally, backups should complete within a reasonable amount of time, without effecting critical applications and users.

    Traditional RMAN Backup Sets

    RMAN backup sets are logical entities create by RMAN backup which can be both encrypted and compressed at the same time. A traditional RMAN backup strategy involves performing full backups or any combination of level 0, level 1 cumulative incremental, and differential backups so that it’s possible to restore and recover the database in the event of physical or logical failures. The traditional Exadata DBM backup strategy is to perform a full online database backup on a weekly or daily basis and have at least one copy of a full database backup including control files and sub-sequential archived logs stored on the Oracle ZFS Appliance.  This full backup and archive logs can be used to restore and recover the complete database up to the point of failure in case of a recovery. Additionally, if you have properly sized your redo logs for a maximum of three switches per hour, your RPO should never be more than 20 minutes. Recommended version retention objectives (VRO) should be at least two full backups retained on the ZFS Storage Appliance at all times, with the older backups scheduled to be deleted automatically. Additionally, it’s good idea to perform full database backups for small databases to achieve better RTO.

    As per Oracle MMA best practices, a common implementation is a tiered approach that combines incremental level 0 and level 1 backups. Level 0 incremental backups are often taken on a weekly basis with level 1 differential or cumulative incremental backups performed daily.  It is also important to enable RMAN block change tracking because it can drastically improve the performance of incremental backups.

    Incrementally Updated Backup Strategy

    An RMAN image copy backup is a block by block copy of the target database that consists of datafiles, archive logs and control files. A block-by-block copy has a limitation that it cannot be compressed, so storage requirements should be taken into consideration before opting for RMAN image copy backups. If your target database is sized in the terabyte range, however, it will take significant storage space to hold image copy backups. Fortunately, if you are using a ZFS Storage Appliance to store image copy backups, you can use Oracle ZFS Storage Appliance native compression to save storage space. Oracle ZFS Storage Appliance supports many different types of compression for different types of workloads, but it is recommended to use LZ4 for image copy backups.

    An RMAN image backup strategy involves creating a level 0 image copy backup of the target database using Oracle ZFS storage, and then performing differential or incremental backups on daily bases that can be applied to an image copy backup to roll it forward in time. Basically, you are maintaining an up to date image copy of the target database using full and incremental backups. This gives you an opportunity to quickly restore the target database in case of a disaster.

    With image copy backups you have two options to restore the target database: You can either restore all the database files to their original location; or you can use the SWITCH TO COPY option and run your database from the ZFS Storage Appliance. In the second later scenario, you will be able to restore the database in the shortest period of time, but you will lose your image copy backups; further, if your application performance depends on Exadata DBM native features like Smart Scan and storage indexes, you might not be able to achieve comparable performance.

    Opening Image Copy Backup Sets 

    If your backup strategy is to use image copy with incremental updates, then you have the luxury to open an image copy backup of a target Exadata database in a matter of minutes regardless of database size. Since Oracle ZFS Storage Appliance fully supports Hybrid Columnar Compression (HCC), it will also allow you to restore, recover and open databases that leverage HCC.  Additionally, Oracle ZFS Storage Appliance can create read-only snapshots and read-write clones of shares and projects holding image copy backups. ZFS Storage Appliance clones can be opened in read-write mode to support alternate processing needs, such as development or reporting systems similar to Oracle Active Data Guard.

    General Guidelines for Using ZFS Storage Appliance for Exadata DBM Backups 

    • It is recommended to utilize ZFS storage compression (LZ4) to reduce the amount of space required to store backups of the Oracle database and also disable RMAN level compression to increase backup throughput and reduce CPU overhead on the Exadata DBM.
    • It is recommended to utilize all Exadata DBM nodes to maximize backup throughput for both traditional and Image copy backups with using below set of recommended channels.


    Umair Three.png

    Source: Oracle


    • It is recommended to set the following ZFS project/share attributes to achieve optimal performance for both traditional and image copy backups:


    Best Practices for Traditional RMAN Backup Strategy

    Record Size 1M
    Sync Write Throughput
    Read Cache Do not use Cache devices
    Compression LZ4

    Best Practices for Incrementally Updated Backup Strategy

    Record Size 32K
    Sync Write Latency
    Read Cache Default
    Compression LZ4



    • It is recommended to create dedicated database services for backups across all Exadata nodes to achieve optional performance by parallelizing workload to all nodes and maximizing availability in case of instance or node failure.
    • It is recommended to use Oracle Direct NFS (dNFS) for all database and RMAN workloads when using Oracle ZFS Storage appliance with Exadata DBMs. It reduces CPU utilization by bypassing the operating system and boosts[JC1]  parallel I/O throughput by opening an individual connection for each database process.   
    • It is recommended to set RMAN parameter SECTION SIZE to 100G and FILESPERSET to 1 to achieve optimal performance and throughput. 

    Protecting ZFS Storage Appliance Data

    Even though ZFS Storage Appliance dual controllers provide additional protection against hardware failures, it will not be protected from the complete loss of its site. This why it’s recommended to protect data stored on a ZFS Storage Appliance using one of the following methods:

    Secondary ZFS appliance

    Based on the criticality of data stored on the ZFS Storage Appliance, you may want to consider protecting that data using a second ZFS Storage appliance for disaster recovery. The DR ZFS storage appliance can also be used for backup or general-purpose storage for local Exadata Database Machines by partitioning the ZFS Storage Appliance using storage pools, projects and shares. The ZFS Storage Appliance replication feature can be used to replicate a data set from the primary to the DR Oracle ZFS storage appliance; it can also be used to set up full-scale DR or partial DR to replicate only a subset of data using project/shares level replication. Note that Oracle ZFS Storage Appliance Replication is asynchronous in nature which means that WAN latency is not impacted by acknowledgment of client requests; therefore, it can sustain terabytes of replication between two ZFS appliances over the WAN network.

    Backup to TAPE

    There are many methods available in the market to back up NAS storage devices like the ZFS Storage Appliance including NDMP and non-NDMP methods. NDMP is an open standard protocol for network-based backup of Network-Attached Storage (NAS), and NDMP provides an efficient, non-intrusive mechanism for backing up and restoring shares and LUNs on the Oracle ZFS Storage Appliance. NDMP methods are therefore preferred because of efficient network and server resource utilization. NDMP provides many benefits including fiber channel connectivity for extreme performance and backup files with all of attributes like permissions and metadata.

    The Oracle ZFS Storage Appliance can greatly reduce the time needed to back up Oracle databases running on Exadata DBMs while also reducing the Recovery Time Objective (RTO) of the resident Oracle databases for critical applications.  It can also provide a simpler way to refresh non-production Oracle databases from production databases with the ability to restore and open database image copy backups using clone snapshots.

    1. Deploying Oracle Maximum Availability Architecture with Exadata Database Machine :
    2. Protecting Oracle Exadata with the Sun ZFS Storage Appliance: Configuration Best Practices :




    Umair Mansoob is an Oracle Certified professional, working with Oracle Technologies since 1999. He is an Oracle Certified Database Professional from Oracle version 7.3 to 12c and Oracle Certified Implementation specialist for Exadata Machine, Business Intelligence, and Data Warehousing. He has served as Oracle Consultant for many large financial clients and specialized in Exadata Implementations, data warehousing, business intelligence, cloud architecture and cloud migration. He is a regular contributor to Independent Oracle User Group (IOUG) and presented at many Oracle conferences including Oracle OpenWorld. He is currently working as a Senior Principal Consultant for META7, where he is primarily focused on Oracle Engineered Systems and Cloud implementations.

    Released: April 20, 2018, 9:05 am | Updated: April 3, 2019, 1:03 pm
    Keywords: Feature | SELECT Journal | Exadata | SELECT | SELECT Journal

    Copyright © 2019 Communication Center. All Rights Reserved
    All material, files, logos and trademarks within this site are properties of their respective organizations.
    Terms of Service - Privacy Policy - Contact

    Independent Oracle Users Group
    330 N. Wabash Ave., Suite 2000, Chicago, IL 60611
    phone: 312-245-1579 | email:

    IOUG Logo

    Copyright © 1993-2019 by the Independent Oracle Users Group
    Terms of Use | Privacy Policy