NetApp XCP 1.6 Delivers Open File Analytics and Infrastructure Improvements
Karthikeyan Nagalingam
In my previous blog, I discussed 2 ways to quickly and reliably move data using NetApp® XCP, our data migration software. This blog goes deeper into the features of NetApp® XCP 1.6 and their use:
- File analytics – graphical user interface
- Logging enhancements
File Analytics
Today, data plays a more crucial role than ever before. It helps organizations fulfil their customers’ requirements by creating the right tools and solutions to meet their demands, especially in the financial, energy, and service industries. File analytics analyze unstructured data and provide classified and high-quality data to support the business.
As organizations move toward digital transformation, good insights help them reap the value of their data and find the best solutions and tools to address their customers’ requirements.
NetApp XCP natively provides the ability to identify, scrutinize, and analyze unstructured data to improve insights—a key requirement for enterprise customers who want to use those insights for better planning, to operationalize high-value digital assets, and for data governance through reporting and assessment.
Customers who deal with sensitive data can use NetApp XCP to answer typical operational questions like the following:
- Where is my data?
- How much data and what types of files do we have?
- What data is actively used and how much is dormant?
File Analytics higher-level view and architecture
In XCP 1.6, we introduced a web-based graphical interface (GUI) for file analytics. The XCP File Analytics GUI helps to run file system scans by using XCP at the back end and visualizing statistics as graphs and views for any NAS (NFS, SMB) file system.Starting in 1.6, XCP can be run as a service with the help of simple deployment steps using Configure and Listen. The XCP Configure option guides you to install and configure Postgres and web server as well as collect credentials. The XCP Listen option runs as a service for REST API communications from the GUI.
Higher-level architecture[/caption]
XCP File Analytics has three major components: GUI, REST interface, and database (Postgres) to collect and store analytics information. The GUI was developed using a NetApp web framework and an angular framework, like any NetApp GUI products. The REST interface runs as a service on port 5030. The Postgres database stores the file analytics results. By logging in to Postgres, you can view the file analytics details used in the GUI.
When you run XCP operations from the GUI, it communicates to the REST interface, runs as a job, and provides the results in JSON format. The results are stored in Postgres for the UI to show them in graphical format.
XCP file analytics communication from the UI.[/caption]
XCP 1.6 supported and unsupported features
- XCP File Analytics Server (web-portal and Postgres) is currently supported only on Linux servers.
- Windows customers can run XCP as a service, but they need a Linux server to run the GUI and Postgres to store the scan results.
- Linux customers can run XCP as a service, and run Postgres and the GUI from a Linux server.
- REST APIs are not exposed to customers for general operations.
- SMB shows hot and cold data graphs and does not show access time. This feature is available only for NFS data.
- When using XCP only for file analytics purposes, an XCP catalog NFS volume or share is not required.
- In 1.6, it is not possible to scan all the shares at one time..
XCP File Analytics views
You can check the XCP File Analytics details from the dashboard and in File Server view.The dashboard offers a holistic view of XCP agents, job status, and file servers. The XCP agent is either a Linux server or a Windows server in which XCP is running as a service. In 1.6, a maximum of two XCP agents can be added for each XCP file analytics instance. Job status provides the status of the XCP scanning operation. File servers have either NFS or SMB exported directories; they can be added by using the Add File Server option in the UI.
Dashboard views[/caption] File Servers View provides details about shares such as a drop-down list of shares for analytics scan, a summary of shares, and available shares. The View File Share option offers two additional views: Stats View and File Distribution View.
The Stats View provides the following details:
- Hot and cold data by file count and size help the customer to understand the importance of their data and access frequency. This information guides the customer to move hot data to faster storage for frequent access and cold data to slower storage for less-frequent access. In XCP 1.6, this feature is available only in the GUI, not in the CLI.
- Space consumed by the user based on ownership helps the infrastructure team to proactively manage the required disk space for their users to support capacity-intensive workloads.
- File distribution by size helps the customer to identify workloads based on the file size and categorize them into different tiering solutions.
- An activity histogram by file accessed, modified, and create time helps security teams to analyze before and after a data breach or suspected incident.
The File Distribution View provides the following details:
- File details by extension and types of files help the customer to understand the characteristics of the file contents and their intended use. This information guides the artificial intelligence platform to identify the types of processor to process the different types of data. For example, GPUs are required to train the model on large amounts of unstructured and semistructured data, and CPUs are required for structured data for transaction processing.
File Servers view[/caption] For more information, watch the step-by-step video, XCP File Analytics.
File Analytics conclusion
In this competitive world, data is digitized and customer demands continue to grow. File analytics extract valuable insights from large amounts of unstructured data, providing increased opportunities for management teams to make right decisions in the market. Using NetApp XCP File Analytics, you can dive deep into your organization’s data in a visualized format for better understanding and higher-level insights into your enterprise.Logging Enhancements (Infrastructure)
Log data is crucial for managing, monitoring, and troubleshooting IT products and for drilling down on application issues. Log information provides accountability for operations in production and helps to measure the time to complete the task and to review the product functionality. It also helps the infrastructure team to resume their work after a long break. A logging infrastructure workflow feature is introduced in XCP 1.6.
XCP logging at a higher level
XCP log enhancement has three major features: log rotation, log level, and log name.- Log rotation helps to manage small log files instead of very large log files; save disk space; archive and compress old log files; remove very old log files; and control the log files for analysis. In XCP 1.6 log rotation, up to 10 log files are created, each 50MB in size. These values are hard coded by default, but they can be modified with the help of the NetApp Support team.
- Log level is the key feature that provides support for the product. The log level captures the required details. You need deeper information for troubleshooting, but when you collect more information, that consumes resources such as disk space and can affect the application’s performance. You can run XCP operations with different log levels based on purpose. XCP 1.6 includes five log levels of severity: CRITICAL, ERROR, WARN, INFO, and DEBUG, which can be chosen by –loglevel <SEVERITY> with the XCP command. The xcp.log file content is updated based on the severity level. The value set in the CLI takes precedence over the value set in JSON config. In XCP 1.6, you can modify the log level JSON config file with the help of the NetApp Support team.
- XCP 1.6 has a different log name for each type in file analytics:
- scp console: All the console logs coming from REST APIs, as well as GUI clicks that results in the REST calls in the back end, are stored in this file. Developers use this file to debug when the customer raises a support case with NetApp.
- scp rest: Debug logs for file analytics are stored in this file. By default, debug mode is enabled for XCP operations.
- xcp_stdout, xcp_stderr: Standard output and error for XCP. Recent CLI scan results are stored here and overwritten for every new scan result.
- xcp out: Combined results of standard output and error for recent and history scans results are stored in this file.
The log file locations for file analytics log files are in /opt/NetApp/xFiles/xcp/xcpfalogs for NFS and C:\NetApp\XCP\Logs \xcp_fa for SMB. All of the XCP non-file analytics operations results are stored in xcp.log file, which is stored in /opt/NetApp/xFiles/xcp/xcplogs for NFS and C:\NetApp\XCP\Logs for SMB.
When you run ‘xcp scan’ for the first time, the ‘xcplogs’ folder is created in /opt/NetApp/xFiles/xcp or C:\NetApp\XCP\Logs.
Logging infrastructure conclusion
The logs are crucial for applications and products for troubleshooting, accountability, timing, review, and transitioning. The XCP logging infrastructure guides you in managing your data movement and analytics.Let me know what you think about this blog in the comments field below! Or please send your feedback on NetApp XCP and NetApp XCP File Analytics to ng-xcp-feedback@netapp.com.
Karthikeyan Nagalingam
Karthikeyan Nagalingam is a Principal Technical Marketing Engineer at NetApp for NetApp XCP, Fpolicy, Filesystem Analytics and Antivirus. His previous roles in Emerging Technology Solutions involved in Pre-Sales and Post-Sales technical activities with fields, partners and customers. He holds an Master of Science in Software Systems from Birla Institute of Technology and Science.
