Red Hat's Pen Drive Brings Offline Cluster Analysis to Air-Gapped OpenShift

Why disconnected clusters need a dedicated tool

Red Hat says telecommunications providers, government agencies and financial institutions run OpenShift in environments that are intentionally air-gapped. Keeping those clusters healthy has always been the hard part, the company writes in a blog post by Tomas Dosek, Principal Product Manager for Red Hat Insights.

According to Red Hat, isolation makes disconnected or restricted Kubernetes clusters prone to configuration drift and a buildup of issues over time. That raises the risk of production incidents during Day-2 operations. Pen Drive is meant to give operators a way to check cluster state without a connection to the outside world.

What Pen Drive includes

Pen Drive is delivered as a single Podman image and provides a toolbox for cluster state analysis. Red Hat says it offers an offline version of the Red Hat Lightspeed advisor, enhanced with over 1,000 proactive recommendations. These cover debugging of cluster architecture, scalability, performance and day-to-day maintenance.

It also includes Cluster Compare capabilities, part of Red Hat OpenShift, for blueprint-based cluster drift analysis and partner recommendations. A lightweight direct state validation mode is meant to provide safe analytics in live environments.

Red Hat describes three main modes:

• In-cluster check: the least resource-intensive option, which runs a targeted set of commands for basic health analysis. It was co-developed with Nokia and originally designed for telecommunication clusters. It covers hardware and firmware inventory, hostgroup consistency and baseline cluster health.
• Kube-compare: compares actual cluster configurations against a pre-existing blueprint provided by Red Hat or a partner in the Red Hat Design System blueprint format.
• Full run: collects and analyses an optimised must-gather archive, inspecting the last 24 hours of cluster metrics, configurations, logs and other resources. Red Hat says this gives the most comprehensive look at cluster state and drift, with proactive recommendations for resolving problems.

How it works

The tool can run at any location that has connectivity with the cluster to be checked, using the Podman toolchain. Users download the image from the Red Hat Ecosystem Catalog and can load it onto a USB drive, laptop, jump host, Red Hat Advanced Cluster Manager Hub cluster, or the cluster itself.

Both interactive and non-interactive commands are supported across five workflows:

• full-run: collects lightweight must-gather data while fetching default must-gather pod logs from the cluster, then scans against Red Hat Lightspeed rules, in-cluster rules and kube-compare if a cluster reference is given.
• gather: collects comprehensive must-gather data for offsite or delayed analysis.
• scan: scans Red Hat Lightspeed rules against an existing must-gather archive.
• in-cluster-check: runs direct rule checks on cluster nodes without collecting must-gather data. Red Hat calls it the preferred choice for live traffic environments with minimal cluster impact.
• kube-compare: detects configuration drift by comparing the live cluster against a mounted reference directory containing a known baseline configuration.

Results come as a JSON file suitable for automation and parsing, for instance in workflow management tools, and as a standalone HTML file that highlights key findings and recommendations.

What it means for OpenShift operators

Red Hat says the image is only a few hundred megabytes and operates entirely offline, which it argues makes it suitable for the most restricted and isolated environments. For teams running air-gapped OpenShift, the tool brings Lightspeed-style recommendations and drift checks to clusters that cannot reach cloud-based services. Red Hat also notes that its own and partner field teams contribute to the recommendation set through a collaborative knowledge model.

Existing OpenShift subscribers can download the image today, according to Red Hat.