How Ai Is Pushing Kubernetes Storage Beyond Its Limits

As enterprises unreserved to deploy AI and data-intensive applications successful Kubernetes environments, modular Container Storage Interfaces (CSIs) aren’t capable to meet business requirements successful nan caller operating model.

A decade ago, erstwhile Kubernetes first burst onto nan scene, nan mostly of containerized workloads were stateless, redeeming nary discourse crossed different sessions. A emblematic Node.js aliases NGINX exertion would beryllium reinstantiated based connected disposable metadata, but it wouldn’t publication and constitute information to aliases from a persistence store.

The Rise of Stateful Applications successful Kubernetes

These patterns are comparatively easy to use to stateless web applications, and designing your microservices to beryllium arsenic stateless arsenic imaginable produces highly reliable, manageable systems.

However, arsenic Brendan Burns, Joe Beda, Kelsey Hightower and Lachlan Evenson wrote successful “Kubernetes: Up and Running,” “Nearly each strategy that has immoderate complexity has authorities successful nan strategy somewhere, from nan records successful a database to nan scale shards that service results for a web hunt engine. At immoderate point, you person to person information stored somewhere.”

Integrating this information pinch containers and instrumentality orchestration solutions is often nan astir analyzable facet of building a distributed system. The “Kubernetes: Up and Running” authors propose that this complexity stems from nan truth that “the move to containerized architectures is besides a move toward decoupled, immutable and declarative exertion development.”

Around 5 years ago, astatine Nutanix, we started to spot an uptick successful nan number of stateful applications utilizing containerized databases, specified arsenic Cassandra, Redis, PostgreSQL, MySQL and Kafka. With that displacement already underway, nan caller accelerated take of AI successful enterprises importantly accelerated nan process.

The acceleration is unsurprising. As Phil Winder, CEO and laminitis of Winder.AI, noted successful “Reinforcement Learning,” AI is “a kid of information science, which is an overarching technological section that investigates information generated by phenomena.”

In different words, your organization’s information is basal to nan occurrence of immoderate inaugural you mightiness prosecute utilizing AI.

While information is important because of AI, it is besides foundational to astir each exertion for things for illustration personalized recommendations for amended personification experience, personification behaviour analytics, security, observability (e.g., logs and metrics), Internet of Things (IoT) and edge.

A corollary comes from Gartner expert Julia Palmer, who predicts that, “By 2027, 80% of Kubernetes deployments will require precocious features for persistent containers storage, compared to 30% successful early 2023.”

Understanding CSI, nan Foundation of Kubernetes Storage

The Kubernetes CSI is nan modular system for dealing pinch persistence successful Kubernetes. This furniture consists of a group of APIs that applications tin usage to execute sounds and writes to nan underlying retention system.

Since CSI is simply a standard, each retention vendor has its ain implementation — Nutanix CSI, Dell CSI, Red Hat OpenShift CSI, Portworx CSI and truthful connected — and each CSI driver has vendor-specific attributes offered via nan built-in CSI hold mechanism.

Nutanix CSI provisions Nutanix Unified Storage (NUS) to containerized stateful applications. NUS is simply a software-defined information services level that consolidates file, entity and artifact retention into a single, high-performance, dense and cost-optimized platform, packaged according to a customer’s needs:

Limitations of CSI for Enterprise Workloads

CSI is good for providing persistent retention to a azygous cluster, but beyond that, it has immoderate limitations. Chiefly, it doesn’t supply a system for information protection aliases business continuity and disaster betterment (BCDR). This is peculiarly important successful heavy regulated industries specified arsenic financial services and healthcare. The request for BCDR successful regulated industries is not new, but it is becoming much pertinent alongside nan expanding number of applications moving wrong Kubernetes clusters.

Regulations besides dictate wherever information must reside. In regions for illustration EMEA, policies whitethorn instruction that each information copies stay wrong nationalist boundaries, adding a furniture of geospecific compliance to an already analyzable method challenge.

For immoderate application, persistent information needs to reside arsenic adjacent arsenic imaginable to wherever nan exertion is running, necessitating information replication for BCDR and related usage cases specified arsenic workload rebalancing and precocious availability. This is peculiarly important successful heterogeneous deployment models, for example, unreality bursting from connected premises to a nationalist unreality to grip transient spikes successful demand, specified arsenic Black Friday, assemblage admissions deadlines, online summons income aliases media streaming surges. Cloud bursting requires rapid, accordant replication of nan exertion situation and associated information to and from nan cloud.

Synchronous vs. Asynchronous Data Replication

Data replication tin beryllium either synchronous aliases asynchronous, depending upon really constitute operations are managed:

• Synchronous information replication intends nan information is perpetually copied from nan main server and to each replica servers simultaneously.
• Asynchronous information replication intends that information is first copied to nan main server and past copied to replica servers arsenic per nan preconfigured protection argumentation dictating nan wave of information replication and long of information retention.

Although synchronous replication ensures nary information is lost, asynchronous replication requires substantially little bandwidth and is little expensive.

Filling nan Gaps

Nutanix Data Services for Kubernetes (NDK) tin capable successful nan gaps near by CSI, letting you manage, power and run nan disparate worlds of virtual machines (VMs) and containerized apps arsenic a azygous entity, from 1 unified platform.

NDK uses acquainted Kubernetes mechanisms to thief trim nan learning curve. It is shipped arsenic a Helm chart, and users interact pinch it from nan bid statement utilizing kubectl. Data services are Kubernetes-distribution agnostic. While we would for illustration that customers usage our Kubernetes distro, nan information services will activity pinch alternatives, specified arsenic Red Hat OpenShift aliases Amazon EKS Anywhere. NDK supports some synchronous and asynchronous information replication.

In NDK, asynchronous replication tin beryllium performed astatine a maximum wave of erstwhile per hour. The argumentation is group astatine an exertion level alternatively than a cluster level, truthful different applications wrong a azygous cluster tin tally different information replication strategies.

Asynchronous replication is utilized successful BCDR. In a emblematic example, you mightiness person 2 information centers successful different countries — say, a superior successful Spain and a backup successful Germany — truthful that you tin move from 1 to nan different successful nan arena of a awesome disaster.

Alongside BCDR, Nutanix besides supports precocious readiness utilizing synchronous replication.

Synchronous replication guarantees zero information nonaccomplishment successful nan arena of a failure, but it requires some information centers to beryllium located successful adjacent beingness proximity. This intends it is unsuitable for protection against earthy disasters specified arsenic earthquakes aliases hurricanes, but depending connected your business, it tin beryllium a valuable approach.

One of our customers, for example, runs cruise ships and has 2 abstracted information rooms. They are successful abstracted locations but physically adjacent to each other, connected via a high-bandwidth web pinch a latency of nether 10ms. The use is that if 1 information room fails, possibly owed to a powerfulness outage aliases flooding, nan vessel tin move to nan different 1 and proceed to operate.

Beyond CSI: Why AI Demands More From Kubernetes Storage

The convergence of VMs and containers into a unified level is simply a applicable necessity for enterprises navigating nan complexities of distributed, data-intensive applications. As stateful applications proceed to proliferate successful Kubernetes environments — a inclination accelerated by AI take — nan request for enterprise-grade information services becomes critical.

While CSI provides nan instauration for persistent storage, solutions for illustration NDK are basal for organizations that require nan information protection, compliance and operational elasticity demanded by endeavor containerized workloads. NDK is offered arsenic portion of nan Nutanix Kubernetes Platform (NKP) solution, a complete full-stack level that combines infrastructure, Kubernetes orchestration, storage, information services and exertion life rhythm guidance successful a azygous platform.

Group Created pinch Sketch.