Run:AI Creates First Fractional GPU That Effectively Creates Virtualized Logical GPUs

Run:AI | May 07, 2020

  • Run:AI's fractional GPU system effectively creates virtualized logical GPUs, with their own memory and computing space that containers can use and access as if they were self-contained processors.

  • Run:AI also solved the problem of memory isolation, so each virtual GPU can run securely without memory clashes.

  • The addition of fractional GPU sharing is a key component in Run:AI's mission to create a true virtualized AI infrastructure, combining.


Run:AI, a company virtualizing AI infrastructure, today released the first fractional GPU sharing system for deep learning workloads on Kubernetes. Especially suited for lightweight AI tasks at scale such as inference, the fractional GPU system transparently gives data science and AI engineering teams the ability to run multiple workloads simultaneously on a single GPU, enabling companies to run more workloads such as computer vision, voice recognition and natural language processing on the same hardware, lowering costs.

Today's de facto standard for deep learning workloads is to run them in containers orchestrated by Kubernetes. However, Kubernetes is only able to allocate whole physical GPUs to containers, lacking the isolation and virtualization capabilities needed to allow GPU resources to be shared without memory overflows or processing clashes.

Read More: IT Management Solution Announces Updated Range of Virtualization Services for Businesses

Run:AI's fractional GPU system effectively creates virtualized logical GPUs, with their own memory and computing space that containers can use and access as if they were self-contained processors. This enables several deep learning workloads to run in containers side-by-side on the same GPU without interfering with each other. The solution is transparent, simple and portable; it requires no changes to the containers themselves.

 

To create the fractional GPUs, Run:AI had to modify how Kubernetes handled them. "In Kubernetes, a GPU is handled as an integer. You either have one or you don't. We had to turn GPUs into floats, allowing for fractions of GPUs to be assigned to containers. Run:AI also solved the problem of memory isolation, so each virtual GPU can run securely without memory clashes,

Dr. Ronen Dar, co-founder and CTO of Run:AI.



A typical use-case could see 2-4 jobs running on the same GPU, meaning companies could do four times the work with the same hardware. For some lightweight workloads, such as inference, more than 8 jobs running in containers can comfortably share the same physical chip.

The addition of fractional GPU sharing is a key component in Run:AI's mission to create a true virtualized AI infrastructure, combining with Run:AI's existing technology that elastically stretches workloads over multiple GPUs and enables resource pooling and sharing.

 

Some tasks, such as inference tasks, often don't need a whole GPU, but all those unused processor cycles and RAM go to waste because containers don't know how to take only part of a resource. Run:AI's fractional GPU system lets companies unleash the full capacity of their hardware so they can scale up their deep learning more quickly and efficiently,

Run:AI co-founder and CEO Omri Geller.



Read More: Synamedia Advances Video Network Portfolio with HPE ProLiant Servers for Network Virtualization

About Run:AI

Run:AI has built the world's first virtualization layer for AI workloads. By abstracting workloads from underlying infrastructure, Run:AI creates a shared pool of resources that can be dynamically provisioned, enabling full utilization of expensive GPU compute. IT teams retain control and gain real-time visibility – including seeing and provisioning run-time, queueing and GPU utilization – from a single web-based UI. This virtual pool of resources enables IT leaders to view and allocate compute resources across multiple sites - whether on premises or in the cloud. The Run:AI platform is built on top of Kubernetes, enabling simple integration with existing IT and data science workflows.

Spotlight

Artificial intelligence (AI) and machine learning (ML) have forever changed the landscape of the digital economy. These disciplines not only influence the way products, software and solutions are built, but artificial intelligence and machine learning are part of a game-changing technology phenomenon that gives enterprises a competitive edge in the market. It also lets them operate with agility to align faster with customer needs. However, despite the fact that enterprises know AI and ML can help extract value from their data, they still have lot of operational challenges to see the business benefits.

Spotlight

Artificial intelligence (AI) and machine learning (ML) have forever changed the landscape of the digital economy. These disciplines not only influence the way products, software and solutions are built, but artificial intelligence and machine learning are part of a game-changing technology phenomenon that gives enterprises a competitive edge in the market. It also lets them operate with agility to align faster with customer needs. However, despite the fact that enterprises know AI and ML can help extract value from their data, they still have lot of operational challenges to see the business benefits.

Related News

VPN

NordVPN enters the antivirus market by launching Threat Protection

NordVPN | February 11, 2022

NordVPN, the world's leading VPN service provider, took yet another step to solidify its position as a market leader in cybersecurity with the release of Threat Protection, which is integrated into the NordVPN app. This new feature offers comprehensive security against cyber threats by blocking trackers, phishing attempts, intrusive ads, malicious websites, and infected files. At Nord Security, we know that cybersecurity evolves rapidly, and cybersecurity tools must evolve too. And as part of a broader effort to shift into a more encompassing cybersecurity company, introducing Threat Protection brings us one step closer. With the introduction of Threat Protection, we will be able to offer more services and more comprehensive protection that doesn't depend merely on your VPN connection." Vykintas Maknickas, product strategist at NordVPN Threat Protection mitigates three primary kinds of threats. These include web trackers and malicious ads, harmful websites, and infected files. Web trackers and malicious ads. Fighting off trackers typically falls outside of the scope of a VPN, which is where Threat Protection comes into play. Threat Protection enhances the whole web privacy experience by blocking trackers entirely. You can monitor and manage the blocked trackers in real time. Harmful websites. Also use Threat Protection while browsing the web to avoid malicious websites. Threat Protection displays a warning prompt prior to the page loading. This allows you to exit in lieu of continuing to a harmful site and being phished or having your data collected. Infected files. Additionally, when you download a file from the internet, Threat Protection can be the first line of defense. It will scan for malware and, if no threats are detected, the file will be marked as safe without any interruptions. But if malware is found, the file will be deleted before it can do any harm. You will also be able to access a log of the scanned files available to view at any time. "Threat Protection takes what was once the task of antivirus software and merges it with NordVPN," says Vykintas Maknickas, product strategist at NordVPN. "Regardless of how long you have been using the internet, you can end up falling victim to some kind of precarious website or download. This is exactly why Threat Protection was introduced — to add a layer of security to make your online browsing safer, cleaner, and more private." To utilize Threat Protection, users need to download the latest OpenVPN version of the NordVPN app from the official website. Once enabled, the feature will protect users without them needing to connect to a VPN server. Threat Protection is currently available for all macOS users and is being gradually rolled out for Windows. NordVPN is part of Nord Security. Nord Security is moving with confidence towards becoming an all-around cybersecurity solution. Apart from NordVPN, the company is home to three other market-leading products: NordPass, a next-generation password manager; NordLocker, an encrypted cloud storage solution; and NordLayer, an advanced network access security solution.

Read More

CLOUD

Vultr Talon Cloud Virtualization Provider Now Available on Beta

Vultr | May 27, 2022

Vultr, a leading independent cloud infrastructure provider, announced today that Vultr Talon, powered by NVIDIA GPUs and NVIDIA AI Enterprise software, is currently in beta. Vultr Talon, a ground-breaking cloud-based platform, enables GPU sharing, allowing several workloads to execute effectively on a single NVIDIA GPU. Vultr is the first cloud service to provide GPU sharing by virtualizing NVIDIA A100 Tensor Core GPUs. Vultr is launching a set of virtual machine (VM) packages incorporating the NVIDIA A100 today, with prices beginning at $90 per month, or $0.13 per hour. Breakthrough Cloud Platform Enabling AI Workloads at a Fraction of the Cost Vultr Talon is powered by a cutting-edge NVIDIA GPU virtualization technology based on NVIDIA's vGPU software. Rather than connecting whole real GPUs to VMs, Vultr adds just a portion in the form of a virtual GPU (vGPU). The NVIDIA AI Enterprise software package, which incorporates NVIDIA vGPU software and is geared for remotely performing AI workloads and high-performance data analytics, powers these vGPUs. A vGPU appears to a customer's system just like an actual GPU. Each vGPU has its own devoted memory, which is a fraction of the memory on the underlying card. The vGPU has access to a fraction of the computing capacity of the actual GPU. NVIDIA's Multi-Instance GPU (MIG) technology is utilized for Vultr plans with at least 10GB of GPU RAM to give tenants with assured QoS, completely isolated GPU high-bandwidth memory cache, and dedicated compute cores. High-end GPUs can cost hundreds of dollars per month to rent. While this cost is often justified for the most demanding corporate workloads, for many enterprises and developers, the cost of even a single GPU might be prohibitive to getting started, experimenting, or running apps in development and testing environments. Even organizations with large IT budgets may wind up losing money by supplying more GPU capacity than required or just electing not to use GPUs at all. At Vultr, we pride ourselves on making high-performance cloud infrastructure affordable for everyone. With Vultr Talon, we have turned the GPU delivery model upside down. Because of our breakthrough cloud platform, AI developers and data scientists can provision exactly as much NVIDIA GPU processing as they actually need, at prices they can afford.” J.J. Kardwell, CEO of Vultr’s parent company,

Read More

VMware Unveiled Virtual Cloud Network to Create a Modern Network that Better Supports Current and Future Business Initiatives

VMware | September 30, 2020

Today at VMworld 2020, VMware, Inc. unveiled Virtual Cloud Network innovations that will help customers create a modern network that better supports current and future business initiatives. With advancements across the VMware networking and security portfolio, customers will be able to more effectively manage the rapid shift to remote work, deliver traditional and modern applications faster and more securely, and reduce the cost and complexity of connecting and protecting the distributed enterprise.

Read More