Amazon Redshift and the art of performance optimization in the cloud

| November 21, 2018

article image
People often ask me if developing for the cloud is any different from developing on-premises software. It really is. In this post, I show some of the reasons why that's true, using the Amazon Redshift team and the approach they have taken to improve the performance of their data warehousing service as an example. The Amazon Redshift team has delivered remarkable gains using a few simple engineering techniques. The biggest difference between developing for the cloud and developing on-premises software is that in the cloud, you have much better access to how your customers are using your services.Every week, the Amazon Redshift team performs a scan of their fleet and generates a Jupyter notebook showing an aggregate view of customer workloads. They don't collect the specific queries, just generic information such as the operation, count, duration, and plan shape. This yields hundreds of millions of data samples. I picked a few graphs to demonstrate, showing frequency, duration, and query plan for both SELECT and INSERT/UPDATE/DELETE statements.

Spotlight

Emergent Networks

Since 1985 Emergent Networks, LLC has been providing technology consulting and solutions for customers by leveraging current and emerging technologies relevant to your business. We provide consulting and services that deliver real results.

OTHER ARTICLES

Benefits of Kubernetes on bare metal cloud infrastructure

Article | March 3, 2020

Let’s start with a brief recap on cloud native and CNCF – Cloud Native Computing Foundation. With the introduction of 5G, new use cases drive the need for designing applications based on containers and service-based architecture to address some technology gaps associated with virtualization. The most important technology gaps involve smoother software upgrades, automation and the realization of a CI/CD software pipeline to end customers. In the center of cloud native technology development is CNCF, an open source community driving the adoption of the cloud native paradigm across industries by fostering collaboration between the industry’s top developers, end users and vendors. Since CNCF is such a huge community, the focus on the telecom industry has been limited. But now with the formation of the Telecom User Group (A Special Interest Group) within CNCF, Ericsson has taken a leading role in telecom related discussions in the community.

Read More

Virtualization Leader Cuts IT Delivery Costs by 20% with HyperFlex

Article | February 20, 2020

IT is at the center of rapid innovation in today’s world where data is everywhere. To keep up with the data, companies worldwide are looking for the simplicity of hyperconvergence from core to edge and multicloud. One such company faced with data center challenges was Malicis. A boutique consulting firm specializing in desktop and systems virtualization, Montréal, Canada-based Malicis was tackling issues such as: simplifying data center operations, improving the performance and stability of IT services, and integrating service offering with third-party solutions. With challenges of the pace of modernization at hand, founder and CEO Marc-André Lapierre had to be judicious with prospective customers and make sure their workload would fit the company’s environment. While his firm’s virtual desktop and disaster recovery services have always been in high demand, they used to come with limitations.

Read More

VMware embraces Kubernetes with vSphere 7

Article | March 10, 2020

VMware has added Kubernetes support to run containers and virtual machines simultaneously in the new vSphere release. The virtualization giant can now also offer a single management domain that covers containers and VMs in the hybrid cloud. vSphere 7, launched today, represents the first fruits of the company’s Project Pacific. Project Pacific is in turn a component of VMware parent Dell’s wider Tanzu initiative to enable its overall product set to build, run, manage, connect and protect containerised workloads alongside virtual machine workloads. (Read more about Tanzu deliverables, in a Dell blog.) Deepak Patil, SVP and GM for cloud platforms and solutions at Dell Technologies, provided a quote: “As organisations look to solve for managing their private clouds seamlessly with multiple public clouds, we’re now able to extend our capabilities to both VMs and containers with a single hybrid cloud platform.

Read More

Security concerns hampering adoption of containers and Kubernetes

Article | February 19, 2020

Enterprises are having significant problems with security when it comes to Kubernetes and container deployments, according to a new survey from security company StackRox. In the winter 2020 edition of its State of Container and Kubernetes Security Report, StackRox researchers found that 94% of respondents experienced a security incident in their Kubernetes and container environments during the last 12 months. This very high number of security incidents led to about 44% of organizations delaying or outright halting application deployment into production.Researchers spoke with more than 540 IT professionals, the majority of whom work for tech companies or organizations involved in financial services.

Read More

Spotlight

Emergent Networks

Since 1985 Emergent Networks, LLC has been providing technology consulting and solutions for customers by leveraging current and emerging technologies relevant to your business. We provide consulting and services that deliver real results.

Events