Managing large databases is hard. Having changes go through a dynamic environment to production with thorough testing is also hard. When you have a 100TB+ production system, keeping non-production copies up to date is even harder still. Most companies can’t afford to keep exact copies of production due to the size and scale, so compromise the non-production environments. This is no longer necessary. Nutanix Era solves this acute database management problem, and many others!

Nutanix Era is a Database-as-a-Service Solution that solves many of the acute problems facing businesses and DBA’s today when managing large number of databases, or large databases, which impact productivity, reliability, availability and performance.

One of the biggest benefits for customers that have large databases is the ability to take a complete clone of the production system and make multiple copies for non-production uses. Any number of clones can be created without impacting storage capacity requirements, this is due to using native Nutanix API’s in the cloning process. The following 9 minute video shows an example of a 100TB Production Oracle RAC DB being refreshed to 10 clones, for use in testing and development. The clones don’t take up any additional space until the moment data is changed. I hope you like the Jazz music :).

If you like what you see and you’d like to get some hands on experience with Nutanix Era, please take a look at the Nutanix Era Test Drive – Available Online from the comfort of wherever you are.

Recently version 2.0 of Era was released, including the following enhancements.

Hybrid Cloud Flexibility and Multi-cluster Management

With Era 2.0 we have extended our database platform capabilities across clusters and hybrid cloud. This provides our customers the flexibility to build and manage databases on their own terms, giving them the freedom to develop and deploy databases of their choice in the environment they want.

Expanding Database Engine Portfolio with SAP HANA

SAP/HANA has joined Oracle, MS SQL Server, PostgreSQL, MySQL, & MariaDB in the portfolio of database engines supported by Nutanix Era. SAP/HANA customers can now leverage Era’s 1-click capabilities to create an end-to-end sandbox environment on the Nutanix HCI platform.

Extended Choice and Support with PostgreSQL

With Era 2.0, PostgreSQL admins can now take full advantage of all Era capabilities, including patching and in-place restore. In addition, Nutanix will now provide 24×7 support for PostgreSQL databases provisioned by Era, allowing customers to call our support number with any issues.  

Enhancing Service Model Delivery with HCL Partnership

For customers seeking fully-managed database services, Nutanix has partnered with global technology company HCL, to offer a joint solution called SKALE DB powered by HCL and Nutanix. The partnership provides a secure, scalable, cloud-ready managed DBaaS offering integrated with Nutanix Era and Prism for automation and management for database & hyperconverged infrastructure.

There is much more to come in future releases!!

Final Word

Good DBA’s are far too valuable to allocate to mundane tasks that can be automated. The DBA’s can create and control the policies, and delegate to competent team members the rights to perform tasks for themselves. DBA’s can then focus on solving real business problems, improving data efficiencies and delivering far greater insights. Utilising Nutanix Era can greatly reduce the storage required to manage database environments and maintaining large numbers of non-production systems. The capital saved from not needing excess storage infrastructure can be redeployed to more business focused investments. There is no need to migrate defects to production because you couldn’t test changes on a full scale copy in your non-production systems! Don’t forget to get some hands on experience with Nutanix Era, please take a look at the Nutanix Era Test Drive – Available Online from the comfort of wherever you are.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster. Copyright © 2012 – 2021 – IT Solutions 2000 Ltd and Michael Webster. All rights reserved. Not to be reproduced for commercial purposes without written permission.

 There are a lot of environments that are running MySQL and PostgreSQL to support their systems. My team at Nutanix and I have been getting a lot of enquiries about how to set up these databases for best performance, and customers have also been using them to benchmark and baseline different systems. One of the challenges with these databases is that they give only limited control over where data files and transaction logs can be placed, and this makes increasing parallelism of IO a little bit of a challenge. Your database is just an extension of your storage and all storage devices, even virtual ones, have a limited queue depth that you can work with. Unlike Oracle, SQL Server, Sybase, DB2, etc you can’t just create a whole bunch of mount points and spread your data files over them (which increases available queue depth and potential IO parallelism). But the solution to this problem is made quite simple with Linux LVM (Logical Volume Manager). I’ll take you through some of the steps I took to set up a test VM for MySQL testing with HammerDB and PostgreSQL for testing with PGBench.

Firstly what is HammerDB and PGBench?

HammerDB is an open source database testing tool that simulates a test that is styled along the lines of TPC-C. But it doesn’t implement a full TPC-C test and can’t be compared to published audited results. There is documentation on the HammerDB Test here, and the binaries can be downloaded from SourceForge here. Although these tests don’t compare to published audited results they can be used as a comparison between two systems that have run the same test. The output is two values, a TPM (Transactions Per Minute), which is database specific and can’t be compared between two different database types, and a NOPM value (New Orders Per Minute), which can be compared between different HammerDB tests and database types. NOPM is the more relevant metric to keep an eye on.

PGBench is a built in performance test benchmark that comes with PostgreSQL and it simulates a TPC-B type test, which is a database stress test. TPC-B as a test has been obsoleted by the TPC org, and the TPC-B test within PostgreSQL couldn’t be used to compare against any audited results, but it can be used to compare against like tests using the same tool. This should be used only to compare between different PostgreSQL database performance results.

Setting up a Linux LV for use with PostgreSQL or MySQL (Example based on PostgreSQL):

For the tests I was doing I used a striped logical volume (8k stripe size) across 8 disks of 10G each and copied the data directory of the postgresql to the new location. All eight disks were equally distributed across 4 vSCSI controllers (PVSCSI recommended with VMware vSphere). The total volume size was about 80GB.

1. vgcreate vol0 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

2. lvcreate -i8 -I8 -l 100%FREE -n lvol0 vol0

3. mkfs.ext4 -m 0 /dev/vol0/lvol0

4. mkdir /home/postgres

5. Add “/dev/vol0/lvol0         /home/postgres          ext4    defaults,noatime,nodiratime        0 0” in /etc/fstab

6. mount -a 

7. By default, postgresql installs the data directory at /var/lib/pgsql/9.3/data. I copied the data directory to 

/home/postgres/data

8. Now, when we start the postgres server, we should use the -D option to point the new location of the data directory. 

/usr/pgsql-9.3/bin/postgres -D /home/postgres/data

This should now use the database at /home/postgres and write it across all 8 disks. vgdisplay and lvdisplay should look like this.

[root@RT17-CentOS-1 9.3]# vgdisplay

  — Volume group —

VG Name               vol0
System ID
Format                lvm2
Metadata Areas        8
Metadata Sequence No  4
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               1
Max PV                0
Cur PV                8
Act PV                8
VG Size               79.97 GiB
PE Size               4.00 MiB
Total PE              20472
Alloc PE / Size       20224 / 79.00 GiB
Free  PE / Size       248 / 992.00 MiB
VG UUID               hmmbRZ-xoHF-69K6-7DkM-EIqo-jqoY-3X8qz8

[root@RT17-CentOS-1 9.3]# lvdisplay

  — Logical volume —

LV Path                /dev/vol0/lvol0
LV Name                lvol0
VG Name                vol0
LV UUID                ZHija9-OXek-59Ud-Q4uJ-gThy-55Zf-CJRIaD
LV Write Access        read/write
LV Creation host, time CentOS-1.local, 2014-10-14 10:34:17 -0400
LV Status              available
# open                 1
LV Size                79.00 GiB
Current LE             20224
Segments               1
Allocation             inherit
Read ahead sectors     auto
– currently set to     256
Block device           253:2

One additional step you could take it to create another logical volume and move the pg_xlog directory. This can also increase performance for PostgreSQL. For MySQL the steps are similar, except you move the MySQL Home Directory to the LVM, which you might mount at /home/mysql (check /etc/my.cnf for location details of the default install).

An example of a performance results based on PostgreSQL 9.3 from a single VM of 4 vCPU’s, 32GB RAM, running on a Nutanix 3050 node is below. The templates used to do the MySQL HammerDB and PostgreSQL PGBench testing are available from Nutanix via your SE or Nutanix Support.

PostgreSQL Test Results with PGBench:

Running tests using: psql -d pgbench
Script tpc-b.sql executing for 120 concurrent users
transaction type: Custom query
scaling factor: 1500
query mode: simple
number of clients: 120
number of threads: 6
duration: 300 s
number of transactions actually processed: 1021500
tps = 3402.171378 (including connections establishing)
tps = 3403.517277 (excluding connections establishing)

An example of the TPM for a single MySQL Database VM with 4 vCPU’s, 32GB RAM, running on a Nutanix 3050 node is below. NOPM is around 31K. Note that TPM from one database type can’t be compared to another database type. So the MySQL TPM below can’t be compared to the TPM of an Oracle database. But the NOPM can be compared. If you’re running a test with MySQL with a similar configuration though then you can compare the TPM.

Final Word

Before migrating any system to a new platform you should record or baseline the exiting performance and then validate the performance after the migration. You should make sure to read and follow any relevant vendor best practices to get the best out of your platforms. Any improvement in performance means less hardware overall needed to achieve acceptable performance, or the longer you can run a system before an upgrade is required. Although the HammerDB test doesn’t fully stress out the storage, the performance of your storage will make a difference to your test results. As will the set up of your OS and the CPU/Memory configuration of your VM’s and platform overall. It’s a good test of the overall platform and all it’s sub-components.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

Your database is just an extension of your storage. Many of you may have heard me say this before in presentations, such as at VMworld or a VMUG. Understanding this concept is the first step on the way to configuring your database servers for optimal performance. This is true regardless if the databases are deployed natively or virtualized, and regardless if they are still living on mainframes, Unix platforms, or on x86. This is one of the 5 key principles that I go into detail about in the book I’m co-authoring with Michael Corey and Jeff Szastak titled Virtualizing SQL Server on VMware: Doing It Right by VMware Press (available for pre-order, see my books page or the right side bar). In this article I will take you through some of the areas I have seen for over a decade cause the biggest problems for performance of database servers of all types, and on all sorts of platforms, at a high level. If you want more, and the detail, then you should seriously consider buying my book. Based on my experience these principles apply regardless of the platform, technology, or database management system you choose to use.

Because your database is an extension of storage most of the performance issues I see are related to storage. This is just the high level of what you need to know in order to tackle this area. Although storage is correlated with other resources as well, which you shall see.

Your database is a big cache sitting at the end of the storage chain that is designed to retrieve and update data as quickly as possible. This cache effect is important as it means read IO’s can be reduced to back end storage if we allocate enough RAM. This also means there is a direct trade off between memory allocation and read IO performance. The more RAM the less read IO needs to go to physical disk and the less burden on your storage resources and the faster the read responses. If only you could load your entire database in memory (such as with SAP HANA and Oracle Times Ten).

When IO’s do need to go to physical disk, both reads and writes, the database expects fast and parallel access to storage resources. The parallel part is the most important. Databases expect to be able to query and update many blocks and files all at the same time. As an architect of a database platform it is our job to reduce as much as possible any bottlenecks that will prevent a database from accessing all the storage resources it needs, and to the degree of parallelism that it needs. We need to architect each logical layer of the database for optimum parallel performance.

The first place we can get contention and bottlenecks is inside the database itself in it’s caches. This can be caused by many queries trying to touch the same database pages and update the same database pages all at the same time. It can mean that you haven’t split the data over enough data files and that there is insufficient numbers of data files to keep the system busy processing queries. So the first thing you need to do to reduce logical contention in the database itself is have enough data files.

The number of CPU cores can play a part in determining what is ‘enough’. Each CPU Core (physical or virtual) is a single thread, and can only do one thing at a time, even though it does it very fast. So if you want to keep your CPU’s busy processing transactions and doing useful work (which you do), you need enough data files for all your CPU’s to be doing something with them. The general rule of thumb to reduce logical contention and improve parallelism of access in a database is by having at least one data file per CPU core. Having a single data file and 64 CPU cores is probably not going to result in great performance, it will also lead to other bottlenecks, which I will now touch on.

The next step to increase parallelism of a database system is to have access to wide and deep storage queues, and enough physical storage devices to handle the likely peak loads in a reasonable response time. When it comes to storage, queues are everywhere and we need to understand them. They are not necessarily a bad thing, unless they are a bottleneck, as they allow us to issue parallel IO’s and get more responses concurrently. We want to make sure that we are not queueing IO’s up inside the operating system unnecessarily, which will delay responses, and put additional load on the database server.  So we want to get as many IO’s as fast as possible from the database system, through the operating system, to the storage controller and to the physical storage.

The first place we will encounter the queues is in the disk device where our database files are stored. Each individual disk device has a particular queue depth limit. The actual limit depends on the type of disk and the type of controller that is being used. For example, an LSI Logic Controller has a queue depth limit of 32 outstanding IO’s (OIO) per disk device. Other controllers may have other limits, some can be tuned (see this article), and some can’t. This means, in this example at leasts, that you can issue a maximum of 32 IO’s in parallel to that device. In most enterprise databases a single disk device does not have a deep enough queue to handle enough IO’s in parallel to meet the performance requirements, and therefore multiple devices (disks or LUNs) are required.

When you deploy those multiple devices you then place your multiple data files over those multiple disk devices to increase the parallelism of access. If you have a DB that doesn’t allow easy striping of data base files and table spaces over multiple disk devices then you can use a logical volume manager to stripe the underlying disk devices and present them as a single volume to the operating system (this is before we touch a SAN or other type of underlying storage). But this does increase management complexity somewhat compared to a simple filesystem, depending on the volume manager chosen. You can’t indefinitely increase the number of data files and the number of disk devices without reaching our next bottleneck.

The next bottleneck is the storage controller itself. Each storage controller has a queue limit as well. In the case of LSI Logic SCSI Controllers it’s 128 total outstanding IO’s. So one controller can support up to 4 disk devices, each running at 32 queues, if all the queues are in use, before there is any contention or bottlenecks. This is even though a LSI controller can have 15 disks connected to it (more if it’s an HBA). If you have too few queues available on a disk device you start queuing IO’s in the operating system, and if you have too few queues in the storage controller you start queuing up IO’s in the operating system. Every moment an IO is queued in the operating system is a moment that the IO can’t be serviced from disk. Some storage controllers allow their queue depths to be modified and some don’t. HBA’s generally can have their queues modified, and this involved tweaking the settings of the particular driver. One easy way around the per controller limit though is to have multiple controllers.

So now we have multiple data files, multiple disk devices and multiple storage controllers. At this point from the database down to the operating system we have increased the parallelism of IO’s and the amount of queues that the database has access too. The chances are, if we’ve given the database enough resources at this level, that the actual storage devices will now be the bottleneck, as there is a smooth IO path through the OS. We are not adding any unnecessary IO latency in our operating system configuration. Below are two example images of how this might look. One for Oracle and one for SQL Server.

In both of the above example diagrams the database is deployed on a virtual machine. But both have a similar layout with multiple disk devices and multiple storage controllers. With the SQL Server example you could add further storage controllers to increase queue depth and parallel IO access.

All of the above has really been about designing for performance. When you design for performance you will usually end up with sufficient capacity. If you design for capacity you will find that you end up with lots of unusable capacity at the end of your performance. The physical storage devices matter, but this article is not going to go into physical storage design. That is between you and your storage vendors. This is all about the OS and Database layer and the bottlenecks that commonly occur there. However I will cover one more topic briefly and that is IO size.

Not all IO’s are created equal. The IO pattern and the size of the IO’s matters a lot. You will commonly see storage vendors quoting IO performance in 4KB IO size and in different IO patterns. Usually however applications use much larger IO sizes (8KB, 16KB, 64KB and higher), databases in particular can have lots of random IO, and can be either mixed read and writes, or heavily write biased. The size of the IO’s can have an impact on the controllers efficiency at processing those IO’s. Lots of small IO’s may have a higher overhead than a smaller number of larger IO sizes, although they may have similar throughput. The randomness has a big impact on performance when relying on rotational disk media (spinning hard disks) especially, as the heads have to move to the different block locations and this causes delays that wouldn’t exist for more sequential workloads. In the real world small IO sizes and lots of IOPS generally has a high overhead and low overall throughput and you see lots of random IO.

So what does this mean? Well it means that an IOPS figure by itself is completely worthless to you. It gives you absolutely no information about the performance, requirements or constraints. It also does not allow you to compare to any vendor benchmarks to get an estimate of the performance you should expect (at least not in an accurate way). If you are issuing 512KB IO sizes then 300 IOPS might be a lot. But if you are using 4KB IO sizes then 12K IOPS might be a lot. It all depends. Oracle databases issue a lot of 8KB and 16KB IO sizes for data files and 512KB or above for transaction log. SQL Server issues lots of 64KB IO sizes. SQL Server issuing 5K random mixed IOPS at 64KB might be the equivalent of 160K 4KB IOPS, depending on the type and layout of the disks. The best way to understand your workloads is to baseline them and get to know the IO patterns, many monitoring tools can make performance management easy even at large scale and give you good information about performance. You should treat any vendor performance data reported in 4KB IO sizes and without mixed random workload patterns with skepticism. You should ask your vendors for applications IO performance data instead, as that will give you a more accurate indication of capabilities.

Final Word

I see problems in database performance every week in my interactions with customers all over the world (which for me also usually means at all times of the day and night). Usually where there is bottlenecks in the operating system design and where the database is not able to issue enough IO’s in parallel primarily due to not splitting out database files, disk devices and storage controllers. These problems exist with physical databases just as much as they do with virtual databases. Probably more with virtual databases as some people think the rules of the operating system and virtual controllers have somehow changed what you should do for a database. They haven’t. Virtualizing a database does not change the best practices of the operating system much if at all. The same things you did to a physical database to improve IO performance, as I’ve covered here, still apply in a virtual world. The queue limits of SCSI disks and SCSI controllers still exist. So you need to properly design your systems to avoid these bottlenecks where necessary. Hopefully by gaining a high level understanding of these concepts I can help you avoid some performance traps.

I go into a lot more detail, with a lot more examples, diagrams, and ways to tune and tweak your designs to improve performance in my book. So if you are virtualizing SQL Server, or thinking about it, then Virtualizing SQL Server on VMware: Doing It Right might be a good resource for you. The book is available for pre-order now and will be published in August 2014. Michael Corey, Jeff Szastak and I will be doing author signings of the book at VMworld in the USA and possibly Europe and at vForum in Sydney, hope to see you there.

As always your feedback and comments are appreciated.

—

This post appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.