| Unique Visitors |
Powered By 
| Unique Visitors |
We recently changed over from Vodafone VDSL to Chorus UltraFast Broadband (UFB) Fibre Max with Vodafone (See Vodafone Broadband Plans). The new Fibre connection is amazingly fast, but we soon discovered some limitations with the Vodafone router. I found out that the Vodafone UltraHub doesn’t support static routing. This meant that most of our home network couldn’t access the Internet. This article will show you how I worked around this limitation and created a Home Fibre Router / Firewall in the process. Warning: Sorry to anyone reading this in Australia who has NBN, feel free to migrate to New Zealand (Or Singapore).
You might be wondering why the featured image on this article is the Palo Alto Networks Logo and one of the 3020 Firewalls. When I ran into this problem of the Vodafone UltraHub not supporting static routing I contacted a friend at PAN to see if there was anything they had that might be able to help. I was offered a trail of a Virtual Edition Palo Alto Firewall (VM-300 – 4 vCPU, 9GB RAM), which I am keen to try out. I’ll definitely be writing about it when I get hold of the trail, and letting you know how it compares to a roll your own home solution. Thanks Palo Alto for being willing to work with me on a trial. All going well I may well switch over to it, as it’ll make connecting to other people around the world, who also have PAN’s much easier.
I’ve previously written about how we converted our Home Area Network (HAN) to a Leaf Spine Architecture, and how to create a low latency L2 network with Dell Force10 and Cumulus Linux. So we have a fairly complex set up here and a lot of servers and other devices and lots of VM’s on different VLAN’s and subnets. So when our Fibre connection came along it was a bit frustrating that most of the network couldn’t access the Internet. To complicate matters we also have a Vodafone SureSignal so we can get mobile service as we are in a coverage dead spot. The SureSignal requires special configuration, we’ll get to that later.
We have OSPF as our main home routing protocol, and I have the spine now on 2 x Dell S6000 32x40GbE switches, with the main leafs being 2 x S4048’s. We have our 1G boarder leaf’s running 2 x Dell N2048’s. All of the 1G segments, Wifi access points, and Internet access is off the boarder leaf.
I knew I needed something that could talk OSPF to the rest of our switches for all the VLANs and subnets as that would make it easy to get the routing correct and minimize the number of static routes. It needed to be able to support NAT and be stateful (and allow IPSEC) so it could handle the Vodafone SureSignal. It had to be able to talk to the Chorus GPON ONT directly, as any other router / modem would. I also had to minimize downtime in the process (lack of Internet access is a P1 at home and I have to apply for a maintenance window in advance unless I want to hear a lot of screaming). I decided to try my hand at setting up a Linux VM and seeing if I could make that work in the interim until I could find a more robust solution.
The speed through our original Vodafone UltraHub wasn’t too bad, as you can see below.
I’ll explain the physical network set up as well as the VM configuration as we go. First let’s look at the physical network and how I connected the ONT (the box that sits inside the house) to the environment without having a dedicated modem/router. This ended up being easier than I had thought.
Regardless of who your ISP is, one of the physical network providers provides the actual connection. In Auckland that is Chorus. They use an IPoE connection, which means basically it’s DHCP on an Ethernet link. This is much simpler than using PPPoE or other types of connection. Some Fibre connections may still need PPPoE, so please check with Chorus or your provider / ISP as you may need some additional things to get a Linux based router working with PPPoE. For the example I’ll give we just used an Interface with DHCP enabled. The physical connection comes in on a tagged VLAN. In our case that was VLAN 10.
The original Vodafone UltraHub had been configured to allow tagged packets from VLAN 10 get a DHCP IP address. All I needed to do to allow our physical home network access then was to create a VLAN 10, allow tagged packets to the relevant ports (making sure all the switches would pass it to each other), and then create a port group on my VM hosts with VLAN 10 tagged for the VM to connect to for the external network. Apart from having to configure VLAN 10 on quite a few switches and port channels it was surprisingly easy to get working. Don’t make the mistake I made at one point and set one of the switch ports to be untagged on VLAN 10 and connected that port to the ONT, that won’t work, as the ONT is sending tagged packets.
Once the physical network connectivity was established I created my Router VM. You can use any Linux distribution you like, in my case I chose CentOS. I configured the VM with 2 vCPU and 2GB RAM, with a 100GB Disk. I gave it 2 NIC’s, one on the Internal Network (zone=internal) and one on the External Network (zone=external). I did a minimal install of the OS and then added in Epel-Release, Quagga (dynamic routing), Traceroute, IPTraf, TCPDump, Links, DDClient (to update DynDNS), NSLookup (Bind-Utils), WireShark (wireshark-gnome), Net-Tools. The minimal install has SELinux, FirewallD, SSH and the basics that you need.
Here is the config I used on the Linux VM:
# cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Optimize Memory Management Settings
vm.overcommit_memory = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 15
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100
vm.swappiness = 0
# Network MTU Probing - Michael Webster
net.ipv4.tcp_mtu_probing=1
# Optimize Network Stack and Memory Buffers
# Increase TCP max buffer size setable using setsockopt()
net.core.rmem_max = 536870912
net.core.wmem_max = 536870912
# Increase Linux autotuning TCP buffer limit
net.ipv4.tcp_rmem = 4096 87380 536870912
net.ipv4.tcp_wmem = 4096 65536 536870912
# Increaes max backlog of packets and congestion control method
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_congestion_control=htcp
# advanced network stack tuning
net.core.somaxconn = 65535
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 65535
fs.aio-max-nr = 1048576
fs.file-max = 6815744
# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos_router/root rd.lvm.lv=centos_router/swap rhgb quiet elevator=noop iommu=soft apm=off numa=off transparent_hugepage=never vmw_pvscsi.cmd_per_lun=256 vmw_pvscsi.ring_pages=32"
GRUB_DISABLE_RECOVERY="true"
# grub2-mkconfig -o /boot/grub2/grub.cfg
# cat /etc/quagga/ospfd.conf
!etc/quagga/ospfd.conf
hostname router
log file /var/log/quagga/ospfd.log
interface ens192
description Internal
ip ospf network broadcast
!
router ospf
ospf router-id 192.168.xxx.xxx
network 192.168.xxx.xxx/24 area 0
default-information originate always
access-list localhost permit 127.0.0.1/32
access-list localhost deny any
line vty
access-class localhost
Then I configured the firewall using firewall-cmd
# Put the correct interfaces into the correct zones
firewall-cmd --zone=internal --change-interface=ens192
firewall-cmd --zone=internal --change-interface=ens224
# Allow ospf to work for the Internal network
firewall-cmd --add-protocol=ospf --permanent --zone=internal
# Set internal target to allow traffic from Internal network
firewall-cmd --permanent --zone=internal --set-target=ACCEPT
#Forward email and https traffic to internal system
firewall-cmd --permanent --zone=external --add-forward-port=port=25:proto=tcp:toport=25:toaddr=xxx.xxx.xxx.xxx
firewall-cmd --permanent --zone=external --add-forward-port=port=443:proto=tcp:toport=443:toaddr=xxx.xxx.xxx.xxx
#reload firewall config and display zone settings
firewall-cmd --reload
firewall-cmd --zone=external --list-all
firewall-cmd --zone=internal --list-all
It was at this stage that I ran into trouble with OSPF not wanting to work properly on the Linux VM. None of the neighbours would respond correctly. After some Googling it looked like the most obvious issue was going to be the MTU and sure enough the logs supported that as the physical switches were trying to send packets of 9198 bytes and the Linux VM could only accept 9000. Due to the various different switches in the environment I found I had to configure multiple different MTU settings to get everything to converge. On some switches it was MTU 9018 and on some others it was 9000. After getting the MTU’s correct across the different switches everything started working.
After switching over to the Linux Firewall the speed of the connection was even better. However the Vodafone SureSignal would not work. I stumbled across a configuration article for complex networks on the Vodafone NZ web site. This advised to open up and forward port 123, 500 and 4500 UDP from the Firewall to the SureSignal and ensure the SureSignal can resolve DNS correctly. In spite of doing this, and calling Vodafone NZ support it was not possible to get the SureSignal to function. In the end I decided to configure the Vodafone UltraHub and SureSignal on a separate VLAN so they could directly communicate and then put the Linux Firewall VM on that same VLAN so the rest of the network could have Internet access. This resulted in some Double NAT, but everything is now working.
Final Word
I hope in the future we don’t actually need a SureSignal and we can get proper mobile service, maybe when the new Vodafone NZ 5G network is launched in December 2019. Then we’ll be able to directly connect our Linux Firewall VM to the ONT and keep it simple (and fast!). I’m also hoping that when I try out the Palo Alto Networks VM-300 that I can get around the annoying issues I had with the SureSignal and that it works directly via the PAN to the Fibre ONT. Time will tell, and I’ll write about how the evaluation of that goes when it’s completed.
I’ve also been in contact with Chorus regarding their 10G Trial. I have the equipment to utilize this and enough devices at home and plenty of requirements to upload and download large data sets to the USA and elsewhere. The 10G service is due to go live in 2020 anyway, when it does, I’ll definitely be looking to get on it, even if I don’t make it onto the trial.
If you don’t have a SureSignal then you won’t have any of the problems I ran into and you’ll be able to connect your ONT directly to your Linux host or VM to act as your Internet firewall using the above info. Alternatively you could check out a Palo Alto Networks VM-300 (or one of their physical firewalls) if you run a corporate environment and want something robust and simple to use that also offers context aware security. I hope this is helpful in some way, let me know in the comments below.
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2019 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Some vendors in the storage, hyper-converged, and cloud industries may be playing Russian Roulette with their customers’ data. Solutions are not created equally, some turn off basic data integrity features such as data checksum by default, or when there are performance problems. Some don’t have background consistency checks and scrubbing to protect against silent data corruption or latent sector errors. Others might use consumer grade devices that may have a higher risk of error and higher failure rate. In the age of software defined solutions, the customer has become the storage platform architect. There is enough rope to hang yourself (your data and your platform availability) any number of different ways. Which is why having a software foundation and integrated solution that has been properly validated from end to end, and that contains data integrity and enterprise data protection features at it’s core, should be the highest priority. Return of data, in the form it was originally written, at any scale, while protecting against known data and device risks, is of upmost importance. How important is performance (IOPS, Latency and Throughput) if you can’t even read back the data you originally wrote? Here are the top 10 questions you can ask potential vendors to find out if they really have protecting your data as their top priority.
Before we get started with the questions, it’s always good to have some science and evidence to back things up. Here is one paper – An Analysis of Data Corruption in the Storage Stack. Another paper – Characterizing Private Clouds: A Large-Scale Empirical Analysis of Enterprise Clusters. Both papers cover large scale studies. Any study across a small population of devices or a very small sample size is going to be invalid. Any conclusions from something like a 30 drive study isn’t going to be valid when you have tens of thousands, hundreds of thousands, or millions of devices.
Questions to ask your potential solution vendor:
There are plenty more questions that could be asked, but the 10 questions above cover the most common areas of risk in terms of data integrity, data protection and data loss prevention, and that are not always protected against, at least not by default, with some systems.
Final Word
From a Nutanix point of view, as a leader in the Gartner Magic Quadrant for Hyperconverged Infrastructure, we take data integrity seriously and it’s our top priority. We protect against all of the areas highlighted in the questions above and we have a paper that explains the Infrastructure Resiliency of Nutanix Solutions, which compliments the research paper on enterprise clusters. We also have many hundreds of thousands of devices in production that are proactively monitored from which we can draw real world data, and a very thorough device qualification and QA process, which limits risk. We use similar high standards across all hardware platforms that our software supports. Our software is built based on a philosophy that hardware will eventually fail, so we must deal with these failures gracefully. Your data deserves better protection!
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2018 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
I recently had the pleasure of attending SAP & ASUG SapphireNow conference in Orlando Florida, where the audience learned that SAP systems now have up to 76% of world GDP running through them. I had many questions about sizing SAP for virtualized environments, which also comes up on a daily basis in the work my team and I do at Nutanix. I realized that nobody has really published anything on this topic since 2013 when Vas Mira from VMware wrote SAP on VMware Design and Sizing Example, which is also still relevant today. But given the changes the IT industry has been through, and the rise of hyper-converged infrastructure (such as Nutanix), it is about time we did a brief review, so you can easily size and successfully deploy your SAP environments virtualized. This article will take you through some of the basic sizing guidelines, including specific considerations for hyper-converged environments.
Before we get into sizing, why would you want to virtualize SAP systems anyway? Some of the most common reasons are as follows:
I have written about these and other aspects in the article covering Nutanix becoming SAP Netweaver certified 2 years ago and there are more details below.
Nutanix: First Hyperconverged Vendor with SAP Certified Platform
However to illustrate some of these advantages it makes it easier to understand if you can see it in action. Here are two quick videos that show some of these benefits with live running SAP systems. The first video covers infrastructure maintenance and upgrades without any SAP application impact.
This second video covers rapid cloning of SAP systems, which only takes just over a minute and doesn’t consume additional storage capacity due to smart modern storage techniques. This allows higher quality releases to get to production faster, with less defects, as they can be more rapidly and accurately tested. It also removes the infrastructure from being a bottleneck to SAP system testing.
SAP Sizing for Virtualized Environments
To keep the sizing examples and process simple there are a number of assumptions. The guidelines don’t apply to every situation, or every SAP product. But they can be used as a general guideline to get a sense of how much infrastructure may be required and how VM’s may be sized if an SAP system were to be virtualized. For new SAP environments it is recommended that qualified system integrators are engaged to ensure proper sizing based on actual business requirements, which may involve a QuickSizer exercise, which can then feed into later virtualization sizing. Where you have an existing physical SAP environment you can use SAP Application Performance Standard (SAPS) from SAP Sales and Distribution (SD) Benchmarks to help. We will use SAPS in the examples here.
When considering virtualizing an SAP system it is important to consider the following:
What are SAPS?
SAP Application Performance Standard (SAPS) is a hardware-independent unit of measurement that describes the performance of a system configuration in the SAP environment. It is derived from the Sales and Distribution (SD) benchmark, where 100 SAPS is defined as 2,000 fully business processed order line items per hour. In technical terms, this throughput is achieved by processing 6,000 dialog steps (screen changes), 2,000 postings per hour in the SD Benchmark, or 2,400 SAP transactions.
Because the benchmark is hardware-independent it can be used to calculate sizing from Unix systems to X86 when considering a system platform migration. It is also useful when considering non-SAP applications because it provides a relative performance metric between dissimilar systems. It is widely used by many different system vendors and provides consistency in the method of measuring performance.
SAPS benchmarks are available for 2 tier and 3 tier application configurations from the following location:
Nutanix has an SAP certified SD 2 Tier Benchmark that can be referenced for sizing.
Certified and published benchmarks are always on supported platforms from certified and supported SAP system vendors. This means that SAP customers can rely on the support of both SAP and the vendor publishing the benchmark for their critical systems.
Benefits of using SAPS include:
Using SAPS for a Simplified Sizing Calculation
For arguments sake, lets use the Nutanix Certified 2 Tier benchmark for sizing an SAP production system that needs 140,000 SAPS @ 65% utilization.
Firstly, we need to discount the benchmark SAPS value by 10% as it was done on bare metal to allow for a virtualized SAPS number. This suggests that at 100% utilization virtualized the hardware platform (8150-G5) can support 94K SAPS.
To get the SAPS per core we divide the value by the number of cores, in this case 44. This gives us 2140 SAPS per Core at 100% utilization.
To get the SAPS per core at 65% utilization, we need to multiple the SAPS per core by 0.65. This gives us ~1400 SAPS per core (SAPS / Cores, in this case 44 cores per server) at 65% utilization. Note: These numbers used have been selected to make the calculations easy.
If our SAP system requires 140K SAPS and we get 1400 SAPS per core, we know that we’ll need 100 CPU cores. We also know from this we will require 800GB RAM (8GB per CPU Core). Remember, this is a production instance, so we assume 1 vCPU = 1 core. Based on 1400 SAPS per core, we can calculate how many SAPS the NX8150-G5 can do at 65% utilization, which is ~61K SAPS. However we need to reduce this by the resources that will be consumed by the storage controller, which is 4 cores. This leaves us with 56K SAPS at 65% utilization per NX8150-G5. Immediately we can tell that we will need 3 x NX8150-G5 to cover the workload (140K / 56K and round it to nearest whole number), and at least 1 for failover and maintenance capacity (N+1 design), so we would use 4 x NX8150’s for the environment.
Now, we can allocate the resources between app servers and database servers based on the above calculations. Assuming the split of 70/30 between app servers and database we will use 70 CPU cores for app servers and 30 CPU cores for database. The Database server will be configured with 240GB RAM at least (probably 256GB). App servers could be configured with 4 vCPU and 32GB RAM, and to cover the requirements we would deploy 18 app server VMs (72 cores in total). ASCS with 1 vCPU, 8GB RAM, and other AD or utility servers will also be supported in the environment. Why do we split apps into multiple instances over multiple VMs? Because it helps in getting better performance out of your virtual environment by scheduling compute resources more efficiently and even for SAP, having multiple smaller instances instead of 1 large application instance helps in better CPU context switching in the work processes and eventually, lower compute overheads. You can refer to SAP Note 9942 for a detailed explanation on this.
With 512GB RAM per NX8150-G5 node in this environment it would have sufficient resources for the current workload and room for additional growth, while being able to allow for non-disruptive maintenance and recover in the event of component failures.
From an IO perspective, assuming this is an OLTP environment, the Database will require 90% of the IOPS at 0.6 x SAPS, which is ~76K IOPS, which is easily achievable from the proposed configuration.
Remember that the above sizing is only an example for 1 SAP Netweaver based product production instance requiring 140K SAPS to operate. Typical SAP environments have a 3-system landscape (Dev, QA, Production) with their respective workload profiles, a mandatory 1 Solution Manager instance at a minimum and other SAP and non-SAP products to support the SAP environment ecosystem.
Some Sizing Gotchas
For SAP QuickSizer, the SAPS are already adjusted to 65% utilization. So you don’t need to reduce the SAPS per server based on the benchmarks. Don’t make the mistake a lot of people do and double discount. It will mean your utilization from your system will be extremely low and you will not have an optimal ROI.
SAP EarlyWatch Alert Reports show actual system utilization and system metrics. They are one of the most useful tools for sizing new platforms for existing environments. If SAP EarlyWatch Alert Reports are not available you can use ST03N, ST06 and DBACOCKPIT transactions in the SAP system to find out the system metrics. Some systems, especially SAP JAVA based systems will require additional tools and information, to size them.
With better information, your sizing can be more accurate, and therefore you may be able to size for less physical resources, and provide a better TCO and ROI. The lesser the information, the more conservative the sizing needs to be. Virtualize but without compromise, this is required for critical systems and may be different to how dev/test systems have been handled in the past.
At least some non-prod systems will require 100% of prod resources to allow for accurate reproduction of performance expected in the prod systems. Usually this is QA or Perf environment. Other dev, test and training environments may not require the same level of resources. Once you know the resources or relative resources per landscape you can calculate the total systems required. My experience has shown there can be between 2 and 13 landscapes per product (where a landscape is an environment, such as dev, test, QA, training, support, prod etc).
Going from Non-Unicode to Unicode system will require more resources, assume at least 1.5x the resources in terms of storage capacity, CPU and RAM.
Additional Resources
VMware SAP Best Practices Guide
Nutanix is a SAP Global Technology Partner
SAP on Nutanix Support / Certification OSS Notes:
1122387 – Linux: SAP Support in Virtualized Environments
Nutanix SAP Resources Web Page
Virtualizing SAP on Nutanix Tech Note
Final Word
SAP has supported virtualization for production systems since 2007 and most customers are choosing to virtualize their systems to yield some of the benefits explained in this article. It is critically important however that the design and implementation of the virtualized environment is done such that it is verified and tested against the business requirements, including performance, availability, failure scenarios, and that resources are not over allocated and cause unnecessary support incidents. Virtualize without compromise, and when in doubt, seek assistance from your vendor’s SAP teams. Thanks to Kasim Hansia from Nutanix for his constant support with all things SAP & Databases over the years that we’ve been working together, and to our entire SAP engineering team at Nutanix that help our customers implement successful solutions for the critical applications all over the globe.
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2017 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
That sort of thing has happened to me too. BTW, one day you should do a how to perf test article.
— Michael White (@mwVme) January 5, 2017
This all started from a tweet in response to some test results that I had written about in my article Disable Windows Disk Write Cache for Data Integrity and Better Performance. Michael White suggested I write something up about how to perf test. Thanks for the inspiration Michael. So this is it, a simple 101 introduction to performance testing. I’m pretty sure it’s not exactly what you will expect. Just like the outcomes of the tests in my previous article. I utilize these same techniques in my job at Nutanix when performance testing new application workloads
Performance testing can be a lot of fun. It can also be very tedious, and sometimes frustrating. There are high’s and lows. You can break things, and spend hours figuring out what went wrong. Like professional test driving for a car company, you often do hundreds or thousands of test iterations of very similar configurations to prove or disprove a theory, or to measure differences in configurations to find the optimal combination. This could be seen as mundane. For example, in my article VMware vSphere 5.5 Virtual Network Adapter Performance I performed over 2,000 (two thousand) individual combinations of tests to get the results, each combination run multiple times. But overall it is very similar to performing scientific experiments in other fields. If you are not familiar with the scientific method, check out here and here.
Before we get into the how of performance testing, lets first look at what it is and what makes it different from others types of testing. Broadly in IT there are two different types of testing, functional testing, where you are checking that a system performs the functions it was designed to perform in the way it was designed to perform them ( a red screen is red ), and non-functional testing, which is where performance comes into it. A non functional test might be performed to prove that a system can perform a function a certain number of times over a given period with the same result, or to see the maximum number of times something can be performed in a given time period. Rather than the colour of a screen or text in an error message.
Often performance tests will try and find the limit (before response time degrades or the system breaks), or headroom available in a particular system, or the response time under a defined load, or be designed to find the optimal configuration. In all cases, they are measured against requirements, and a baseline or control. There are quite a few types of performance testing, such as benchmark testing, load testing, soak testing, stress testing, unit performance testing, integration performance testing, spike testing, headroom testing, failure performance testing.
What follows is quite a simplified and basic description of some of the high level areas that are important. There has been quite a bit written on the topic over time and for different types of systems and applications. Such as this article about hyperconverged performance.
Step 1: The Question, Objective, or Success Criteria
Before you start anything you need to decide what question are you looking to answer or what is the objectives of your test. What are you trying to learn from the testing? For example, in Disable Windows Disk Write Cache for Data Integrity and Better Performance, I was trying to find out what the impact to performance was when turning off the disk write cache in Windows. If you are testing an application your objective might be to prove if the application can meet your business performance requirements such as x load over y time, with z response time. You might ask what is the maximum for x value I can expect for this particular configuration before response time is impacted? I particularly like answering questions like, where does this system break, to the point that it can no longer process transactions, or what happens when you subject a system to 10x expected load. I also like questions such as what happens under heavy load if a particular component malfunctions or fails?
If you are performing benchmark testing with defined industry benchmarks then others will have most likely defined the questions you need to answer. Such as how many transactions per second and what system response time did you receive from a certain configuration and under a load factor of a certain number of virtual users for a sustained period of time. There are many different types of benchmarks, some of the most useful from from the Transaction Processing Council (TPC) for database type workloads, and Standard Performance Evaluation Corporation (SPEC) for many other types of workload. Many vendors also specify their own performance tests, such as SAP Sales and Distribution (SD) 2 Tier Benchmark.
Step 2: The hypothesis or expected result
Once you know what question, now you should define what you expect, what do you think will happen, or what do you require as the result? With the example I used earlier regarding disk write cache, I could have expected write performance to reduce when cache is disabled. After all, the idea of caching is to increase performance, such as IO’s per second (IOPS) and to reduce write latency / response time. Another example might be that you expect a certain number of transactions per second at a certain response time from an application that is being used concurrently by a certain number of real users, and a given transaction mix. Just as with the questions the expected results or hypothesis can get fairly complicated. Often you can use prior research done by others to form the hypothesis or expected results or as a guide line as to what you should expect, such as reviewing the published results on the TPC.org or SAP web site if you are performing a benchmark.
Step 3 A: Testing Part A – Baseline or Control
Regardless of what you are testing you need to have a point to measure from. This is true in system performance testing as in other scientific experiments. There is a baseline or a control that can be compared to the thing that is being changed. Such as in a medical trial, there is usually a placebo that contains nothing of the drug being tested, and then there is a real drug. In system performance testing we would usually have a simple standard configuration installed that meets the functional requirements and test that to determine what the baseline results are,. Alternatively we may have one system configured as per the current standard, and then another system that gets modified over time that we run continuous tests against and then compare to the original. Your baseline or control system should not be modified during the testing period, in case you need to re-run a baseline test and collect more information.
Lets say you have an existing system based on a certain version of an application, you are planning to upgrade to a new version. You could perform a standard set of tests based on your business requirements against a non-production, but similarly configured system, as your current production system and measure the results. This becomes the baseline or control. Then you upgrade that non-production system to the next version of the system and perform the same tests again. After this you can compare the results to find out what the differences are, and see if they meet your requirements.
Chapter 11 in the book I co-authored titled Virtualizing SQL Server with VMware: Doing IT Right (VMware Press 2014), focused on baselining, which is an important part of virtualizing any applications to ensure you achieve acceptable business outcomes.
Step 3 B: Testing Part B – System Under Test and Performance Test Iterations
Now that you have your baseline you can perform multiple iterative tests on the system under test, which just means the system you are testing. In order to have a valid result the tests must be repeatable. This means that each configuration or system modification needs multiple tests in order to have a valid result. You may do 3 or 5 identical tests of the same configuration before making a change and repeating the tests. Between each iteration you should keep the modifications between test runs to a minimum so you can easily tell what setting has resulted in which change. It is not always possible due to time constraints to just make a single change between test runs, but that is ideal. Otherwise how do you determine which configuration change made the difference? If you are testing many combinations the number of test iterations can easily reach into the thousands. In the testing of the disk write cache setting I had a very defined test and only a single parameter to change between each test, however I had multiple types of IO (read and write) to test, and different patterns and sizes. The combinations can easily increase exponentially, so you need to decide what are the most important tests. The important thing to remember here is that you need to have a repeatable test and multiple consistent results (3 – 5) before your test could be considered valid and before you should move on to change configurations. I usually do 3 per iteration.
Step 4: Monitoring and Analyzing Results
There are a lot of metrics that you could measure for every system under test that can help with analyzing the results. Depending on the progress of the testing you might want to dial up or down the detail and frequency of metrics collection and increase or decrease the metrics that are monitored. Performance testing can generate a lot of data for every test iteration, so being selective about which metrics to monitor, and only selecting the most important or most relevant ones is critical for success. You don’t want too few that you miss important information, or too many that you can’t see the relevant data points. The granularity of data points, i.e. number of metrics, and frequency of collection will determine how much capacity is required for monitoring data collection.
One of the problems with monitoring is that is can impact results and it can impact performance. You need to try and keep it as light weight as possible. You also need to keep the monitoring consistent between the baseline and the system under test, else the results will become impossible to compare accurately.
Once you have the data you can start to analyze the results. Analyzing the results between test runs can take almost as long as the testing itself, it can also take your testing plan in different directions depends on the results. If you find a result that is very unexpected you may have to repeat a test with more data collection, or perform some troubleshooting. If everything has gone according to plan however you can compare the results to the baseline and determine the differences and then compare to the requirements and determine if it is a success of not. If your test was just designed to see how fast something goes with a certain configuration you might have an easy result.
In the IO testing I was doing with disk write cache, I was primarily measuring IOPS and latency of different IO sizes and patterns with disk cache enabled and disabled. So there was no defined pass or fail result, it was just designed to find out what the difference was. However I got quite an unexpected result. The performance was better with disk write cache disabled, than it was with it enabled. Based on this data I performed more iterations to validate that the results weren’t a fluke or a coincidence.
With more complex system testing you may need to run tests at different user loads, or up to the point of system saturation, to see what the maximum number of users the system can support, at what transactions per second, and what the response times are. If you are measuring to a set of business non-functional requirements, then you will know what the minimum is the system needs to achieve in order to be determined a success. Usually when considering migrations, the baseline of the existing system is the starting point and you are usually trying to achieve x percent improvement over the existing system. It is important to have an accurate baseline so you can say for sure you have achieved the desired outcome as performance to end users can be quite subjective. But you also need to know you’re measuring the things that are the most important to ensure a good end user experience.
Step 5: Drawing Conclusions, Communicating Results and Further Research
You’ve done the testing, you’ve crunched the results, now it’s time to draw conclusions, communicate the results and identify areas of further research. In terms of drawing conclusions, I’ll take the graph below as an example, the conclusion you might draw in this case is that the performance of the solution increases almost linearly as you add workload and add resources, in a predictable and consistent manner. Depending on your goals, this could be good or bad, and the amount at which it increases could also be good or bad. It’s just an example and needs to be interpreted in combination with question / hypothesis or success criteria, the baseline / control, and the requirements. Your conclusion might be that the system configuration tested did not have the resources required to meet the business requirements, or that it was over engineered and actually performed well beyond expected results and could be configured with less resources.
To communicate these results you can use many methods, but this is the time when graphs and pictures paint a thousand words (maybe tables for the very detail oriented). In answer to the question what is the performance and scaleability of IO for a defined number of Database VM’s on a defined number of servers you might draw a graph like the following (from Nutanix Performance with Oracle SLOB on All Flash Nodes) :
The type of communication and graphs you use will depend on the type of testing you did. You might want to show differences in system metrics are user numbers increase, under different types of hardware for the same tests, in a slide deck presentation, in a video, in a blog, or in a system performance report that documents all of the tests you performed and their results relevant to the stakeholders who are to read the report. Whatever method is used it should be concise, and relate back to the objectives.
Then you might want to propose some further research, such as changing hardware configuration, using a different platform, putting some different patches on the system or using a different version of the system software. One of the goals of the future testing might be to determine how few systems of a newer generation you need, knowing full well compute power is increasing over time, you may need half as many assets to run the same workload in the future.
Final Word
It would be very easy to write a book on performance testing, and there are thousands of resources already available on the topic. The aim of this article is to give a broad brush overview. By using a scientific approach to performance testing you can gain reliable and accurate results that can achieve business outcomes and provide assurance when things change that they can still meet the requirements. Updating baseline tests between versions and keeping good records are all part of the process of performance testing and constant system improvement. Even though the thousands of iterations of tests and small modifications between tests can be mundane, you can get some very exciting and sometimes unexpected results. I wish you successful testing and welcome all your comments.
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2016 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Heads Up! If you’ve updated to ESXi 6.0 U1b, build 3380124 and you have lots of templates, you may run into some problems if you update VMware Tools to the latest version. I just upgraded my environments to the latest VMware patches ESXi 6.0 U1b (build 3380124), that has just come out. As you do usually when there is a new hypervisor build you upgrade VMware Tools. Well that proved to be a big problem for my VM templates that I use to provision new systems. But I’ve got a workaround.
As soon as VMware Tools is updated on any templates you will no longer be able to clone those templates. If you’ve updated any templates with the version of VMware Tools that comes with ESXi 6.0 U1b then you need to uninstall it and reinstall the prior version that came with ESXi 6.0 build 3247720. After the couple of reboots that you have to go through with an uninstall and reinstall of VMware Tools you will find that you can now clone VM’s and have them automatically customized. I ran into this problem on Windows 2008 R2 Server. So I know it will impact this guest OS. I haven’t tested other OS’s yet, but others could be impacted. I’ve logged a support call with VMware to address this problem. In the meantime, the workaround is fine. The Official VMware KB Article 2142982 explains the situation.
[Updated 14/01/2016] After further testing I have narrowed down the problem area to new installs where the complete option is selected, and any upgrades where the complete options was previously selected, or where the VMCI / NSX Guest Introspection Driver is included. I have been able to successfully clone from a new VM Image that has had a fresh install of Windows 2008 R2 and VMware Tools without the VMCI / NSX Guest Introspection Driver, or where VMware Tools was installed twice / installed and repaired on the same VM, when the complete option was previously selected. This seems to be similar to what other of you have also reported. I have completed the Upgrade Scenario testing as well and confirmed that after an upgrade, if the complete install option was previously selected the VM will not clone due to the same VMCI driver problem. If VMCI driver is removed by running VMware Tools Install again and selecting Modify and unselecting VMCI, then you will be able to close the VM.
This update just in from VMware Support “VMware Engineering have confirmed that the issue is dependent on the install/upgrade sequence. Specifically, the issue is aligned to the version of deploypkg.dll in the vmtools package. GSS and Engineering are mapping the ESX and vmtools update versions to the deploypkg.dll versions to confirm which upgrade sequences are problematic. A KB article will be published once this information is finalised.“
Thanks to VMware Support for getting to this stage very quickly. The VMware KB Article 2142982 has now been published.
Final Word
I guess someone has to take the risk and patch their systems to the latest versions first, especially as these were security patches with a critical severity. Fortunately like all good IT environments I only did my test systems first. This is the whole point of having infrastructure test systems. You can test infrastructure hardware and infrastructure software changes first before putting them into production. The old saying goes that software eventually works and hardware eventually fails, but these days a lot of your hardware is also software, especially in a virtualized software defined datacenter. It pays to have appropriate test systems and test plans to mitigate the risks associated with software updates and changes of all types, including infrastructure software. Thanks to all of you in the community that contributed to this effort and commented on this blog post. I have been relaying your feedback during my discussions with VMware Support.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
After a lot of hard work and hundreds of hours of preparation the journey to VMware Certified Design Expert (VCDX) certification is finally complete for Nick Bowie from VMware and Nutanix partner ViFX. Nick was the first candidate to submit and defend a VCDX design based on Nutanix technology and has become only the 2nd VCDX in New Zealand. It’s a great achievement, and it was great to be able to help Nick with his journey and to have one more VCDX in such a small country. I caught up with Nick to ask him a few questions about his experience and where to from here. Hopefully this will be useful for other candidates.
Firstly I’d like to congratulate Nick on his achievement. VCDX is the pinnacle of VMware certifications. It is a solution architecture certification that encompasses many supporting technologies and you need to be expert in multiple areas to a great level of depth, as well as being able to translate business requirements into a successful solution.
Here is what Nick had to say about his VCDX experience:
Q: What does it feel like to be only the 2nd VCDX in New Zealand, and the only one working for a NZ based VMware partner?
A: It’s pretty intense 
I told you I’d do it!
Q: What made you decide to pursue VCDX?
A: This was a personal goal of mine for quite some time. Back in 2009, I had the opportunity to focus on VMware virtualisation when I was selected to work on the virtualisation stream of a data centre migration project, bringing in ESXi 3.5 and SRM into the mission-critical space of the company I worked for. This really turned my professional life around and gave me a career direction. It re-ignited my interest in technology and fuelled my desire to reach that expert level. Once I started working at ViFX it changed from a possibility to a reality – the team here are truly excellent and inspiring.
Author: I think I remember this project :).
Q: How long did it take you to prepare for VCDX, including background experience and the actual VCDX process itself?
A: If I count starting at the VCP5-DCV – a long time! I achieved VCAP5-DCA in July 2013, and shortly after joining ViFX I achieved the VCAP5-DCD certification in December the same year. Joining ViFX was a key contributor, as I gained a lot of exposure after coming out of the customer role and started delivering solutions. The project I chose to base the VCDX submission on began around March 2014, and ran longer than expected, so I held back on submitting for the 2014 October round. Without the opportunity to defend at PEX I had everything prepared for the April submission/June defence round. After submitting my application I took a quick segue and achieved VCP-NV, but once I found out I had passed the application phase I immediately joined a study group with fellow candidates for proper focus. We got together three nights a week between Monday-Friday for an hour at a time, and at least once a weekend for two hours right up until the defence.
Q: What are some of the key factors that you think helped you succeed at VCDX on your first attempt?
A: Easily number one was having an understanding and patient wife 
Up until the document submission I had a few looks as I sat down to work on design tweaks late into the night, but once I had the invitation to defend she gave me the time to give it my all. I also think patience on my part had a lot to do with it, taking my time to really work on the document set and not rushing it for an earlier defence opportunity. The project it was based on had some challenges, which really helped validate my design decision process and the conclusions I had come to. I also can’t ignore the guidance I had from my mentor, Josh Odgers. He gave nothing away directly, but would give me a nudge here or there that would each evolve into 20-or-so more pages worth of detail once I realised what more was needed. That process helped immensely.
Q: Was your employer supportive and how do you think it will change your role now that you have VCDX?
A: ViFX were immensely supportive. I had let it be known that I wanted to achieve this when I started, and with a particular project in mind and they gave me the opportunity to do so. I’m not sure what may change with regards to my role (it’s only been a week!), but I hope it will help further differentiate us in the market. We have some serious talent here – I suspect we’ll have NZ’s 3rd VCDX soon enough.
Q: How did you choose your design and was there anything unique about it that you think helped you succeed?
A: One of the reasons I wanted to work for ViFX was that they saw the future in Hyper Converged Infrastructure early on, and the architecture and deviation from what I had previously managed really caught my interest. When I first saw Steven Poitras build up a Nutanix infrastructure in 15 minutes, I asked myself “why aren’t we doing this?”, so it was definitely a technology set I was excited about. Outside of pilot/POC’s, it was my first real Nutanix and large-scale Horizon View design, which might seem like I was biting off more than I could chew – I could have selected a BC/DR or vBCA Oracle re-platforming project, but I felt particularly invested in this one and the platform selection made if quite different. I started the whole process with the VCDX framework in mind and this really helped. I recommend anyone looking to undertake this challenge similarly considers beginning a design with the goal in mind rather than taking an older design and retro-fitting it to the mould as some have blogged about, as although I had some re-work to do it didn’t result in large-scale changes and continuity errors.
Q: You are the first VCDX to successfully defend a design based on Nutanix technology, how do you think that impacted the defence?
A: Not having exposure to the VCDX scoring rubric, I’m not sure I can say. I’ll say this though: the panellists seriously did their homework and I don’t think the design being on Nutanix made the defence process any easier! In the end, understanding all the constraints and risks I had meant my decisions were the right ones to deliver to the requirements. That would be true regardless of the platform type.
Q: If you had to do this all over again, what if anything would you do differently and what advice would you give to other candidates?
A: I’m not sure I would do anything differently. Even though it was stressful at times, I was successful on my first attempt and learned a lot in the process. My advice would be this: Understand the framework and understand the blueprint. If you have the opportunity to start a design with this in mind, regardless of ‘scale’ or perceived complexity, it will make the whole process smoother. Care about the customer outcome and the results will be in your favour.
Q: After a well-deserved break, what’s next for you?
A: I’m finding it hard to wind down after the experience The family is definitely appreciative of my new-found free time. I’ve still got a lot to learn I’ll keep working hard, learning and applying myself.
Q: Was going through VCDX worth it and how did your company respond when they heard the news?
A: Absolutely! Shortly after I received the congratulatory email (and pinched myself a couple of times to make sure it was real), I let the team know and immediately received excited phone calls from the company Directors and team mates who were all immensely proud.
Author: Not to mention the congratulations you got from the rest of the country as well, myself included. Well done!
Q: Are you going to attempt the Nutanix Platform Expert (NPX), and what are your thoughts about its value?
A: Eventually, yes 
There are some very compelling capabilities on display through the infrastructure automation that Nutanix have worked hard on, both through the Acropolis hypervisor and the ability to support a multi-hypervisor, distributed platform. The structure of the NPX is pretty daunting, and I’ve got a lot of learning to do before I’ll be ready. But that’s great for me – if I’m not learning something I get bored As to its value: with the prolific acceleration of cloud platform adoption, we as architects have the requirement to broaden our knowledge beyond the vSphere hypervisor to meet the requirements of particular use-cases. So displaying expertise across multiple hypervisors will definitely be of value.
Final Word
It’s great to have another VCDX from New Zealand and I’m looking forward to many more. I and the rest of the VCDX community are happy to help anyone interested in achieving this certification as it raises the bar for enterprise architecture expertise and allows better solutions to be delivered to customers. In a world where end to end business solutions are complex regardless of the underlying platform chosen, you need experience architects to drive the solutions design. VCDX helps build better outcomes for customers and better loyalty and revenue for partners. With the Nutanix Platform Expert (NPX) Certification you can now achieve an expert level certification across multiple hypervisors and application runtimes, building the swiss army knife of enterprise architecture for a hybrid cloud software defined datacenter world.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
While a lot of people (me included) are excited about the technical speeds and feeds of the vSphere 6 launch, there is something much more fundamentally important about this release. Some technical highlights include 64 node clusters, 8000vm’s per cluster, 480 pCPU’s & 12TB RAM & 2048 VM’s per Host, 128 vCPU & 4TB RAM per VM support, SMP FT (up to 4 vCPU FT), enhancements to NIOC, VVOLs, SIOC enhancements etc, and much more. The reason this release is more fundamentally important though is related to the same reason that Amazon with AWS went from nothing to cloud leader. It’s not just about the technology, but what the technology enables, changing the business model, reducing friction, enabling flexibility. What might seem like a relatively small feature on the surface has the potential to change the landscape in hybrid cloud SDDC. If you’d like to know more about this, and all of the goodness coming as part of the launch of vSphere 6, keep reading.
VMware is about to release the latest version of the flagship vSphere product in what I predict will be a defining moment for the Mobile / Cloud Era. For the first time you will be able to live migrate, without any disruption, between private cloud datacenters, to public cloud, and over long distance, a true hybrid cloud and software defined datacenter. You will be able to implement improved quality of service for all applications with additional SLA guarantees, and scale to unprecedented levels. All while reducing management overheads and complexity across the entire ecosystem. With the policies following the virtual machines and virtual applications regardless of where they are physically located.
This release has been baking for a while and for good reason. There is a big commitment to product quality, which was evidenced by the first ever public beta for VMware vSphere. This is a major release, and is well deserving of the 6.0 version number. A lot of hard work has gone into this release by thousands of people. I was able to test a lot of the functionality during the beta and it was great to be able to contribute to the product.
So why do I think this is such a defining moment? The world is changing with the massive explosion of mobile smart phones and the applications that support them. Billions of users are now demanding their applications wherever and whenever they want. So not only are the users mobile, their applications need to be. The applications need to be able to scale massively and on demand, and move to wherever it makes sense.
Previously migrating workloads from a private cloud or private SDDC to a cloud provider and to support a hybrid cloud required the systems being migrated to be shut down. You could migrate templates and power them on and update load balancer records, but that’s not quite the same as being able to dynamically live migrate any workload from your datacenter to a cloud without any downtime or disruption, and across long distances. If you really wanted to deliver cloud workloads and mobile workloads at scale, they had to be written for a particular cloud environment. Then you are stuck in a hotel California, where you can check out, but can never leave. This is the fundamental difference, and the fundamental technical change that is potentially enabled by vSphere 6, which in turn will deliver business and commercial disruption to current models.
The enhancements to VMware vMotion have the potential to change the way organisations run their datacenters, applications and interact with cloud service providers. It is conceivably possible to migrate workloads between different clouds on demand, based on various business rules and policies, provided they are based on vSphere 6.
So where does the comparison to Amazon and AWS come from? The reason I believe AWS became successful it not because of technology, it’s because it changed the economic and business model of consuming infrastructure. It reduced the friction, made everything on demand, and delivered to development and applications teams, in a way that was transparent. With the VMware vMotion enhancements allowing Cross vCenter vMotion and Long Distance vMotion, Cloud Service Providers can provide even less friction, on demand, run anywhere appropriate type of service. Some of the tyrannies of the network and live migration are being demolished. This again can change the way infrastructure is consumed and make it easier for app teams to deliver. But this needs to be blended with a commercial construct that also supports it.
At VMworld in 2014 Bill Fathers, Father of vCloud Air, reported that some 6% of workloads were running in Cloud environments. I believe part of the reason is because of the difficulty in migrating workloads to a Cloud, and between different Clouds, without disruption, and without having to change the underlying apps. With the changes that VMware is starting to deliver from vSphere 6, conceivably this could rapidly change the adoption of VMware compatible Clouds. There is still much to do in terms of the Network, which is still one of the barriers to Cloud, but this will go a long way. Soon you will be scaling workloads on demand to support the billions of mobile users and migrating those workloads to the cloud of choice that makes sense, almost anywhere in the world.
I was recently at the Singapore VMUG User Conference and listening to one of my colleagues, Scott Drummonds, talk about Cloud and locality. Locality is important because there are orders of magnitude computational difference the further the users are from their applications and data. How this relates back to the VMware vSphere 6.0 launch and vMotion Across vCenter and Long Distance is that it will now be possible to migrate workloads on demand closer to where the users are, especially as we start to see Cloud services become more local, and more miniaturised over time.
Final Word
At first glance vMotion Across vCenters and Long Distance vMotion may not seem that revolutionary. When you put this all in the context of the Hybrid Cloud and Software-Defined Datacenter that VMware has been building towards for over 5 years it is easier to see that this actually delivers a fundamental change in the way infrastructure resources can be used. It won’t be long before the live migration of VM’s is happening as depicted in the image above. What new possibilities will this open up for businesses? What impacts will this have on data sovereignty? Your thoughts and comments are appreciated.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Another VMworld event is over and it’s hard to believe it’s been a whole 12 months since the last one. Certainly during the keynotes there was a lot of coverage about what VMware has achieved over the last 12 months and it is impressive especially in the end user computing and hybrid cloud spaces. But overall I felt that VMworld USA 2014 lacked some of the sparkle of last year. But I guess it’s hard to top last year considering it was the 10th anniversary. This year seemed much more about building a solid foundation for a software defined datacenter, a software defined enterprise and a hybrid cloud model integrating applications with infrastructure, providing ability and flexibility, but without compromise. Although attendance was flat or a little down on last year the breakout sessions were packed, right up to the last session on Thursday. Instead of having our heads in the clouds this year it was all about the vCloud Air, and we vRealized the product naming is about to be changing. So lets dive into what I think are some of the highlights.
My VMworld started on Sunday with a Nutanix sponsored VCDX study group. Nutanix is a big supporter of the VCDX program for the entire community. The study group was put on for candidates that wanted to know more about the VCDX process and practice the design and troubleshooting scenarios. It was completely vendor agnostic, and it needs to stay that way. Nutanix understands that the only way sponsoring a VCDX study group can be of value is if the content is vendor agnostic and covers a wide range of topics. There were many VCDX helping in the room and giving advice from across many companies. This really is what the community is all about. Everyone helping each other.
Then I moved on to opening acts at VMunderground that was put on by vBrownBag. I was on the storage panel and it was a good discussion around Virtual Volumes, Hyper Convergence and Flash. I even agreed with a traditional SAN vendor that hyper converged appliances will not help SuperDomes and Mainframes, but then again I can always migrate the workloads and processes off those systems, and the Unix mid range systems as well, to a Hyper Converged world. SuperDomes, Mainframes and Unix systems is where the legacy SAN technology will stay for the foreseeable future and it will be a decline over a number of years, just like we’ve seen with the traditional big iron systems themselves. The move away from traditional SAN for x86 connected environments isn’t going to happen over night, but it’s a trend that is starting to take hold, but honestly it’s not even scratching the surface of the potential opportunity yet. The announcements from VMware and EMC around their hyper-converged offerings are just more validation of that. Flash is definitely the way of the future, and it opens up things that were previously not possible. I have a section on flash technology in the storage chapter of Virtualizing SQL Server with VMware.
There were a number of VMware announcements during the keynotes that are worth mentioning. But before I do I have to get something off my chest. vRealise is the worst name ever thought of for anything. My initial reaction to the new name for VMware’s hybrid cloud, vCloud Air was somewhat similar, but at least Air has a cool ring to it, like iPad Air for example. vRealize, just NO! I feel sorry for the sales team who have to try and sell that now. Ok, rant over. The overall themes about this being a brave new world and requiring bravery from all of the customers and the community participants was interesting. I’ve been doing virtualization for a very long time and even for business critical apps it’s a very safe bet. But SDDC and the Software Defined Enterprise going to further reduce silos and this will require some organisational changes and maturity. This is really where the bravery comes in, most of the challenges are not technical.
Two major overall themes were used during the keynotes. Firstly – “Compatibility, Compliance, Choice”, and secondly the “Power of &”. VMware has done a great job of building a partner ecosystem across a number of technologies, including the vCloud Air Network, which has 3900 partners, and the broader ecosystem around the hypervisor and NSX. This is where compatibility, compliance and choice really comes in. Seamless compatibility, compliance with regulatory and industry requirements, and choice of multiple partners and technologies. This is then extended to the OpenStack, NSX and Containers, which can run extremely well in a VMware environment, and this is the Power of &. Have your containers without compromise. Have your OpenStack on a platform easily fit on top of VMware vSphere.
By far the biggest highlight was meeting a lot of people who regularly read my blog and have benefited from the work that I and others in the community have done over the years. This is why we keep doing it. Because it makes a difference. It was also great to meet a lot of people who had bought Virtualizing SQL Server with VMware: Doing IT Right, and had got a lot out of it too. My co-authors, Michael Corey, Jeff Szastak and I were blown away by the stories that were relayed to us about how the book had helped people, especially when it was being used to explain to DBA’s how virtualization works and that SQL is a great candidate for virtualization. The book was so popular that it actually sold out at VMworld, and we had a lot of people come up to us during the meet the authors session and book signing.
Here is a photo of my co-authors and I with the happy customer who purchased the very last copy of our book at VMworld.
Shortly after the above photo was Michael Corey and I recorded an interview with VMworld TV’s Eric Sloof regarding virtualizing SQL Server Databases on VMware.
I was lucky again this year to present a session that was included in the top 10 sessions of VMworld for the second consecutive year – VMworld 2014 SDDC1600 Art of IT Infrastructure Design The Way of the VCDX Panel.
Final Word
It was another great VMworld and a very successful VMworld. I’m very much looking forward to next year. Hopefully we’ll see the return of the Monster VM sessions and some other business critical apps sessions from me in next years VMworld.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Although my Monster VM panel was in the top 10 sessions of VMworld 2013 and we did a Monster VM and Business Critical Apps panel for TAM day this year neither session will be included at VMworld in the USA or Europe. But not to worry. There is plenty of great content at VMworld for everyone to enjoy and I’ll be there to talk about Monster VM’s on vSphere as always, but mostly at the Nutanix Booth #1535. This doesn’t mean you have to miss out on all the Monster VM goodness though, you can grab it all online right now, for free, thanks to VMware opening up the session catalog to some great sessions online through VMworld TV. So below I include two great panel discussions from VMworld 2013 that you can review now, and get a better understanding of how to virtualize critical apps and Monster VM’s.
This first video is from TAM day and includes myself and a number of leading experts discussing virtualization of business critical apps and taking questions from the audience. The topics covered are broad from Microsoft apps such as SQL, Exchange, AD to Unix applications and migrations from Unix systems to vSphere, Oracle, SAP and Java.
This second video is from my Monster VM Software Defined Datacenter Design Panel, which was one of the top 10 sessions for VMworld 2013. We cover sizing, tuning, and scaling, as well as answering all the questions from the audience. When you get 5 VCDX in the room talking about performance and monster VM’s at the same time it’s got to be good.
Final Word
Even though we don’t have a Monster VM panel at VMworld this year there is no reason you should miss out. Come and visit the Nutanix Booth #1535 and I’d be happy to discuss everything Monster VM related, and also how the Nutanix platform can make it uncompromisingly simple. I will have copies of the book Virtualizing SQL Server with VMware: Doing IT Right (VMware Press), that I co-authored with Michael Corey and Jeff Szastak that we’ll be signing. If you can’t make VMworld in the USA and you’d like to go to VMworld in Barcelona then you might be in luck. VMTurbo has started a competition and will be giving away free tickets to VMworld Europe. VMTurbo started the vendor trend to give away free tickets to the VMworld events and I think it’s a great thing. As always your feedback and comments appreciated.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Back in 2010 I was helping a large company troubleshoot their virtualized SAP environment, which was experiencing instability and performance problems. One thing we noticed was that the buffers on the NIC’s were periodically overloading due to the large amount of small packets. This was on vSphere 4.0, with Windows 2003 64bit OS at the time and using VMXNET3. Unfortunately at that stage the VMXNET3 driver for Windows didn’t support increasing the send or receive buffers and as a result we had to switch over to E1000 and increase the TX and RX buffers, which resolved the problem (in addition to adding memory reservations to the VM’s). However since vSphere 4.1 it has been possible to modify the buffers in VMXNET3 to resolve these sorts of issues. I have been experiencing this myself in my home lab and have as a result modified the buffers, but it appears I may not be alone in experiencing this.
I thought this was just something I had done in my lab environment. But after reading Michael White’s Newsletter and the VMware KB 2039495 – Large packet loss at the guest OS level on the VMXNET3 vNIC in ESXi 5.x / 4.x, it appears I’m not alone in this. Fortunately it is easy to make the necessary modifications to the buffers and resolve the majority of the packet loss issues as follows:
In my environment I’ve also set my Large RX Buffers to 8192 and my TX Ring Size to 4096.
If you suspect that your virtual machines may be dropping packets or losing packets then you should consider adjusting the RX and TX buffers. This may well lead to increased performance and more importantly application stability. Sometimes in addition to increasing the buffers you may need to reserve the memory if it’s a very important app. This will ensure it can receive the resources it needs.
Final Word
In most cased the default settings are fine. In some cases there are some adjustments needed. This is one of the cases, if you are experiencing this problem, where adjustments are needed. There is no patch as such to address this problem at this time. But VMware will hopefully make improvements to its drivers and IP stack in future versions of vSphere.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.