From Virtualization Management to Private Cloud – Lab Test Log: Upgrading Hyper-V cluster on IBM BladeCenter to SCVMM 2012

IMHO the only way to provide relevant coverage of vendor capabilities on Virtualizationmatrix.com is through hands-on experience with the products in client projects or lab test – finding the  time to always document this in detail is however a challenge.

So initially this “test log” was not intended to be a blog but I decided to post it after running a few colleagues through my experience with VMM 2012 and they asked me to share it with others.

So here it goes …

Scenario:

Migrate one of our Hyper-V lab clusters from VMM 2008 R2 to VMM 2012 and evolve the managed virtualization environment into a private cloud that facilitates controlled Self-Service access for visitors (remote demo users and developers).

• Upgrade the VMM management of our existing 2-node Hyper-V cluster “HypVCluster” to VMM 2012 (RC) by installing a NEW instance of VMM (not an upgrade of the existing instance)
• The original HypVCluster consists of 2 x W2K8R2 hosts “Hyperv1” and “Hyperv2” on IBM blades HS21XM in IBM BladeCenter-H chassis connected to IBM Nseries7800 storage (Netapp FAS6070)

Architectural Comments:

• The VMM appliance (downloadable vhd) will be used (It contains the following components: Eval of Windows Server 2008 R2 Standard SP1, eval of SQL Server 2008 R2, eval of VMM for System Center 2012, Windows Automated Installation Kit (AIK) for Windows 7, Microsoft Storage Management Service, Web Deployment Tool
• The new SCVMM instance will reside in a highly available virtual machine hosted on this Hyper-V cluster. The new high availability feature for SCVMM (where you can install SCVMM as “cluster aware application” on each node will not be used (no “hard” technical reason, mainly to keep VMM “portable” as vhd in our test environment).
• As the VMM appliance is based on an evaluation version of W2K8R2 Standard and an evaluation version of SQL2008R2 we will activate Windows to make it permanent and use an existing (full) version of SQL2008R2 for the DB (installed on old SCVMM server: SCVMMR2.eebc.dom) to avoid any expirations.

Preparation:

• Downloads: Download VMM 2012 appliance from HERE
• Accounts: Even if you normally may not bother with dedicated accounts in test environments – DO create a dedicated VMM “service account” in your domain (do not use your “default” admin account as various steps will check that you don’t use it and hard stops and problems can occur if you do – more details are covered in the VMM documentation http://technet.microsoft.com/en-us/library/gg697600.aspx)
  • Ensure to make the VMM service domain account a member of local admins group on your new VMM Server (vhd)
  • If you use an external SQL server, ensure you have an authorized account for the DB creation
  • Add the VMM service account (using SQL studio) as e.g. “sysadmin” (otherwise DB creation during setup will fail)

Get Started:

• Extract the download vhd to the existing SCVMM 2008 library
• Create virtual machine using the extracted vhd file
• Start virtual machine, run through the initial Windows setup, adjust domain and network settings to integrate the vm into your environment.
• Start VMM setup by clicking the existing icon on the desktop of the VMM server
• Ensure to specify the created “VMM service account” (I’d suggest to add it as “run as” account in VMM to allow for easy future re-use)
• Specify the SQL instance you want to use (in our case “SCVMMR2” but it could be in your case the bundled evaluation version)
• Check that the setup finished without any issues – you will now have the VMM start icon on your desktop – start the VMM console.
• Get a feel for the GUI, explore the new Office style “ribbon” menu but do not start adding hosts or clusters yet

Configure the Fabric

Before we upgrade and add our hosts we will prepare the “fabric” environment we want to add the hosts to.
Fabric is the new collective term for servers, network and storage managed by VMM.

Host Groups:

Host groups are hierarchical folder structures to group virtual machine hosts in meaningful ways, often based on physical site location and resource allocation.

• From the Fabric pane start off by creating a hierarchical folder structure that reflects your logical infrastructure layout (e.g. by datacentre locations and sub-locations).
• Review the available properties you can set on the host group level, like “Placement Rules”, “Network” and “Storage” allocations
  Note: Resources typically need to be allocated on the host group level before they can be assigned to hosts and clusters so familiarize yourself with these options (right-click on the host group and select properties)

Library:

If unfamiliar with the VMM library simply think of it is a repository for any resources you might need to access such as virtual hard disks, virtual floppy disks, ISO images and application packages (new in SCVMM 2012) as well as virtual machine and service templates and profiles that reside in the VMM database.
The library in VMM 2012 has been enhanced to support additional resource types and cloud structures. You can also store driver files that are used during the bare-metal deployment of a Hyper-V host and custom resources that would normally not be recognized as VMM reources (such as scripts)

• Explore the library view of the default library (installed during setup), including the templates and profiles as well as “equivalent objects” (you can now mark identical objects as equivalent – that allows VMM to have multiple sources for the same object in order to decrease dependency on physical resources and consider locality)
  Note: VMM has no integrated mechanism to syncronise changes to the equivalent objects (you need to ensure e.g. replication or manual copy after changes)
• Optional: Install a second library server (just to test some of the new functions like marking objects as equivalent for location sensitive deployments)- we installed a second library server: “w2k8r2-trial-0.eebc.dom”
  Note:  If you plan to implement a private cloud I strongly suggest to review the topic “Implementing a private cloud” in the VMM documentation before creating your final library structure as organisations might require dedicated library resources (to ensure access to THEIR resources)

Observed problem: Adding second library server did not reliably add all the default resources (skipped some of the default categories), circumvented by manually copying remaining files from one library to the new one)

Comment: There is no integrated functionality to view/monitor capacity information on the library servers i.e. to avoid running out of space or make decisions where to store images)

Set up your Logical Networks

• Review the global network settings -> Settings Pane -> General -> Network Settings (they e.g. determine the behaviour of default creation and association of virtual and logical networks when none exist on certain components)
• Create logical networks, think of them as descriptions (virtual switches) that reflect your external network structure. This is a new feature, previously you could only create virtual networks.

The structure is obviously entirely depending on your environment. I started out creating a new set of logical networks:

• EEBC Lab Network (DHCP on 192.168.x.x)
• Test Network for static IP pool (managed by VMM): 192.168.90.1-10 (with 192.168.90.9, 192.168.90.10 for IVP reserved for Load Balancer for future use)

Note: We will later verify the assignment of the logical networks to hosts by mapping them to the physical adapters (once the hosts have been added). As part of logical network creation, you can create network sites to define the VLANs, IP subnets that are associated with the logical network in each physical location.

Comments:

• Reviewing the structure of the logical networks and the relationship to the hierarchical structure (e.g. host groups) is not easily done, there is no good view in VMM to get an “at a glance” view of the architectural network structure.
• Try to remember the difference between logical and virtual networks. VIRTUAL networks are “virtual switches” on the host, providing connections to vms (typically vms to physical host NICs or private inter-vm connectivity), while logical networks are descriptions of external networks (“switches” connecting the host NICs to external networks) with properties like DHCP v IP pool, VLANs etc …

Setting up Storage

I will deploy the new SMI-S based storage management that allows the admin to perform common storage activities from the VMM GUI (e.g. create LUNs, assign storage etc) and also allows to offload certain storage functions directly to the storage array (if you are familiar with vSphere then this is a similar approach to – but not identical – to VMware’s VAAI/VASA approach).

Note: You need a supported storage array (array: storage system) to integrate VMM with SMI-S but you can of course use standard storage using non-SMI-S based storage allocation but you won’t be able to manage them through VMM.

Comment: You can work with existing disk resources or create new ones.Depending on the storage you might have to perform some actions using the native array GUI

• In our case I created a new “SCVMM” aggregate on the Nseries using array GUI
• Downloaded and install your SMI-S provider – in or case I installed the Nseries (Netapp) SMI-S provider and installed it on SCVMM server (could obviously be on another server)
• Added hostname of SMI-S provider system to Providers under Storage (without SSL), the array was discovered OK and new aggregate “SCVMM” was listed

Observation: Please note that there can be delays in updating the array status in VMM after VMM driven configuration updates, ensure to “refresh” before performing new actions if problems occur Fabric Pane -> Storage -> Providers

• Selected the disk resources you want to manage through, I selected the “SCVMM” aggregate to be managed by VMM
• I also tested the array interaction by creating and deleting a test LUN through SCVMM and verified the activities through the Nseries GUI – all successful.

OK, we have prepared the fabric environment, now we have our hierarchical folder structure, added library servers to store images, created logical networks and prepared the storage.
Let’s add the hosts.

Adding Host Resources

Preparing Hosts

• If you have not already done so add the storage multi-path (MPIO) feature to each host before adding the host/cluster to VMM. MPIO will then be configured automatically when adding hosts to VMM.
• As the IBM blades are configured with Broadcom NICs I installed and configured the BASP failover driver with the defaults.
  Note: If the hosts are already configured for Hyper-V (as in our case) you will have to un-associate the NIC from the hyper-v virtual switch as the BASP installation will otherwise not be able to continue with the following error “The selected Adapter is bound to Hyper-V Virtual Network …”:

Re-associated logical network (hyper-v switch) with the team (rather than a physical adapter) as shown

Note: If you receive a warning 26179 when adding hosts/cluster “Couldn’t enable multi-path i/o for known storage arrays xxx” you have either not or incorrectly configured multi-path on the hosts before adding them. VMM will attempt to configure MPIO when adding host. Correct the MPIO settings before continuing.

• If the hosts were part of a SCVMM 2008 cluster remove the existing SCVMM agents from the host before adding the host to the new SCVMM instance

Add Hosts/Cluster

• From the Fabric Pane, select the appropriate host group and add the cluster (you can specify a cluster node and it will pick up the existing cluster.
• Your cluster should now be imported into the new VMM instance and any existing virtual machines should be visible and operational.

Adding Storage to the Cluster

The storage allocation to hosts can be slightly confusing to the new user.

1. Be sure that you can see the storage array and any resources on the array you want to use from VMM
2. You first need to select to managethe relevant “storage pools” (aggregates in our case): Storage ->arrays, select your array -> properties, select the storage pool(s).
   1. As part of this you should create storage classification to describe the properties (e.g. if you have different storage tiers)
   2. Then “allocate” storage to a host group (folder containing hosts or clusters): properties -> storage
      You can allocate existing storage pools, existing (unmapped) LUNs or create new LUNS (free space on existing pool) and allocate them
   3. Then (and only then) you can “assign” storage to the cluster: select the cluster -> properties
      Note: you can add (assign) LUNs as “available storage” (think “normal” LUN) or “shared volumes” (think “cluster shared volumes”) – for what it’s worth – I don’t like the naming convention here nor the way of allocation
   4. Feel free to convert between CSV and “normal” LUNs – I selected all shared storage as CSV for the obvious advantages (there aren’t many reasons why you’d want to have “normal” LUNs in a cluster scenario)

In our case we had two existing cluster LUNs (Quorum + 1x CSV) on an existing aggregate (30 spindles) from the initial SCVMM2008R2 managed cluster, and as mentioned above I added an aggregate “SCVMM” with 2 additional LUNs (5 spindles) on the Nseries

• We then selected both aggregates (original + new) to be managed by VMM and created 2 VMM storage classifications to reflect the performance differences (spindles) as shown below

Comment: There is no feature to exclude LUNs of a managed Storage Pool from the management (in our case we added an aggregate that also contains LUNs not used for the VMM environment). This distorts the capacity information (as unrelated LUNs are included) and introduce potential admin errors (e.g. can delete unrelated LUNs).

Observation: I ensured that the managed disk pool is allocated to host group but any attempt to add the new storage pool (or LUNs within the pool) to the cluster failed with error 26184 “The Storage Group existing for xxx doesn’t match storage group setting at array xxx”

Resolution: As VMM will create relevant LUN-to-host mappings at this point, any existing conflicting configurations may cause problems. Use the native array GUI to remove invalid old mappings for the HBAs/hosts (in our case in the “initiator” section of the Nseries GUI). After deleting invalid old mappings the process worked.

• As expected, after fixing the “ghost mappings” the assignment of available storage created automatically the respective LUN to host (initiator) mappings on the Nseries storage.

• As the storage was assigned to the cluster it also created automatically the cluster resources (as seen in failover manager)

• I then converted the volumes to CSVs – no problems – the CVS were created automatically and made available to the cluster nodes.

 Verify Host Network Config

•  Again, verify that MPIO is configured correctly: Admin Tools -> MPIO, if you have added MPIO before adding the hosts (or configured MPIO manually correctly) you should see something like the below
• Perform the association of logical network association with the hosts: Select host -> properties -> Hardware -> select NIC -> check that the relevant logical networks are connected

Observation: Adding an additional host to an existing cluster fails with error 25343: “No network adapter found on host xxx that matches cluster virtual network xxx”. The error refers to a miss-match with the VIRTUAL network. However the recommended action points out that you should set the LOGICAL network on the NIC.

 Therefore do NOT just try to create a matching VIRTUAL network like below on the host:

Instead as described above, select the host before adding it to the cluster -> properties ->hardware -> NIC and ensure that the associated logical network is connected correctly.

Configuring Dynamic Optimization and Power Optimization

Dynamic Optimisation for Hyper-V (again, if you are familiar with vSphere think “DRS”) is now very easy to set up. Forget the extremely awkward SCOM/PRO dependency for even basic optimization in SCVMM 2008.

• In the properties for the host group containing the cluster, enable Dynamic Optimization with the appropriate settings – literally nothing else is required at this stage …
• 10 mins later first “optimisation” took place:

Note: Power optimization requires direct out of band BMC access for IPMI control (i.e. try to ping the BMC IP address from the VMM server … since the BladeCenter chassis uses central management of the blades through its management module it will not work on this setup.

VMM Updates (WUS)

VMM now supports compliance scanning and remediation of the fabric servers (again, think “VMware Update Manager” in vSphere). VMM supports orchestrated updates of Hyper-V host clusters (VMM places one cluster node at a time in maintenance mode and then installs updates) while vms are being live migrated. If the cluster does not support live migration, VMM saves state for the virtual machines

We will install a dedicated WUS server for VMM (installed on the VMM server). You can also use an existing WUS server in conjunction with SCCM.

• Downloaded WUS from Windows Server Update Services 3.0 SP2
• Installed prerequisites:
  • Microsoft Report Viewer Redistributable 2008 http://www.microsoft.com/download/en/details.aspx?id=6576
  • IIS with
    • ASP.NET
    • Windows Authentication
    • Dynamic Content Compression
    • IIS 6 Management Compatibility
• Installed WUS with following options:
  • Full server installation including Administration Console
  • Create a Windows Server Update Services SP2 Web site
  • Selected relevant settings regarding updates (limited languages and selected relevant W2k8R2 updates only)
• WUS console showed that initial sync was successful

• Added WUS server to VMM server (fabric -> add resources) – port 8530 – no problems
• Reviewed the default baselines and created a new test baseline – added critical and security baselines to “all hosts” host group

Comment: There seems to be no intuitive method of filtering/selecting updates at this stage and the baselines are not continuously maintained (e.g. you sorted all updates  by “critical” and created an “all critical updates” baseline. That means that critical updates released in the future are not automatically added to this baseline)

• Scanned all hosts for compliance:

Remediated the non-compliant server (if I had a non-compliant cluster then remediation would have put hosts into maintenance mode in round-robin before applying updates)- This is what you should see after the Remediation:

Comments: This all works and is straight forward but …

– No integrated WUS synchronization (to download new updates) – only “on-demand” (marketing term for “manual”) – No dynamic updates of baselines to include the new updates (i.e. by category “all critical”)

So in order to stay updated one needs to:

1)      Manually sync the WUS server (to download new updates)

2)      Manually update baselines to include the new (synced) updates

OK, so now we have added our hosts and associated them with storage, logical networks, enable Dynamic Optimization and configured updates for the hosts.

Our virtualization environment is basically configured and we could go ahead creating vms, templates and deploy workloads. However, what we really want is to create a private cloud ….

Private Cloud

I will assume that the reader is familiar with the concept of a private cloud. Essentially we want to create an environment that allows us not only to pool our underlying resources (which we have essentially done) but to enable shared Self-Service access for users from different organisations, delegate management without requiring users to ask the private cloud provider for administrative changes beyond increasing capacity and quotas as their needs change. While you can create private cloud from either Hyper-V hosts, VMware ESX hosts and Citrix XenServer hosts we will only u

Scenario:

We want to make the resources in the host group “ATS Lab” available through two private clouds:

• Private Clouds:

1. 1. Cloud 1: ATS Department
   2. Cloud 2: Visitors and Test/Dev

• Capacity:
  1. ATS Cloud with have unlimited capacity quotas on the underlying resources
  2. Visitor and Dev Cloud will have limitations on memory, storage and number of virtual machines
• Network:

1. 1. All will have access to the same logical network (DHCP)
   2. Only ATS will additionally be given a dedicated IP pool (fixed IPs)

• Storage Tiers:

1. 1. ATS: Gold
   2. Visitors: Silver

• Library:

1. 1. ATS: Both Library shares on SCVMMLibrary1
   2. Visitors: SCVMMLibrary2

Prepare Cloud Libraries 

Please spend some time to properly plan the library structure to accommodate multiple orgs/dptms

• Distinguish between read-only “catalogue resources and write-able “repositories” resources (store virtual machines)
• Create read-only library folder structures (not shares) on the library server(s) that allow dedicated folders (with unique paths for each “organsisation”) to store vms. You can see below that we created dedicated “write” folders on the same library server as the “read-only” library share but not within the share! (suggest to review the impact of user rights and folder structures in the documentation)

Note that (just as a test) in this example we have selected separate folders on the same server (Library1) for both orgs to store vms (while the read-only shares are dedicated to Library1 and Library2 respectively. This is by no means intended to be a “best practices” library setup.

Comment: A “reference library layout” in the GA VMM documentation would be useful – the library structure can be confusing given the different types of folders, shares and access requirements for the cloud libraries (in addition to the standard libraries)

Creating the “ATS Cloud”

From the “VMs and Services” Pane select “create  cloud”, then specify the cloud properties

Creating the “Visitors and Dev_Test” Cloud

• Verify that the clouds were successfully created from the VMs and Services Pan

Comment: The capacity settings have some inconsistencies and limitations:

• Danger of “over-committing” capacity
  • There is no way to guarantee resource – only limit/cap the usage
  • There is no warning when “overcommitting”, i.e. you can only have 36GB of physical RAM combined in the resource pool shared by two clouds but you can “limit” to e.g. 64GB on each cloud – no warning or visibility of how much of the resource has been “committed” (it’s not commited as such as it’s a “limit”)
• Values shown as “unlimited” – which is strictly speaking correct but meaningless i.e. how much is “unlimited”?

Configuring Self Service

Self-service users can deploy their virtual machines and services to private clouds.

• Role-level quotas on the self-service user role are used to allocate computing capacity and other storage within the cloud.
• Member-level quotas set individual limits for self-service user role members.

Self-service users can also create their own templates and profiles. The Author action for a self-service user role grants self-service users authoring rights. Users with authoring rights can create hardware profiles, guest operating system profiles, application profiles, SQL Server profiles, virtual machine templates, and service templates.

Preparation:

You typically create security group(s) in active directory and associate Self Service User Roles to these groups.

• In AD:
  • Created Security Groups “SelfService_ATS” and “SelfService_visitors”
  • Added “ATS1” and “Visitor1” as new test users to the respective groups

Creating the Self Service User roles in VMM

We will create two Self Service User Roles in order to test different levels of entitlements to the cloud environments.

• 1. ATS Self Service User (“unrestricted” access to both clouds)
  2. “Visitors Self Service User” (restricted access to visitors cloud (only) without “Author” rights and limited quota for max of 2 vms per user)
     (The Author right determines whether a user can create their own templates)

Create ATS Self Service User Role:

• • From the Settings Pane -> Create User Role
  • Add “SelfService_ATS “ role
  • Gave access to both private clouds (ATS and Visitors)
  • Granted all Self Service rights
  • Created and shared a folder for the user role data path (where SS users will be able to upload and share the physical resources that they use to create service templates and deploy services)
    • Ensure to give the user group associated with the role read and write access on the share
    • Add a library share for the path (e.g. \scvmm2012-vhd.eebc.domATS_DepartmentUser_Role_path
    • Add the user data path to the user role (properties)

Create “Visitors Self Service User” Role:

• From the Settings Pane -> Create User Role
• Added “SelfService_visitor “ role
• Gave access only to visitor cloud
• Granted all Self Service rights except Author
• Limit quotas as shown below
• Created and shared a folder for the user role data path (see above example)

Note: In order to test assigment and sharing of resources between user roles I subsequently created and added a vm template and a guest OS profile to the library and added them as available resources to the ATS Self Service User role only!

Observation: Capacity and Quota assignment is straight forward in VMM but viewing the (effective) allocations is not intuitive as the cloud overview does not seem to correctly reflect the assigned values.  Example:

• The “visitors” User role restricts the member to the following quotas:
  • the role (group) level is unrestricted
  • the member level is restricted (e.g. to 2 vms only) as show below

• However, logging in with as the self service user “visitor1” (which is member of the security group that is associated with the “visitor” user role does not display any limitations in “ Quota for visitor1” – see below:

Changing the role level quota to a restricted amount is however correctly reflected so the user is only able to see group-level quotas NOT user-level quotas (which would be more appropriate for Self Service Users in order to understand what is available to the particular user when logged in).

Logging in as Self Service User

We are now logging in as the respective Self Service Users to verify the correct resource assignment.
Note that you can concurrently log in as administrator and self service user(s) from the same system as shown below

• Log in as user “ATS1”
• Note that there is no Fabric Pane
• Library Pane: As expected we can see all cloud library resources (not the physical library servers)
• The assigned resources (guest profile as an example) are visible
  Note:
  • ATS1 can create new templates as we have given “author” right to the user role.
  • Pay attention to the context menu options. ATS1 can create templates as we have given the user role “author” rights.

• Now log in as Self Service user “visitor1”
• As expected we can see only the cloud library resources associated with the visitor’s cloud but not the other library resources.
• No resources (e.g. guest profiles as shown) are available yet as we have not assigned any to the visitors user role
  Note: Pay attention to the context menus – as we have NOT given the author right to the visitor user role there is no option to create a template (see limited menu options) – all working as expected …

Sharing Resources between Self Service Users

Finally we want to test the ability of VMM 2012 to share resources between Self Service Users. SS Users can either be entitled to resources through their user role or through object based sharing of resources if the user role “rights” (“Actions” as defined above in the user role) allow that to happen.
As mentioned above, the ATS Self Service users have already 2 resources allocated (one guest profile and one vm template) – the Visitor SS users have not been allocated any.

In order to share resources between ATS and visitors the ATS SS user role must have the “share” action enabled and the Visitors SS user role the “receive” action enabled (we have done this when we created the user roles)

Also note that the SS user must be the owner of the resource in order to share it (e.g. must have created the resource or be made the owner by an admin)

• We logged in as user “ats1” and created a test guest profile “Shared Guest Profile”
• In the properties of the resource (library view) ats1 can now share the resource with other user roles (that have the “receive” action enabled), see below:
  Note: Ensure that that the logged in user is the owner of the resource and add other user roles for access as desired

After performing the above action and logging in with “visitor1” we can now see the shared profile being available to “visitor1” as expected.

Deploying a Resource to the Cloud

• As expected we are able to (only) specify the visitors cloud as target

The following two tabs change content below.

• Bio
• Latest Posts

Andreas Groth

IBM Cloud Architect with extensive expertise in complex solution design, networking and cloud/virtualisation platforms, translating technologies into real-life solutions for end-clients and channel partners. Currently leading technical consultancy engagements as Channel Solutions Architect for the IBM Watson and Cloud Platform. - Chartered IT professional of the British Computer Society (MBSC CITP) - Chartered Engineer Institution of Engineering and Technology (CEng IET) - IBM Certified Senior IT Specialist - Microsoft Certified Systems Engineer (MCSE) - VMware Certified Professional (ESX2, VI3 and vSphere) – VCP # 1281 Andreas is the creator of virtualizationmatrix.com. Specialties: • x86 virtualization and cloud technologies - VMware, Microsoft and Open Source (Xen and KVM). • Consultancy for server/desktop virtualisation and cloud delivery models - new focus: SoftLayer • Creator of IBM's first virtual desktop reference architectures (SmartCloud Desktop Infrastructure) • IBM System x (x86) and IBM BladeCenter system hardware and IBM Storage/Network Solutions. • Public Speaker at numerous external and internal WW and European Technical Conferences. • Technical Publication Writer with 2 IBM Redbooks, several papers and 2 international patents.

Latest posts by Andreas Groth (see all)

• gambling establishment Options - March 5, 2018
• ‘Lift and shift’ VMware workloads to the Public Cloud (IBM/ SoftLayer Update) - June 29, 2016
• What’s New in Hyper-V 2012 – Extensive Feature Comparison for Server 2012 and System Center 2012 added to the Virtualization Matrix - January 8, 2016