Author: Roshan Chavan

NSX 4.0 Series Part5-Migrate workload from VDS To NSX

Roshan ChavanLeave a comment

Welcome back readers.

Please find the links below for all posts in this series.

NSX 4.0 Series Part1-NSX Manager Installation
NSX 4.0 Series Part2-Add a Compute Manager & Configure the NSX VIP
NSX 4.0 Series Part3-Create Transport Zones & Uplink Profiles
NSX 4.0 Series Part4-Prepare Host Transport Nodes
NSX 4.0 Series Part5-Migrate workload from VDS To NSX

Our NSX env is fully functional and we are ready to migrate workload from vCenter VDS to NSX env.

It’s always a good practice to verify the NSX env before we start working on it.

Login to NSX VIP and look for Alarms,

Check the cluster status,

And then look for host transport nodes if they are showing host status as UP,

For testing purposes, I have created 3 windows vm’s. All three vm’s connects to 3 different port groups on vCenter VDS. We will move these VM’s from vCenter VDS to NSX managed segments.

Following are test VM’s with their respective vds port groups. I have named these VM’s according to PG.

Next, we need to create Segments in NSX env. A Segment is nothing but the portgroup.

Let’s have a look at the types of Segments.

VLAN Baked Segments: In this type, you will define a VLAN ID for the segments, however you also have to make sure that this vlan configure exists on your physical top of the rack switch.

Overlay Backed Segments: This segment can be configured without any configuration on the physical infrastructure. It gets attached to Overlay Transport Zone and traffic is carried by a tunnel between the hosts.

As stated earlier, we would be only focusing on VLAN backed segments in this blogpost. Visit the following blog if you are looking for overlay backed segment.

https://virtualrove.com/2020/06/04/nsx-t-3-0-series-part9-create-segments-attach-to-t1-gateway/

Login to NSX and navigate to Networking> Segments,

Oops, I haven’t added license yet. If you do not have a license key, please refer to my following blog to get the eval licenses.

https://virtualrove.com/2021/11/22/how-to-get-free-vmware-eval-licenses/

Add the license key here,

System> Licenses,

Then we move to create a VLAN backed segment in NSX. You can create vlan backed segments for all networks that exist on your TOR (top of the rack switches). For this demo, I will be using Management-1631, vMotion-1632 and VSAN-1633 networks.

In my lab env, following networks are pre-created on the TOR.

Login to NSX VIP> Networking> Segments> Add Segment

Name: VR-Prod-Mgmnt-1631
Transport Zone: VirtualRove-VLAN-TZ (This is where our esxi host transport nodes are connected)
VLAN: 1631

SAVE

Verify that the Segment status is Success.

Once the segment is created in NSX, go back to vCenter and verify if you see the newly created segment. You will see a letter “N” for all NSX create segments.

Click on the newly created Segment.

Note that the Summary section shows more information about the segment.

We will now move a VM called “app-172.16.31.185” from VDS to NSX.

Source VDS portgroup is “vDS-Management-1631”
Destination NSX Segment is “VR-Prod-Mgmnt-1631”

Verify that it is connected to VDS portgroup.

Login to the VM and start a ping to its gateway IP.

Login to the vCenter> Networking view> Right Click the source port group>

And select “Migrate VM’s to another network”.

In the migration wizard, select newly created NSX vlan backed segment in destination network,

Select the VM that needs to be migrated into the NSX env,

Review and Finish,

Monitor the ping command if we see any drops.

All looks good. NO ping drops and I can still ping to the vm ip from other machines in the network.

We have successfully migrated a VM into the NSX env.
Verify the network name in VM settings,

Click on the NSX segment in vCenter and verify if you see the VM,

You can also verify the same from NSX side,
Login to NSX> Inventory> Virtual Machines> Click on View Details for the VM that we just migrated,

You will see port information in details section,

You will not see port information for db vm, since it has not been migrated yet.

Remaining VM’s have been moved into the NSX env. Ports column shows “1” for all segments.

We see all 3 NSX segments in vCenter networking view,

Simple ping test in cross subnets.  From App To DB,

Well, all looks good. Our workload has been successfully migrated into NSX env.

So, what is the use case here…?
Why would customer only configure vlan backed segments…?
Why No overlay…?
Why No T1, T0 and Edge…?

You will surely understand this in my next blog. Stay tuned. 😊
Hope that this blog series has valuable information.

Are you looking out for a lab to practice VMware products…? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

Leave your email address in the box below to receive notification on my new blogs.

NSX 4.0 Series Part4-Prepare Host Transport Nodes

Roshan ChavanLeave a comment

In the previous blogpost, we discussed Transport Zones & Uplink Profiles. Please find the links below for all posts in this series.

NSX 4.0 Series Part1-NSX Manager Installation
NSX 4.0 Series Part2-Add a Compute Manager & Configure the NSX VIP
NSX 4.0 Series Part3-Create Transport Zones & Uplink Profiles
NSX 4.0 Series Part4-Prepare Host Transport Nodes
NSX 4.0 Series Part5-Migrate workload from VDS To NSX

In this blogpost, I will configure the host transport node for NSX. Basically, in this process, NSX vibs are installed on the ESXi node via NSX Manager. They are also referred to as kernel module. You can see the number of installed vibs on esxi by running following command,

Open up a putty session to one of the esxi and run this command,

esxcli software vib list

Filter the one for NSX by running following command,

esxcli software vib list | grep nsx

We don’t see any since we have not configured this host for NSX yet. Let’s revisit this after the NSX installation.

Note: Preparing ESXi host for NSX does not need host reboot.

Before we prep an esxi host for NSX, check the name of VDS,
vCenter> Click on ESXi host> Configure> Virtual Switches,

Note the VDS name. We will revisit here after NSX vibs installation.

Login to NSX VIP & navigate to System >Nodes >Host Transport Nodes.
Change the “Managed by” drop down to vCenter. Notice that the ‘NSX Configuration’ column shows ‘Not Configured’.

Select the first host & click on ‘Configure NSX’

Next,

Mode: Standard
Name: Select appropriate VDS from vCenter
Transport Zone: Select VLAN TZ that we created earlier.
Uplink Profile: VR-UplinkProf-01

Scroll down to Teaming policy uplink mapping,

Select Uplink1 & Uplink2 respectively.

Here, you are mapping vCenter VDS uplinks to NSX.

Click Finish to begin the installation.

Monitor the progress.

I got an error message here,

Failed to install software on host. Failed to install software on host. esxi127.virtualrove.local : java.rmi.RemoteException: [InstallationError] Failed to create ramdisk stagebootbank: Errors: No space left on device cause = (272, ‘Cannot reserve 272 MB of memory for ramdisk stagebootbank’) Please refer to the log file for more details.

Not sure why this came up. I have enough of compute resources in that cluster. Clicked on “Resolve”

And it was a success. 😊

Next, I see another error.

“The controller has lost connectivity.”

Clicked on “SYNC” here and it was all good.

1st ESXi node has been configured and ready for NSX. Verify the NSX version and node status.

Go back to vCenter> ESXi Host> Configure> Virtual Switches,

We now see the “NSX Switch” added as a prefix to VDS name.

Let’s re-run the command,

esxcli software vib list | grep nsx

We now see all NSX vibs installed on this host.

Let’s move to the next ESXi node and configure it in same way.

All 3 ESXi hosts have been configured for NSX.

That’s all for this post.

I hope that the blog has valuable information. See you all in the next post.

Are you looking out for a lab to practice VMware products…? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

Leave your email address in the box below to receive notification on my new blogs.

NSX 4.0 Series Part3-Create Transport Zones & Uplink Profiles

Roshan ChavanLeave a comment

In the previous blogpost, we discussed on Compute Manager and NSX VIP. Please find the links below for all posts in this series.

NSX 4.0 Series Part1-NSX Manager Installation
NSX 4.0 Series Part2-Add a Compute Manager & Configure the NSX VIP
NSX 4.0 Series Part3-Create Transport Zones & Uplink Profiles
NSX 4.0 Series Part4-Prepare Host Transport Nodes
NSX 4.0 Series Part5-Migrate workload from VDS To NSX

This post will focus on Transport Zones & Uplink Profiles.

It is very important to understand Transport Zones and Uplink Profiles to configure NSX env.

Transport Zone:

All types of hypervisors (that get added to NSX env) as well as EDGE VM are called transport nodes and these transport nodes needs to be a part of transport zones to see particular networks. Collection of transport nodes that define the maximum span for logical switches. A transport zone represents a set of similarly provisioned hypervisors and the logical switches that connect VMs on those hypervisors. It also has been registered with the NSX management plane and has NSX modules installed. For a hypervisor host or NSX Edge to be part of the NSX overlay, it must be added to the NSX transport zone.

There are two types of Transport Zones. Overlay and Vlan Transport Zones. I have already written a blog on previous versions of NSX and explained Transport Zones here…

NSX-T 3.0 Series: Part4-Create Transport Zones & Uplink Profiles

There is already a lot of information on the web regarding this topic. You can also find VMware official documentation here…

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/installation/GUID-F739DC79-4358-49F4-9C58-812475F33A66.html

In this blogpost, we will be only focusing on VLAN backed segments. NO overlay, No Edge, No BGP / OSPF routing.

Visit my NSX-T 3.0 blogpost series below if you are looking to configure Overlay, Edge and BGP Routing.

NSX-T

Let’s get the VLAN backed env in place. It’s simple and easy to understand. Here is the small design that explains what we are trying to accomplish here…

Time to configure VLAN Transport Zone,

Login to NSX VIP and navigate to System> Transport Zones> ADD Zone

Enter the name and select VLAN under traffic type,

Verify that the TZ is created,

Time to configure Uplink Profile,

An uplink profile defines how you want your network traffic to go outside of NSX env. This helps with the consistent configuration of the network adaptors.

Navigate to System > Fabric> Profiles > Uplink Profile,

> Add Profile,

Enter the name and description. Leave the LAG’s section for now. I will write another small blog explaining the LAG configuration in NSX env. Scroll down to Teamings,

Select the default policy to “Load Balance Source”
Type “U1,U2” in Active Uplink field. Input keywords really does not matter here, you can type any name comma separated.
Transport VLAN value to remain 0 in our case.

Teaming Policy Options:

Failover Order: Select an active uplink is specified along with an optional list of standby uplinks. If the active uplink fails, the next uplink in the standby list replaces the active uplink. No actual load balancing is performed with this option.

Load Balance Source: Select a list of active uplinks. When you configure a transport node, you can pin each interface of the transport node to one active uplink. This configuration allows use of several active uplinks at the same time.

A teaming policy defines how C-VDS (Converged VDS) uses its uplink for redundancy and traffic load balancing. Wait, what is C-VDS now…?

N-VDS (NSX Managed VDS): In earlier versions (prior to version 3.0), NSX used to install an additional NSX Managed Distributed Switch. So, one VDS (or VSS) for vSphere traffic and one N-VDS for NSX-T traffic. So, technically speaking, you need 2 more additional pnics for an additional N-VDS switch.

C-VDS (Converged VDS): NSX now uses existing VDS for NSX traffic. However, C-VDS option is only available when you use NSX-T 3 or higher with vSphere 7 along with the VDS version 7.0. You do not need additional pnics in this case.

We are done with the Uplink Profile configuration. More information on Uplink Profiles can be found here,

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/installation/GUID-50FDFDFB-F660-4269-9503-39AE2BBA95B4.html

Check to make sure that the Uplink Profile has been created.

That’s all for this post. We are all set to prepare esxi host transport nodes.I hope that the blog has valuable information. See you all in the next post.

Are you looking out for a lab to practice VMware products…? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

Leave your email address in the box below to receive notification on my new blogs.

NSX 4.0 Series Part2-Add a Compute Manager & Configure the NSX VIP

Roshan ChavanLeave a comment

In the previous blogpost, we discussed NSX Manager installation and its requirements. Please find the links below for all posts in this series.

NSX 4.0 Series Part1-NSX Manager Installation
NSX 4.0 Series Part2-Add a Compute Manager & Configure the NSX VIP
NSX 4.0 Series Part3-Create Transport Zones & Uplink Profiles
NSX 4.0 Series Part4-Prepare Host Transport Nodes
NSX 4.0 Series Part5-Migrate workload from VDS To NSX

What is a Compute Manager…?

In simple words, vCenter is named as Compute Manager in NSX term.

NSX uses compute resources from the added compute managers to deploy management components. At the same time, it also fetches vSphere cluster information from vCenter / Compute Manager.

Let’s add a compute manager in our env.

Login to 1st NSX manager and navigate to System > Fabric > Compute Managers > 

Click on “Add Compute Manager”.

Fill in the required information here and click on Add.

You will be prompted to add the vCenter thumbprint. Click Add and Save. Here is how it will look after adding the compute manager.

Make sure that the compute manager is showing as Registered and its UP.

Important Note:

You can not remove a compute manager if you have successfully prepared ESXi host transport nodes & Edge nodes. You need to manually uninstall NSX from all ESXi host transport nodes and remove any edge nodes as well other components that got deployed from NSX before you try to remove a compute manager.

Next, we add 2 more NSX Managers in the env to form a NSX Cluster.

Why 3 NSX Managers…?

It’s a generic “Cluster” definition in IT world. Each cluster has a Quorum. And NSX Cluster too requires a quorum. Which means, 2 of its 3 members should be up at a given time for NSX env to function / operate properly.

So,
1 NSX Manager is single point of failure.
2 NSX Managers cannot full fill the cluster quorum definition / requirement since generic definition always refers to additional witness node in 2 node cluster configuration.
3 NSX Manager is perfect number to form a cluster.

Hence, 3 NSX Managers, Place them on 3 different physical ESXi nodes and configure anti-affinity to avoid two managers being on single esxi node.

Since it’s a lab env, We will not be deploy remaining 2 appliances. However, here is the link if you want to deploy 2 more appliances.

https://virtualrove.com/2020/05/24/nsx-t-3-0-series-part2-add-additional-nsx-t-mangers-configure-vip/

Next, we configure the VIP (Virtual IP) for NSX Cluster.

What is a VIP…?

A NSX cluster VIP is virtual ip that gets assigned to the cluster. A VIP redirects all requests to master / leader node from the cluster.

When we create a NSX cluster (usually 3 nodes), one of the node gets elected as mater / leader node. Any API and UI request coming in from clients is directed to the leader node. If Leader node is down for any reason, the cluster mechanism automatically elects new leader and all requests gets forward to new leader from the VIP. You also have to make sure that all NSX managers are deployed in the same subnet.

Login to 1st NSX Manager and navigate to System> Appliances…

I only have a single NSX manager in my lab environment. Make sure that the health of the Cluster is “Stable”.

Click on “SET VIRTUAL IP”

Enter the IP address and hit save.

It may take several minutes to add the VIP and NSX GUI might not be accessible for couple of minutes until all required services are up and running.

I see that the VIP has been configured and it is assigned to available NSX manager in the cluster.

I have also created a DNS record for the VIP. Going forward, we will be using this FQDN to access NSX UI.

Let’s login to NSX using VIP and make sure that the state is “Stable”.

All looks good. Let’s move to the next step to prepare Host Transport Nodes for NSX.

That’s all for this post.
I hope that the blog has valuable information. See you all in the next post.

Leave your email address in the box below to receive notification on my new blogs.

Are you looking out for a lab to practice VMware products…? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

NSX 4.0 Series Part1-NSX Manager Installation

Roshan ChavanLeave a comment

No more NSX-V & NSX-T…!!!

It’s only NSX from version 4 onwards.

This series of NSX 4.0 includes following parts.

NSX 4.0 Series Part1-NSX Manager Installation
NSX 4.0 Series Part2-Add a Compute Manager & Configure the NSX VIP
NSX 4.0 Series Part3-Create Transport Zones & Uplink Profiles
NSX 4.0 Series Part4-Prepare Host Transport Nodes
NSX 4.0 Series Part5-Migrate workload from VDS To NSX

It’s been a while since I wrote my last blog post. And finally, I found something interesting to write about. Even if this blog starts with NSX installation and configuration, am sure you will definitely find something more interesting as and when I write more. We will be focusing on NSX Security, DFW & Micro-Segmentation in upcoming blogs.

Let’s get started…

First thing first, Release Notes. It’s always good to read release notes for any of the newly launched products. Here is the VMware official link for NSX 4.0…

https://docs.vmware.com/en/VMware-NSX/4.0/rn/vmware-nsx-4001-release-notes/index.html

It’s a major release focusing on NSX networking, security and services. As it mentions in the release notes,

Some of the major enhancements are the following:

  •  IPv6 external-facing Management Plane introduces support for IPv6 communication from external systems with the NSX management cluster (Local Manager only).
  •  Block Malicious IPs in Distributed Firewall is a new capability that allows the ability to block traffic to and from Malicious IPs.

What interests me most are,

Distributed Firewall

  • Block Malicious IPs in Distributed Firewall is a new capability that allows the ability to block traffic to and from Malicious IPs. This is achieved by ingesting a feed of Malicious IPs provided by VMware Contexa. This feed is automatically updated multiple times a day so that the environment is protected with the latest malicious IPs. For existing environments, the feature will need to be turned on explicitly. For new environments, the feature will be default enabled.
  • NSX Distributed Firewall has now added support for these following versions for physical servers: RHEL 8.2, 8.4, Ubuntu 20.04, CentOS 8.2, 8.4.

In this series of blogs, I will be focusing on NSX 4.0 env setup and later NSX Networking Security, Distributed firewall, Micro-Segmentation & will also touch base on IDS/IPS.

With that let’s get started with NSX 4.0 manager installation. This is pretty simple and straight forward process to deploy NSX manger like previous versions. Download the OVA file from the below link.

https://customerconnect.vmware.com/downloads/details?downloadGroup=NSX-4001&productId=1324&rPId=96437

Make sure to download the 1st OVA file from the link.

If you do not have access to download the file, you will see following message…

You either are not entitled or do not have permissions to download this product.
Check with your My VMware Super User, Procurement Contact or Administrator.

If you recently purchased this product through VMware Store or through a third-party, try downloading later.

Not to worry. Follow this link to get the trial version along with the trial license key…

https://virtualrove.com/2021/11/22/how-to-get-free-vmware-eval-licenses/

Once you have the downloaded ova file, login to your vCenter and import the ova file. Before we import the ova, let me show you my existing env.

I have one vCenter with 3 VSAN enabled ESXi hosts.

Each host has 3 disks. All disks have been claimed for VSAN cluster.

Here is the total compute capacity for this lab,

VDS has been configured in the env and it has 2 physical uplinks,

Each host has 3 vmkernel adaptors.
Management – VLAN 1631
vMotion – VLAN 1632
VSAN – VLAN 1633

I know that this is basic and anyone looking at the NSX blog will defiantly know this. However, believe me, I have seen networking (physical) background people working on NSX DFW & LB without having complete understanding of base vSphere setup. I have detailed blog on how to configure base vSphere setup here,

https://virtualrove.com/2020/06/16/vmware-vsphere-7-0-step-by-step-for-newbies-esxi-vcenter-distributed-switch-vmotion-vsan-ha-drs/

Anyways, let’s get going with NSX manager installation.

Let’s have a look at the NSX manager compute capacity requirements. The thin virtual disk size is 3.8 GB and thick virtual disk size is 300 GB.

Import the NSX ova,

Map the NSX ova,

Fill out / select appropriate options in the wizard and hit continue. Make sure to create a DNS record for 1st NSX Manager.

Fill out the required parameters in the wizard. And we should be good to click on finish.

If any of the required parameters are not as expected, the ova will get deployed with hostname.local. In this case, you must redeploy the ova.

Power ON the NSX VM after the deployment. Wait for 10 mins for all services to start and login.

Once logged in, Click on TOP right corner question mark > About, to check the version.

Review the dashboard,

Click on System> Appliances>, and make sure the cluster status is Stable.

Since it’s a lab env, We will not deploy remaining appliances. However, here is the link if you want to deploy 2 more appliances,

https://virtualrove.com/2020/05/24/nsx-t-3-0-series-part2-add-additional-nsx-t-mangers-configure-vip/

Before you deploy 2 more appliances, you need to add a Computer Manager. We will see the steps to add a Compute Manager in my next blog.

That’s all for this post.
I hope that the blog has valuable information. See you all in the next post.

Are you looking out for a lab to practice VMware products…? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

Leave your email address in the box below to receive notification on my new blogs.

NSX-T 3.1 – Backup & Restore_Production DR Experience – Part3

Roshan ChavanLeave a comment

In this last post of NSX-T backup & restore, we will perform the DR test and post tasks after the restore is complete. Just in case if you have missed, here are links to previous posts.

NSX-T 3.1 – Backup & Restore_Production DR Experience – Part1
NSX-T 3.1 – Backup & Restore_Production DR Experience – Part2

We move to next step by deploying an additional NSX-T manager.

Deploy the new nsx-t manger on the DR site. Make sure to deploy same node for which last successfully backup was taken on the primary site. In my case, last successful backup was for ‘nsx78’ node. I have reserved the new ip address (172.16.31.178) for this node. Remember, we do not need to create DNS record for new node at this point. I am deploying the new nsx-t manager at Site-B (Singapore). Review the deployment parameters here…

Next, Its time to shutdown the primary site nsx-t managers. Ideally, in DR scenario, NSX-T managers on the primary site will be already down. And that’s when we start performing the restore process.

Next, We need to change the DNS records for primary site NSX-T managers to new IP’s from the secondary site. In production env, you might have different subnet on the DR site. I have the same subnet with different IP’s.

Login to the DC and update the DNS records.

Existing IP’s on DNS server.

Update to new IP’s.

All 4 DNS records have been updated.

Next, we need to configure newly deployed NSX-T manger for FQDN using the API call.

https://172.16.31.178/api/v1/configs/management

Let’s start the restore process now.

Configure the SFTP server on newly deployed NSX-T manger and check for recent backup that has been discovered by the appliance.

Note that we are getting EULA prompt. We get this prompt only on the newly deployed NSX-T manager.

Navigate to Backup config and enter sftp server details.

Note that the newly deployed appliance now discovers all successful backups.

Next,

Upon selecting recent backup, you should see ‘Restore’ option highlighted. If you do not see ‘Restore’ option, you will have to re-check all steps provided in this article.

Quick Tip here, if you do not see restore option, scroll down and select the backup where it lists the Appliance FQDN as your newly deployed appliance FQDN. 😊 We found this behavior while troubleshooting at customer location. And it does make sense too. That is why I mentioned earlier in this blog that, deploy the new appliance for which you see last successful backup for.

Note that we do not see any Tier-0 or segments in the newly deployed appliance.

Let’s begin the restore process,

Select the recent backup and restore…
You should see a prompt…

Restoring NSX Managers

If you are running NSX on Virtual Distributed Switch (VDS) and are attempting to perform a Restore operation to a backup file that contains N-VDS switches, then the restore operation might fail. Please

Please read the following steps carefully before you proceed:

  • Step 1: Power Off.
  • Power off all NSX Manager appliances that may be running from the earlier deployment.
  • Step 2: Restore NSX Manager appliances.
  • Restore NSX Manager appliances from the backup.
  • Note: During the restore process, your deployment will be in read-only mode and the UI will be inaccessible. Automatic backups will be disabled.
  • Step 3: Go to the Backup & Restore page.
  • After the restore process ends, log in to your NSX Manager appliance and visit the Backup & Restore page to continue.

NOTE: If the restore fails, you must install a new NSX Manager appliance and try the restore process again.

Review and click Continue.
Restore process begins…

The restore process should start and you will lose the connectivity to new nsx-t manger after 10-15 mins.
Since the restore has started, you will have to re-login to new nsx-t manager with your old password from your primary site.

It would take significant amount of time to come back online. Once up, login to NSX-T manager and navigate the backup again. You should see prompt.

“It appears that you have a NSX Management Cluster configuration in your NSX backup. There were 3 NSX Manager VMs in a NSX Management Cluster at the time of a backup. Currently, there is only 1 NSX Manager VM(s) as reference below. Please navigate to the NSX Manager deployment wizard (System > Overview) and add 2 more NSX Manager VMs to form a NSX Management Cluster of 3 NSX Manager VMs.”

Since we had 3 NSX-T managers on the primary site, backup is prompting us to install remaining 2 nsx-t managers before the restore can be proceed.

Note: After you finish deploying the NSX Management Cluster, you will need to return to this screen and click RESUME to finish the NSX restore workflow.

Before you install an additional nsx-t mangers, make sure to set the VIP of the cluster to new ip address…

Change the VIP from 172.16.31.77 to 172.16.31.177…

It will take around 10 mins to bring back services and to be able to access new VIP.

Deploy the 2nd nsx-t manager once the VIP is accessible. And then followed by 3rd NSX-T manager. Please note that the Cluster Status shows as ‘Degraded’ in the entire process.

Login to the VIP once it is up and running.

Next, we deploy an additional NSX-T managers.

On the next page, Compute Managers were not listing. Cancelled the wizard and checked compute managers. It shows as ‘Down’

Click on ‘Down’ to check the error msg.

“Plugin for compute manager is not running.”

And ‘Resolve’ option was grayed out. Looks like something wrong with the vCenter Server Extensions. Anyways, since it is a lab env, I would not worry much to deploy an additional NSX-T manager. However, we did not receive this error while doing the production DR at customer site.

Let’s go back to backup and resume the operation.

The restore process moved to 62%…

Next, It prompts to check the CM/VC connectivity. (Compute Managers / vCenter). We move on by clicking Resume here.

Then the backup process stops again and prompt to check all listed fabric node connectivity. We can ignore this, since it says that, ‘These nodes might eventually discover the NSX Manager’

Restore process continues to 96%

And finally restore was successful. 😊

Logout and re-login to NSX-T VIP. You should see a msg.

However, it does stop here. We need to make sure that all nodes are connected to new NSX-T manager. You can also deploy an additional NSX-T mangers if needed at this stage, however I skipped it due to compute capacity in the lab.

Couple of tasks after the Restore process…
Navigating to host transport node, it shows couple of error and the same time it gives an option to ‘Resolve’ it.

One of the compute managers shows ‘Down’. Let’s try Resolve option.

It’s UP now.

Dubai-vCenter transport nodes are UP.

Singapore-vCenter transport nodes are UP.

In an actual DR scenario, primary site (Dubai) will show as down since the site itself went down and that’s when we started this restore process.

Edge transport node shows ‘Failed’
Host configuration: Failed to send the HostConfig message. 

Surprisingly, putting the edge node in ‘NSX Maintenance Mode’ and Existing from it resolved the error.

We had to reboot all EDGE VM’s at customer location to resolve this issue.

Let’s check the BGP routes on the TOR.

All looks good here.

The DATA plane did not get affected in this entire activity at all. All workload VM’s had connectivity to the TOR.

Note: vMotion of any VM as well as Snapshot revert or any network related changes to VM will end up losing connectivity to the VM until NSX-T mangers are up & running.

Next, We had already verified routing. For testing purpose, we moved couple of test VM’s from primary site to dr site and it was all good. All newly connected VM’s were able to reach TOR. To move the workload from SiteA to SiteB, customer can opt for SRM (Site Recovery Manager) or any third party VMware compatible product.

We have successfully restored NSX-T manager on the DR site. The entire activity took 2.5 hours at the customer site. You will definitely face multiple issues while performing the DR and it is difficult to get this to success level at first run. At the same, you really don’t want to get used to this DR process. 😀

Thank you for reading the post. I hope this post has added some value to perform successful DR. Good Luck. Leave your comments if you face any issues and I should be able to give you some inputs.

Are you looking out for a lab to practice VMware products…? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

Leave your email address in box below to receive notification on my new blogs.

NSX-T 3.1 – Backup & Restore_Production DR Experience – Part2

Roshan ChavanLeave a comment

In previous post, We validated the base NSX-T env which is setup to perform Disaster Recovery of NSX-T. Here is the link to the previous post.

NSX-T 3.1 – Backup & Restore_Production DR Experience – Part1

There are couple of validation steps from DR point of view. Let’s start with the DR Validation Phase. I have bullet point in excel sheet. Will discuss them one by one with screen captures.

We need to make sure that the publish fqdn is set to ‘true’ in all nsx-t managers using the API call.

Here is the Get API call followed by PUT for all 3 NSX-T managers…

https://172.16.31.78/api/v1/configs/management
https://172.16.31.79/api/v1/configs/management
https://172.16.31.85/api/v1/configs/management

Before we go there let’s have a look at the backup details.

Note that the Appliance FQDN or IP lists an IP address of the appliance.

Let’s go back to the API call and run it. I am using postman utility for API calls.

GET https://172.16.31.78/api/v1/configs/management

Paste the above command, change the Authorization type to ‘Basic Auth’, enter the credentials and send. You might get SSL error if it is not disabled already.

Click on Settings and disable SSL verification here…

and send the call again.

Note that the ‘publish_fqdns’ value is false. We need change this to ‘true’

Change GET to PUT, copy 4 lines, Click on ‘Body’ change the radio button to ‘raw’ and select JSON from the drop down box at the end.

Paste copied 4 lines and change the value to ‘true’ and send it again.

Make sure to see the status as 200 OK.

Use the GET command again to verify…

Make sure that you see this value to ‘true’ on all 3 nsx-t managers.

Let’s go back to NSX-T and run the backup again. Note that the ‘Appliance FQDN’ now lists the FQDN instead of an IP address.

All good till here. Move to next step in the excel sheet.

Next step is to verify that all transport nodes in the env using FQDN instead of an IP address.

Run ‘get controller’ command on each edge node. Note that the “Controller FQDN” lists the actual FQDN instead of an IP.

Next,

All good here. Move to next action item…

Change the TTL value from 1 hr to 5 mins of all NSX-T managers in DNS records properties.

We have completed all validation steps for DR test. We now move to actual DR test in the next part of this blog series. Thank You.

Are you looking out for a lab to practice VMware products…? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

Leave your email address in box below to receive notification on my new blogs.

NSX-T 3.1 – Backup & Restore_Production DR Experience – Part1

Roshan ChavanLeave a comment

Hello Techies, This post will focus on NSX-T Disaster Recovery of the production env that I recently did for one of the customer. Post talks about my own experience and the procedure may differ as per your NSX-T design.

Here is the official VMware documentation which was referred while doing the activity.

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-A0B3667C-FB7D-413F-816D-019BFAD81AC5.html

Additionally, following document is MUST to go through before you plan your DR.

https://communities.vmware.com/t5/VMware-NSX-Documents/NSX-T-Multisite/ta-p/2771370

To put the screenshots in this post, I have recreated the env in my lab. All captures in this post are from the lab that I created for testing purpose.

To set the right expectations, This DR was performed to backup and restore the Management Plane of NSX-T and not the Data Plane. Let me explain the existing env to understand the reason for doing Management Plane recovery only.

  • NSX-T Multisite Env
  • Both sites are active and configured with respective BGP routing to local Top of the Rack (TOR) switches.
  • Primary Site hosts the NSX-T Manager cluster
  • Backup of the NSX-T manager configured on SFTP server which sits at DR site.
  • Both sites have a vCenter, Edge VM’s and ESXi nodes.
  • Inter-Site link has jumbo frames enabled.
  • Both Sites hosts active workload. Also, Load Balancer, VPN as well as micro-segmentation is in place.
  • 3rd Party solution is already configured to Migrate / Restart the VM’s on the DR site in case of disaster.

Since both sites are independent and have sperate EDGE VM’s and routing in place, only Management Plane needs to be restored. The 3rd party backup solution will restore the VM’s on the DR site in case of disaster.

Important Note: Data Plane (i.e. host transport nodes, edge transport nodes…) does not get affected even if you loose the NSX-T manager cluster for any reason. Routing and Connectivity to all workload VM’s works perfectly fine. In short, During the loss of Management Plane, Data Plane is still running as far you do not add any new workload. Also, keep in mind that the vMotion of any VM will end up in loosing the connectivity of that VM if it’s connected to NSX-T Overlay Network. So, it would be a good idea to disable DRS until you bring back the NSX-T manager cluster on the DR site.

The other disadvantage is you cannot make any configuration changes in NSX-T since the UI itself is not available.

Here are some additional bullet points…

  • You must restore to new appliances running the same version of NSX-T Data Center as the appliances that were backed up.
  • If you are using an NSX Manager or Global Manager IP address to restore, you must use the same IP address as in the backup.
  • If you are using an NSX Manager or Global Manager FQDN to restore, you must use the same FQDN as in the backup. Note that only lowercase FQDN is supported for backup and restore.

In most of the cases, FQDN is configured in the env which involves additional steps while restoring the backup. We will discuss more about it in detail. Let’s focus on configuring the backup.

Check my following post for configuring the backup for NSX-T env.

NSX-T Backup Configuration on VMware Photos OS

To begin this post, let’s have a look at the existing env architecture…

List of servers in the env with IP’s.

Here is the screen capture from the env…

Site A vCenter – Dubai

Site B vCenter – Singapore

As I said earlier, we are going to perform Management Plane recovery and not Data Plane, hence I did not configure edge, tier-0 etc on the Site-B. However, customer env had another edge cluster for site B and so the Tier-0. (as shown in the above diagram)

Stable NSX-T manager cluster, VIP assigned to 172.16.31.78

Dubai vCenter host transport nodes

Singapore vCenter host transport nodes

Just a single Edge Transport node deployed at primary site.

BGP Neighbors Configuration…

Note the source addresses. We should see them on TOR as neighbors.

Let’s have a look at the TOR…

Established 172.27.11.2 & 172.27.12.2 neighbors.

BGP routes on the TOR.

Let’s create a new segment and to see if the new routes appears on the TOR.

We should see 10.2.98.X BGP route on the TOR.

Perfect. We have everything in place to perform the DR test and check the connectivity once we bring the NSX-T manager cluster UP in the DR site.

That’s it for this post. We will discuss further process in the next part of this blog series.

Are you looking out for a lab to practice VMware products…? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

Leave your email address in box below to receive notification on my new blogs.

NSX-T Backup Configuration on VMware Photon OS

Roshan ChavanLeave a comment

I want to keep this one short, since it is part of parent topic here…

“NSX-T 3.1 – Backup & Restore_Production DR Experience – Part1”

For NSX-T 3.1, following are supported operating systems as per the VMware documentation, however it also says that other software versions might work. SFTP is the only supported protocol for now.

I remember having discussion with someone using VMware Photon OS for NSX-T backup. It is based on Linux OS and lightweight too. Does not consume many resources. Available for download at following location…

https://github.com/vmware/photon/wiki/Downloading-Photon-OS

Get the Minimal ISO…

Installation is straight forward. Just mount an ISO on a VM and follow the instructions to install it. Then we just run couple of commands to setup the VMware Photon OS.

Here is the screen capture of the commands that has been run to setup the sftp server.

Add the sftp user…
root@VirtualRove [ ~ ]# useradd siteA

Create backup directory…
root@VirtualRove [ ~ ]# mkdir /home/nsxbkp-siteA/

Add a group…
root@VirtualRove [ ~ ]# groupadd bkpadmin

Add a user in the group…
root@VirtualRove [ ~ ]# groupmems -g bkpadmin -a siteA

Set the password for user
root@VirtualRove [ ~ ]# passwd siteA

New password:
Retype new password:

passwd: password updated successfully

The chown command changes user ownership of a file, directory, or link in Linux
chown  USER:[GROUP NAME] [Directory Path]
root@VirtualRove [ ~ ]# chown siteB:bkpadmin /home/nsxbkp-siteB/
root@VirtualRove [ ~ ]#

And that completes the configuration on Photon OS. We are good to configure this directory as backup directory in NSX-T.

Couple of things…
The Photon OS is not enabled for ICMP ping by default. You must run following commands on the console to enable ping.
iptables -A OUTPUT -p icmp -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT

Also, Root account is not permitted to login by default. You need to edit the ‘sshd_config’ file located at ‘/etc/ssh/sshd_config’
You can use any editor to edit this file…

vim /etc/ssh/sshd_config

Scroll it to the end of the file and change following value to enable ssh for root account…

Change the value from ‘no’ to ‘yes’ and save the file. You should be able to SSH to photon OS.

Let’s move to NSX-T side configuration.
Login to NSX-T VIP and navigate to System> Backup & Restore…

Click on Edit for SFTP server and fill in all required information.
FQDN/IP : is your sftp server
Port : 22
Path : We created this in our above steps.
Username, Password & Passphrase.

Save

It will prompt to add for Fingerprints.

Click on ‘Start Backup’ once you save it.

You should see successful backup listed in the UI.

Additionally, you can use WinSCP to login to photon and check for backup directory. You should see recent backup folders.

You also want to set an interval to backup NSX-T configuration as pe the mentioned schedule.
Click on ‘Edit’ from NSX-T UI backup page and set an interval.

I preferred everyday backup, so I set it up to 24 hrs interval.

Check your manager cluster to make sure its stable.

And take a backup again manually.

That’s it for this post.

We have successfully configured SFTP server for our NSX-T environment. We will use this backup to restore it at DR site in case of site failure or in case of NSX-T manager failure for any reason.

Are you looking out for a lab to practice VMware products…? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

Leave your email address in box below to receive notification on my new blogs.

How to get free VMware Eval Licenses

Roshan Chavan1 Comment

Greetings VMware Techies,

We often get into the situation where we need to apply VMware licenses to test VMware products. And most of them are not aware that, VMware provides 60 days trial period for most of the products. For, VMware vCenter Server & ESXi gets the default trial period as soon as you install it. However, for some products you have to ask VMware to provide evaluation license key to test the product. It is easy and simple to get trial licenses for VMware Products.

Check the following link to know the products that are available for trial period license.

https://www.vmware.com/try-vmware.html

Further click on (+) sing to see available products…

Lets assume that You want to test VMware NSX-T Data Center product…
Click on Download Free Trial >>
Next, enter the account information if you have one. If not, then you get an option to create it…

Once you login, clickon ‘License & Download’ and click on ‘Register’

Fill out all the details and click on ‘START FREE TRIAL’

Home screen will show following msg…

And you should see the eval license key under ‘License & Download’ with expiration date.

On the same page, you get an option to download the ova / iso file for the product.

That was easy. 😊
Check out other products available on try-vmware page and enjoy the product evaluation.

Are you looking out for a lab to practice VMware products..? If yes, then click here to know more about our Lab-as-a-Service (LaaS).

Leave your email address in box below to receive notification on my new blogs.