Migrate Virtuozzo containers to VMware ESXi virtual machine

Since Virtuozzo uses VZFS, which shares its common files with the host operating system, it's not possible to convert a container to a virtual machine, as you would do a physical to virtual, or virtual to virtual migration.
So the VMware converter cannot be used, you'll get an error indicating that the source server details cannot be retrieved, both on Windows and Linux.

As a workaround, you can migrate the containers to Parallels virtual machines. For this you need the PSBM (Parallels Server Bare Metal) product. Install this hypervisor on a physical machine, it supports installation inside virtual machines, but there are limitations, read about them here.

Once you have the PSBM, you are ready to migrate the containers to virtual machines. The "pmigrate" CLI tool can be used to do this, as such:

Linux:       pmigrate ct root@yourvzserver.com/CTID vm localhost/vmname
Windows: pmigrate ct Administrator@yourvzserver.com/CTID vm localhost/vmname


A prerequisite for this migration, is that you have the Parallels transporter agent installed, on the Virtuozzo source hardware node. Installers for both Linux and Windows can be found under /usr/share/pmigrate/tools/ on the PSBM.

It's a live migration, the container stays online, during and after migration is finished.
Depending on the guest operating system of the container, the Parallels tools (equivalent to VMware tools) will be automatically installed inside the virtual machine during the migration process.

There are limitations on which guest OS can be migrated, I have tried these with success:
Debian 5 x86/x64
Debian 6 x86/x64
Windows 2003 x64
Windows 2008R2 x64

For a complete list of supported guest operating systems, browse to page 31 on this document.

Once the virtual machine is running on PSBM, you can use the VMware converter for migration to an ESXi server. You can use the converter on 2 different ways.

• If the virtual machine is online and accessible from the ESXi and converter, you can do a migration as you would do a powered-on physical machine
• Another way is to use the "Backup image or third-party virtual machine" converter option. You'll need the Parallels VM configuration and harddisk files, these can be found under /var/parallels/ on the PSBM.

A couple remarks:

• The cpu/memory/disk resources from the original containers will have to be reapplied after migration, since these are not preserved
• Because of the hardware changes, Windows (2008R2) will have to be re-activated after migration
• Remove the Parallels tools on Windows guests, before converting them from PSBM to VMware. I wasn't able to uninstall this software anymore once the VM was running on VMware.

Paralells Virtuozzo Linux 4.7 kernel panic

After upgrading from PVCfL 4.0 / 4.6 to version 4.7, kernel panics started to occur randomly on almost all hardware nodes.

I configured kdump on the nodes to collect a kernel dump upon the next kernel panic, and contacted Parallels support team.

When the kernel panic occured and the dump was created, they analyzed it.

The feedback we got was the following:

The latest saved dump points to general protection fault caught by `mysqld`. This looks like just recently found issue - PCLIN-30321. I will pass it to the maintenance team.

As for the type of installation - there is no difference if this is a clean installation or an upgrade from the previous version of Virtuozzo.

The issue is quite complex (General Protection Fault from a process in a container is quite difficult to investigate), and it would take some time for maintenance / development team to check the case.

Though, the investigation might take significant amount of time and therefore the results will be provided through the separate ticket once they are ready.
You can also check the release notes for Virtuozzo kernel updates (CU-2.6.32-042stabXXX), and monitor the status of the request PCLIN-30321 which was assigned for this crash issue.

It's not always caused by 'mysqld', I've seen a kernel panic caused by another process as well, but most were indeed due to mysqld.
We don't know how long it will take for this to be fixed, so we decided to migrate the containers which cause the kernel panics to a 4.0 or 4.6 node.

Finding out which container caused the kernel panic can be done by using the crash tool.
An example:

[root@virtuozzo 127.0.0.1-2011-11-05-08:00:17]# crash /root/vmlinux-2.6.32-042stab037.1 vmcore
crash 4.1.2-8.el5.centos

      KERNEL: /root/vmlinux-2.6.32-042stab037.1
    DUMPFILE: vmcore
        CPUS: 8
        DATE: Sat Nov  5 10:00:13 2011
      UPTIME: 10 days, 10:12:02
LOAD AVERAGE: 19.68, 15.35, 14.01
       TASKS: 2239
    NODENAME: virtuozzo
     RELEASE: 2.6.32-042stab037.1
     VERSION: #1 SMP Fri Sep 16 22:18:06 MSD 2011
     MACHINE: x86_64  (1995 Mhz)
      MEMORY: 32 GB
       PANIC: ""
         PID: 62872
     COMMAND: "mysqld"
        TASK: ffff88024d6892c0  [THREAD_INFO: ffff8802d65e4000]
         CPU: 5
       STATE: TASK_RUNNING (PANIC)

crash> dmesg
[900722.171220] Process mysqld (pid: 62872, veid=7420232, threadinfo ffff8802d65e4000, task ffff88024d6892c0)

Update 15/11/2011
On 14/11/2011, Parallels has released a kernel update which contains a fix for this problem. More information on http://kb.parallels.com/en/112809

Plesk 10: display os/browser/robots in awstats statistics

I'm using Plesk 10.1 on a Debian 5.0 VPS, and noticed that the statistics (Awstats) for my domains did not contain all the information I want, such as statistics for operating systems, browsers, robots/spiders.

I checked several configuration files, and found that the "plesklog" format, that Plesk uses to define the Apache LogFormat value, does store this information in the logfiles.
However, the Awstats model configuration file is not properly configured to retrieve this information from the logs.

By default, the value of the LogFormat parameter in /etc/awstats/awstats.model.conf will be "LogFormat=4", which is the Apache native common log format, not the combined, or plesklog format.
By changing this value to "LogFormat=1", these features (browsers, os, robots,...) will be shown in the statistics of your domain(s).

After modifying the awstats.model.conf file, you'll need to modify the awstats configuration files for your existing domains.
You can use this perl script to do so:
perl -e "s/LogFormat=4/LogFormat=1/g;" -pi $(find /opt/psa/etc/awstats/ -type f)

This will go through all the files inside the /opt/psa/etc/awstats/ directory, look for "LogFormat=4", and replace it by "LogFormat=1". Be sure to create a backup of all your files, in case something goes wrong.

The new statistics features will only be visible after the next calculation.

Parallels Virtuozzo Linux vzbackup error

Last week, I installed a new PVC Linux 4.6 node, and added it as a slave to an existing PVA (Parallels Virtual Automation) group.
I use the vzbackup cronjob to backup all the containers on the hardware nodes.
The node that initiates the backup cronjob, had PVC Linux 4.0 installed, all the other slave nodes have version 4.0 installed as well.

The backup of the new 4.6 node failed with this error message:

vzbackup(2861): Starting backup. Nodes - virtuozzo10
vzbackup(2861): Starting node virtuozzo10 backup...
vzbackup(2861): Checking backup version on virtuozzo10 ... use vzbackup 4.6.0-9
vzbackup(2861): Starting node virtuozzo10 backup...
vzbackup(2861): Starting backup CT 1(virtuozzo10)...
bash: /usr/share/vzbackup-4.6.0-9.swsoft/vzbackup1: No such file or directory
vzbackup(2861): Can not read layout version for CT 1(virtuozzo10)
vzbackup(2861): No containers intended for backup found on virtuozzo10

The cause of this error message is that vzbackup 4.0 cannot create backups of containers on a 4.6 node.

Luckily, the other way around does work. After upgrading the node that initiates backups from 4.0 to 4.6, backups of the new node succeeded.
Backups of the other 4.0 nodes also remained successfully.

Using vzbackup on a PVC 4.6 node to backup 4.0 nodes is a supported method by Parallels.

Disable UAC in Windows Server 2008

You can disable the UAC feature in Windows Server 2008 by following these steps:

• Start -> Run -> msconfig


• Click Tools and scroll down to "Disable UAC"


• Click the Launch button and reboot the server.

Install .NET 3.5 framework on virtuozzo container

When installing the .NET 3.5 framework on a Windows 2003 SP2 x64 VPS I got these errors, using the web or full installer:

XPSEPSC x64 Installer: [2] Error code 1603 for this component means "Fatal error during installation.
XPSEPSC x64 Installer: [2] Setup Failed on component XPSEPSC x64 Installer
WapUI: [2] DepCheck indicates XPSEPSC x64 Installer is not installed.

To install .NET 3.5 framework on a windows virtuozzo container, perform these steps for a successfull installation:

• Restart the VPS


• Rename the C:\windows\system32\catroot2\ folder (not catroot itself but catroot2)


• Start the print spooler service


• Run the .NET 3.5 SP1 installer


• Reboot if necessary

Backup SQL databases to network share

To be able to backup databases to a network share, the MSSQLSERVER service needs access to the location on the network.
For this you need to change the SQL services service account.
Before doing this you'll need to give that user extra permissions, to accomplish this you can add the user to all the SQLServer2005* or SQLServer2008* groups.

To change the logon user go to "SQL Server Configuration Manager". Under Services, click on properties for the MSSQLSERVER.
Here you can change the logon account to the one you'll need. Specify this to a domain user, or a local user that also exists on the machine where you will store the databases, and that has the full access share and NTFS permissions.
After modifying the logon name, you'll have to restart the service. Do the same for the SQL Server Agent.

Now you'll be able to backup databases to a network share and specify the path like this then: \\servername\share.

Reset lost administrator password in Windows Server 2003

While you can use tools like ophcrack to recover your password, these don't always work and in case of strong passwords you'll need to pay for the rainbow table sets, the free included set can only crack alphanumerical passwords up to 14 characters. There is a rainbow table set that will crack passwords up to 33 characters, look on the website for the updated information and prices.

To reset your password you can use tools like the Winternals boot cd, which includes the locksmith tool.
In my experience this doesn't always work, and you have to manually add device drivers to the boot cd. If you work with different types of servers, that come with new types of RAID controllers every x months, it can become hard to keep track of. Not that you loose a password that often, but it can be frustrating when you are in urgent need of such a tool :).

The tool that I have recently found seems to be working great. The password reset did not work, but I was able to set a blank password, logon, and reset the password using computer management.
It's called "Offline NT Password & Registry Editor", visit its website and download it here.
This tool includes a lot of recent raid controller drivers which is necessary, so it detects your Windows installation partition.

Before using this tool, be sure that Windows was shutdown properly, or you will not be able to perform any actions!
If that's the case, the tool will warn you and you'll have to reboot in safe mode, and perform a clean shutdown twice.
Besides that, the tool is pretty self-explanatory, but the author has also written a walkthrough.

I was able to set a blank password on a Windows Server 2003 x64 edition with this tool, which was a great help!
As always, keep in mind that resetting a password can cause problems if not used carefully, for example in cases where EFS is used, those files will become inaccessible.

Add windows components on Virtuozzo for Windows fix

I ran into this issue while using Virtuozzo 3.5.1 SP1 on Windows Server 2003 SP1 Datacenter Edition x64.
When creating new virtual private servers most default components are already installed, but when you want to add extra windows components (add/remove software -> windows components), you bump into this error:

Setup library wssoc.dll could not be loaded, or function StorageServerSetupProc could not be found.
The specific error code is 0x7e.

There is currently no fix available for this, it will be solved in Virtuozzo 4.0.
You can however fix this yourself by creating a new custom application template, using the virtuozzo template creation wizard. Choose to create a template from manually selected data, be sure to have created a temporary VPS already, you'll need to enter the vps id, and desired template name.
When prompted which files you want to include, add these:

c:\windows\system32\wssoc.dll
c:\windows\system32\hpcoc.dll

Finish the wizard, then run the command "vzpkgdeploy -i <templatename>" to enable this template.

Automate Virtuozzo backups on Windows

There is currently no GUI option to automate Virtuozzo backups, although not in current version 3.5.1 SP1.
This will be possible in Virtuozzo 4.0, which is currently in beta.

I'm using 2 bat files to script the backups and remove the oldest backup.
The first script creates a full backup of all virtual private servers using the vzbackup command:

for /F "skip=3 usebackq tokens=*" %%i in (`cmd /c "vzlist -ao veid"`) do (cmd /c "vzbackup "%%i && echo VE ID %%i has been backed up!)

To be able execute the second script you need to download RmEmpty, an executable which allows you to remove empty directories, get it here

First, we'll have to search the destination folder on the backup node for files older then x days, in this example I use 4 days.
forfiles /P e:\vz\backups /S -D -4 /c "cmd /c del /q @file"

The forfiles is a default windows binary, some information about the given variables:
/P indicates the path to start searching
/S instructs forfiles to recurse into subdirectories
/D selects files with a last modified data less than or equal to (-)
/C indicates the command to execute for each file
The /q is to delete the files quietly, otherwise it will ask for a confirmation before deletion.

After deleting the files we'll search for empty directories, and remove them:
C:\system\rmemp12\RMEMPTY.EXE e:\vz\backups\

Be aware that if e:\vz\backups\ itself is also empty, it will remove that one too! So always keep at least 1 file in that directory, I use a simple readme.txt so it never gets deleted.

Install VMware Tools on Debian

During installation the kernel headers and a C compiler will be required, to check which kernel you are using type this command:

# uname -r
2.6.18-5-686

# apt-get install linux-headers-2.6.18-5-686
# apt-get install make gcc

Now that the required packages are installed follow these steps to install the vmware tools:

1. click install vmware tools from the vmware server console menu
2. mkdir /mnt/cdrom
3. mount /dev/cdrom /mnt/cdrom
4. cd /tmp
5. tar zxf /mnt/cdrom/VMwareTools-1.0.3-44356.tar.gz
6. cd vmware-tools-distrib
7. ./vmware-install.pl
8. accept the default paths and choose to run the vmware-config-tools.pl at the end

The VMware tools have now been installed.

Transfer SQL Server Jobs from SQL Server 2000 to 2005

To transfer sql server jobs from sql server 2000 to sql server 2005 you can use SQL Server Business Intelligence Development Studio.

1. Click file -> new -> project to create a new Integration Services Project
2. Click view -> toolbox so you can see an overview of all the items
3. From the toolbox, drag the "transfer jobs task" to the control flow window
4. Right click the transfer job task and click edit
5. Select jobs and enter the source and destination server, you can test the connection immediately
6. Choose to transfer all jobs, or only the ones you can select from the given list
7. Set the desired IfObjectExists and EnableJobsAtDestination option and click OK
8. Right-click the transfer jobs task and click execute task

If the job fails it will turn red, if it succeeds it turns green. Click on execution results to check for errors.
A couple of reasons why the task would fail:

• the job owner does not exist on the destination server, you can modiy this on the source server in the job properties


• if you have e-mail notifications enabled on the job, be sure the operator exists on the destination server or disable this


• if the database for which the job will be executed does not exist that job transfer will fail

A couple screenshots from the transfer jobs task properties and the execution results:





 

Deploy Windows Server 2003 using WDS

• Install WS 2008 and WDS


• Copy boot.wim from \sources folder from a Vista DVD


• Use WDS to add that boot.wim to the boot images


• Right-click this image, and select "Create capture boot image", call it Capture or something similar


• Add this capture image to the boot images


• Disable all boot images, except for the capture image


• Install Windows Server 2003 with service packs, updates, applications,... on a new server


• Extract the content of \support\tools\deploy.cab from the ws 2003 cd to c:\sysprep on the new server


• Run c:\sysprep\setupmgr.exe to create a sysprep answer file for future deployments


• Save the answerfile as sysprep.inf and copy it to the WDS server


• Run c:\sysprep\sysprep.exe /reseal /mini /reboot


• When the server reboots, press F12 twice to do a PXE boot and load the winPE for capturing


• Follow the steps until you get to the Image Capture Source screen, here you can select the volume, usually this would be C:\, you have the option to immediately upload the captured image to the WDS server, be sure to have created an image group first. Type the ip address of the WDS server, and then you have to type the credentials to connect (turn off the windows firewall)

• When finished, click close, the server will reboot and load the sysprep wizard, complete this wizard. In future deployments this will be done by sysprep.inf


• Copy the sysprep.inf that you saved earlier on the WDS server to \RemoteInstall\Images\<imagegroup>\<imagename>\$OEM$\$1\Sysprep\ (you have to create this structure)


• Create an unattended file for the WDS client and attach this to the WDS server:

<?xml version="1.0" ?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
   <settings pass="windowsPE">
	  <component name="Microsoft-Windows-Setup" publicKeyToken="31bf3856ad364e35"
language="neutral" versionScope="nonSxS" processorArchitecture="x86">
	     <WindowsDeploymentServices>
	        <Login>
	           <WillShowUI>OnError</WillShowUI>
	           <Credentials>
	                 <Username>**username**</Username>
	                 <Domain>**domain**</Domain>
	                 <Password>**password**</Password>
	              </Credentials>
	           </Login>
	        <ImageSelection>
	           <WillShowUI>Never</WillShowUI>
	           <InstallImage>
	              <ImageName>**imagename**</ImageName>
	              <ImageGroup>**imagegroup**</ImageGroup>
	              <Filename>**filename.wim**</Filename>
	           </InstallImage>
	              <InstallTo>
	              <DiskID>0</DiskID>
	              <PartitionID>1</PartitionID>
	           </InstallTo>
	        </ImageSelection>
	     </WindowsDeploymentServices>
	        <DiskConfiguration>
	            <WillShowUI>Never</WillShowUI>
	            <Disk>
	                <CreatePartitions>
	                    <CreatePartition>
	                        <Order>1</Order>
	                        <Type>Primary</Type>
	                        <Extend>true</Extend>
	                    </CreatePartition>
	                </CreatePartitions>
	                <DiskID>0</DiskID>
	                <WillWipeDisk>true</WillWipeDisk>
	                <ModifyPartitions>
	                    <ModifyPartition>
	                        <Active>true</Active>
	                        <Extend>False</Extend>
	                        <Format>NTFS</Format>
	                        <Label>SYSTEM</Label>
	                        <Letter>C</Letter>
	                        <PartitionID>1</PartitionID>
	                        <Order>1</Order>
	                    </ModifyPartition>
	                </ModifyPartitions>
	            </Disk>
   </DiskConfiguration>
	  </component>
   </settings>
</unattend>

• Create an auto-cast session for this image to be deployed on other machines

Inject VMWare NIC driver into Windows Server 2008 boot image

This guide only works on WS 2008 or Vista. The WAIK is supported on WS 2003 and XP too, but the driver injection will fail.
You can download the WAIK here.

Install the WAIK, open the Windows PE Tools Command Prompt. Also, have the WS 2008 boot.wim ready, and mount the VMWare Tools image by clicking "Install VMWare Tools", this will mount the drivers on the cd-rom drive.
If you are not doing these steps on a virtual machine, you can also just download these drivers (vmxnet).

1. copype x86 c:\windowspe
2. imagex /info "c:\path\to\x86\boot.wim"
In the WIM Information, look for the Boot Index, in this case the boot index is 2. We will use this in the next step.

3. imagex /mountrw "c:\path\to\x86\boot.wim" 2 c:\windowspe\mount
4. peimg /inf="c:\path\to\vmxnet\win2k\vmxnet.inf" c:\windowspe\mount\windows
5. peimg /inf="c:\path\to\vmxnet\win2k\vmware-nic.inf" c:\windowspe\mount\windows
You should see "PEIMG completed the operation successfully". If you would do this on a XP or 2003 machine, the message would be "PEIMG failed to complete the operation with status code 0x800703fb".
It's possible that this WILL work in future versions, but at the moment it doesn't

6. imagex /unmount c:\windowspe\mount /commit

The boot.wim image has now been injected with the VMWare NIC drivers.

There is an alternative to this, instead of doing the above driver injection procedure, you can add this to your .vmx file to have network support too: ethernet0.VirtualDev="e1000"

Clear System Event Log (SEL) on DELL PowerEdge Server

I had to ran the diagnostic tools on a dell pe 1950 server, but the test stopped very early with the message "the system event log contains memory related errors".
There are several ways to clear the SEL, one that can be performed locally and one remotely.

1. Clear the SEL locally

At boot, press ctrl-E to set the BMC ip address, in this same window you can choose to clear the system event log.

2. Clear the SEL remotely

If you have not already set an ip address for the BMC, do it now by pressing ctrl-E at boot. Besides configuring the ip address, enable IPMI too.
At the client be sure to have the bmc management utility installed, the default installation path is c:\program files\dell\sysmgt\bmc.
Perform the following command to clear the sel: ipmish.exe -ip 192.168.1.44 -u root -p calvin sel clear
To view how many entries are in the sel: ipmish.exe -ip 192.168.1.44 -u root -p calvin sel status

Change logical filename of SQL database

Perform this query to change the logical filename of both the data and logfile of you database.

USE MASTER
GO
ALTER DATABASE MyDatabase
MODIFY FILE
(NAME = <DBFileName_DATA>, NEWNAME='NewDBFileName_Data')
GO

ALTER DATABASE MyDatabase
MODIFY FILE
(NAME = DBFileName_Log, NEWNAME='NewDBFileName_Log')
GO

Install VMWare Tools on Server Core

In VMware Server Console, click "Install VMware Tools".
The setup GUI will not automaticly appear, so you have to run it manually. Actually by clicking on "install vmware tools" the cd-rom drive was mounted with tools, so you can now type "D:", and then run setup.exe.
This will run the setup GUI and you can install the tools like you would normally.

Install Active Directory Domain Services on Server Core

To install active directory on windows server 2008, server core edition, you must first create an answer file for the unattended installation, because server core does not offer the GUI wizard.
This sample answer file is to install a new domain in a new forest, with the installation of a DNS server.


[DCInstall]
InstallDNS = Yes
ConfirmGc = Yes
DNSOnNetwork = no
DomainLevel = 2
DomainNetBiosName= mydomain
ForestLevel = 2
NewDomain = Forest
NewDomainDNSName = mydomain
Password = mypassword
ReplicaOrNewDomain = Domain
SafeModeAdminPassword = mypassword


Save this file as "answerfile.txt" and then run the following command:

dcpromo /answer:answerfile.txt

Change schema for all tables in SQL Server 2005

By using the stored procedure MSforeachtable it's possible to perform the ALTER SCHEMA sql statement for all tables in a given database.
To change the schema for all tables this would look like:

exec sp_MSforeachtable "ALTER SCHEMA new_schema TRANSFER ? PRINT '? modified' "

Which would look like this if you want to alter schema to dbo:

exec sp_MSforeachtable "ALTER SCHEMA dbo TRANSFER ? PRINT '? modified' "

A matching connector cannot be found to route the external recipient

When I wanted to send an e-mail from my exchange mailbox to an external e-mail address, the message wasn't send.
I tracked the message in the Queue Viewer and it couldn't be sent because of this: "A matching connector cannot be found to route the external recipient".

To be able to send messages to external domains you have to create a send connector for domain "*". You can do this in the Organization Hub Transport Configuration: