Paralells Virtuozzo Linux 4.7 kernel panic

After upgrading from PVCfL 4.0 / 4.6 to version 4.7, kernel panics started to occur randomly on almost all hardware nodes.

I configured kdump on the nodes to collect a kernel dump upon the next kernel panic, and contacted Parallels support team.

When the kernel panic occured and the dump was created, they analyzed it.

The feedback we got was the following:

The latest saved dump points to general protection fault caught by `mysqld`. This looks like just recently found issue - PCLIN-30321. I will pass it to the maintenance team.

As for the type of installation - there is no difference if this is a clean installation or an upgrade from the previous version of Virtuozzo.

The issue is quite complex (General Protection Fault from a process in a container is quite difficult to investigate), and it would take some time for maintenance / development team to check the case.

Though, the investigation might take significant amount of time and therefore the results will be provided through the separate ticket once they are ready.
You can also check the release notes for Virtuozzo kernel updates (CU-2.6.32-042stabXXX), and monitor the status of the request PCLIN-30321 which was assigned for this crash issue.

It's not always caused by 'mysqld', I've seen a kernel panic caused by another process as well, but most were indeed due to mysqld.
We don't know how long it will take for this to be fixed, so we decided to migrate the containers which cause the kernel panics to a 4.0 or 4.6 node.

Finding out which container caused the kernel panic can be done by using the crash tool.
An example:

[root@virtuozzo 127.0.0.1-2011-11-05-08:00:17]# crash /root/vmlinux-2.6.32-042stab037.1 vmcore
crash 4.1.2-8.el5.centos

      KERNEL: /root/vmlinux-2.6.32-042stab037.1
    DUMPFILE: vmcore
        CPUS: 8
        DATE: Sat Nov  5 10:00:13 2011
      UPTIME: 10 days, 10:12:02
LOAD AVERAGE: 19.68, 15.35, 14.01
       TASKS: 2239
    NODENAME: virtuozzo
     RELEASE: 2.6.32-042stab037.1
     VERSION: #1 SMP Fri Sep 16 22:18:06 MSD 2011
     MACHINE: x86_64  (1995 Mhz)
      MEMORY: 32 GB
       PANIC: ""
         PID: 62872
     COMMAND: "mysqld"
        TASK: ffff88024d6892c0  [THREAD_INFO: ffff8802d65e4000]
         CPU: 5
       STATE: TASK_RUNNING (PANIC)

crash> dmesg
[900722.171220] Process mysqld (pid: 62872, veid=7420232, threadinfo ffff8802d65e4000, task ffff88024d6892c0)

Update 15/11/2011
On 14/11/2011, Parallels has released a kernel update which contains a fix for this problem. More information on http://kb.parallels.com/en/112809

Plesk 10: display os/browser/robots in awstats statistics

I'm using Plesk 10.1 on a Debian 5.0 VPS, and noticed that the statistics (Awstats) for my domains did not contain all the information I want, such as statistics for operating systems, browsers, robots/spiders.

I checked several configuration files, and found that the "plesklog" format, that Plesk uses to define the Apache LogFormat value, does store this information in the logfiles.
However, the Awstats model configuration file is not properly configured to retrieve this information from the logs.

By default, the value of the LogFormat parameter in /etc/awstats/awstats.model.conf will be "LogFormat=4", which is the Apache native common log format, not the combined, or plesklog format.
By changing this value to "LogFormat=1", these features (browsers, os, robots,...) will be shown in the statistics of your domain(s).

After modifying the awstats.model.conf file, you'll need to modify the awstats configuration files for your existing domains.
You can use this perl script to do so:
perl -e "s/LogFormat=4/LogFormat=1/g;" -pi $(find /opt/psa/etc/awstats/ -type f)

This will go through all the files inside the /opt/psa/etc/awstats/ directory, look for "LogFormat=4", and replace it by "LogFormat=1". Be sure to create a backup of all your files, in case something goes wrong.

The new statistics features will only be visible after the next calculation.

Parallels Virtuozzo Linux vzbackup error

Last week, I installed a new PVC Linux 4.6 node, and added it as a slave to an existing PVA (Parallels Virtual Automation) group.
I use the vzbackup cronjob to backup all the containers on the hardware nodes.
The node that initiates the backup cronjob, had PVC Linux 4.0 installed, all the other slave nodes have version 4.0 installed as well.

The backup of the new 4.6 node failed with this error message:

vzbackup(2861): Starting backup. Nodes - virtuozzo10
vzbackup(2861): Starting node virtuozzo10 backup...
vzbackup(2861): Checking backup version on virtuozzo10 ... use vzbackup 4.6.0-9
vzbackup(2861): Starting node virtuozzo10 backup...
vzbackup(2861): Starting backup CT 1(virtuozzo10)...
bash: /usr/share/vzbackup-4.6.0-9.swsoft/vzbackup1: No such file or directory
vzbackup(2861): Can not read layout version for CT 1(virtuozzo10)
vzbackup(2861): No containers intended for backup found on virtuozzo10

The cause of this error message is that vzbackup 4.0 cannot create backups of containers on a 4.6 node.

Luckily, the other way around does work. After upgrading the node that initiates backups from 4.0 to 4.6, backups of the new node succeeded.
Backups of the other 4.0 nodes also remained successfully.

Using vzbackup on a PVC 4.6 node to backup 4.0 nodes is a supported method by Parallels.

Disable UAC in Windows Server 2008

You can disable the UAC feature in Windows Server 2008 by following these steps:

• Start -> Run -> msconfig


• Click Tools and scroll down to "Disable UAC"


• Click the Launch button and reboot the server.

Install .NET 3.5 framework on virtuozzo container

When installing the .NET 3.5 framework on a Windows 2003 SP2 x64 VPS I got these errors, using the web or full installer:

XPSEPSC x64 Installer: [2] Error code 1603 for this component means "Fatal error during installation.
XPSEPSC x64 Installer: [2] Setup Failed on component XPSEPSC x64 Installer
WapUI: [2] DepCheck indicates XPSEPSC x64 Installer is not installed.

To install .NET 3.5 framework on a windows virtuozzo container, perform these steps for a successfull installation:

• Restart the VPS


• Rename the C:\windows\system32\catroot2\ folder (not catroot itself but catroot2)


• Start the print spooler service


• Run the .NET 3.5 SP1 installer


• Reboot if necessary

Backup SQL databases to network share

To be able to backup databases to a network share, the MSSQLSERVER service needs access to the location on the network.
For this you need to change the SQL services service account.
Before doing this you'll need to give that user extra permissions, to accomplish this you can add the user to all the SQLServer2005* or SQLServer2008* groups.

To change the logon user go to "SQL Server Configuration Manager". Under Services, click on properties for the MSSQLSERVER.
Here you can change the logon account to the one you'll need. Specify this to a domain user, or a local user that also exists on the machine where you will store the databases, and that has the full access share and NTFS permissions.
After modifying the logon name, you'll have to restart the service. Do the same for the SQL Server Agent.

Now you'll be able to backup databases to a network share and specify the path like this then: \\servername\share.

Reset lost administrator password in Windows Server 2003

While you can use tools like ophcrack to recover your password, these don't always work and in case of strong passwords you'll need to pay for the rainbow table sets, the free included set can only crack alphanumerical passwords up to 14 characters. There is a rainbow table set that will crack passwords up to 33 characters, look on the website for the updated information and prices.

To reset your password you can use tools like the Winternals boot cd, which includes the locksmith tool.
In my experience this doesn't always work, and you have to manually add device drivers to the boot cd. If you work with different types of servers, that come with new types of RAID controllers every x months, it can become hard to keep track of. Not that you loose a password that often, but it can be frustrating when you are in urgent need of such a tool :).

The tool that I have recently found seems to be working great. The password reset did not work, but I was able to set a blank password, logon, and reset the password using computer management.
It's called "Offline NT Password & Registry Editor", visit its website and download it here.
This tool includes a lot of recent raid controller drivers which is necessary, so it detects your Windows installation partition.

Before using this tool, be sure that Windows was shutdown properly, or you will not be able to perform any actions!
If that's the case, the tool will warn you and you'll have to reboot in safe mode, and perform a clean shutdown twice.
Besides that, the tool is pretty self-explanatory, but the author has also written a walkthrough.

I was able to set a blank password on a Windows Server 2003 x64 edition with this tool, which was a great help!
As always, keep in mind that resetting a password can cause problems if not used carefully, for example in cases where EFS is used, those files will become inaccessible.