Dec 8, 2022

How to Identify and Solve Clock Skew Problems with NTP

Introduction

As a GlobalSign customer, sometimes, you will encounter issues with Certificate deployment and installation through the AEG. One that is recurrent is when a Certificate cannot be installed because a computer receives it outside the Certificate validity period. This happens because the start validity date is in the future compared to the time the computer reads from its local source or its NTP source. As a result, the computer does not validate the Certificate correctly and denies its installation.

Having an accurate time source in a Windows environment is important for several reasons, including:

• Kerberos authentication, which requires 5 minutes of accuracy between the client and the server
• Government regulations, for some of these 1 ms accuracy are required
• Cryptographic algorithms
• Distributed systems, like SQL/Exchange and Document DBs
• AD Replication for group policies and domain controller synchronization
• Industry standards, PCI require 1 second accuracy

In this article, we will discuss the main reasons around clock/time skew issues when you are making use of our AEG solution. The reasons are:

1. The AEG is processing two or more certificate requests from one user or device at almost the same time or just seconds later. From what you can gather in the Issued Certificates tab, the user or device appears to have a valid Certificate for itself, but certificate requests are still being processed without having a valid Certificate in the User/Computer Personal Store.
2. You do not see any of the issued Certificates either in the User or Computer Personal Store, but you may be able to see them in Published Certificates in Active Directory.
3. In the Event Viewer, you notice the error logs related to the Certificate installation. The error logs show the following message: "A required Certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file. 0x800b0101 (-2146762495 CERT_E_EXPIRED)".

Identifying Issues in NTP Clients

Simple Test

A simple way to test if a client is receiving accurate time information from the time source in your domain is to browse to a public website that hosts a time service, such as a World Time Server. Then, you can compare the time shown on the website against the time show on your computer. If the time on the local computer is nearly the same as the time on the website, you can assume that your time source is accurately delivering information to your device(s). Now you need to dive deep to determine if this time source is NTP-based or not.

Check if the Windows Time Service is running

This will allow you to know if you the time service is running on your machine. You need to go to click Start Menu, type Services, click Services. This will open the Services window, then you will need to scroll down until you find the Windows Time service. Make sure that the startup type is set to Automatic, and the service status is Running, as shown in the following figure:

Windows Time Service (W32TM) Status

The NTP service and settings are determined by the Windows Time Service, also known was W32Time Service. There are some useful commands that you can use to determine whether your computer is correctly configured to receive time information via NTP or not. Here we offer one example, open a command prompt as administrator and run the command:
w32tm /query /status

In the previous example, we can observe at least 4 parameters of interest:

1. Stratum – This is referred to the NTP hierarchy. 0 is for the global time source, which usually is a reliable global NTP server or atomic clock. Values 1-4 are considered healthy in the NTP hierarchy, because they are close to the clock source.
2. Last Successful Sync Time – The last time the computer received information from its time source.
3. Source – The time source, in this case it is the Local CMOS Clock.
4. Poll Interval – the time interval configured to obtain updates from the time source.

You can see how the computer is using the local CMOS clock. The problem with this time source is that it is linked to an internal clock that depends on the CMOS battery. If this battery loses its power, which will happen eventually, the time will not be provided accurately.

Windows Time Service (W32TM) Configuration

Another good source of information that you can use to understand your current time service settings is the following command:
w32tm /query /configuration

The command outputs are extensive, but the section about time providers can help you understand which time sources are configured in your device. In the previous example:

• The NTP Client is enabled (value of 1), and the NTP server is "europe.pool.ntp.local"
  • .local is an internal domain name that is being used for the purposes of this article only
• The flag 0x1 defined in the NTP server parameter indicates a special poll interval, which is 3600 seconds
• The NTP server is disabled (value of 0)
• The VMICTimeProvider is disabled (value of 0) – if this is disabled, it means time integration services for virtual machines is off and domain controllers oversee the network time

Fixing Issues in NTP Clients

In the previous example, it was clear that the client machine was configured as NTP client and the rest of parameters looked almost correct. However, the problem with that example is the NTP server. We configured "europe.pool.ntp.local" as the NTP server for the purposes of this article, but in reality the correct domain is "europe.pool.ntp.org".

Also, best practices will tell you that you should configure NTP clients to contact the best time provider for your domain instead of using a publicly accessible NTP server. In this sense, your domain NTP server would be a domain controller that has been selected for the role of the NTP server. Furthermore, depending on the hierarchy, you could have several NTP servers in your domain, these NTP servers would obtain the time information from your Primary Domain Controller (PDC). Finally, the PDC would connect to the publicly accessible time source for accurate time updates. In an Active Directory environment, the hierarchy would look like the one shown in the following figure:

Configure a Computer to Sync with the Domain Time Source

For this, you need to run a few commands on the NTP client machine or server, and it will automatically synchronize its time information to the most reliable source in your domain:

1. w32tm /config /syncfromflags:DOMHIER /update
2. net stop w32time & net start w32time
3. w32tm /resync /force
4. w32tm /query /status

After running these commands, you will get the following output:

Now, you can see that Stratum has changed to 4. This means there are 3 other time sources above this client machine, which makes sense because one of them is the domain controller, PS2-2019-DC. The time source has changed from "Local CMOS Clock" to "PS2-2019-DC". For security reasons, we do not show the FQDN.

Disable the Hyper-V Time Sync Service

If your environment allows for it, you can disable the Hyper-V Time Sync Service from providing samples to the W32TM service. If you do so, W32TM will get updates only from its domain time source as configured above. To disable the Hyper-V Time Sync Service, run the following command from an elevated command prompt:
reg add HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider /v Enabled /t reg_dword /d 0

Identifying Issues in NTP Servers

The following tests from the section Identifying Issues in NTP Clients can be performed in your NTP servers too:

1. Simple Test
2. Check if the Windows Time Service is running
3. Windows Time Service (W32TM) Status
4. Windows Time Service (W32TM) Configuration

Possible outcomes for the previous tests:

Fixing Issues in NTP Servers

There are several places where you can find the NTP settings on your server. We will cover some of the most important for you to verify on your environment.

Check the W32Time Service Parameters

W32Time parameters are stored in the registry location:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters

Two parameters that you should double check are NtpServer and Type:

• NtpServer – Contains the FQDN and special flags of the time source that your NTP server is using as reference. This should be your PDC, if you are using several NTP servers in your network; or a public, reliable NTP server, if your PDC is the only NTP server in the network.
• Type – Contains the NTP type, this should be set to NTP for NTP servers in your network, or NT5DS for NTP clients.

Check the W32Time Service Time Providers

W32Time time providers are stored in the registry location:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders

There are three sub-folders, and these contain individual information on each of time providers:

• NtpClient – Contains the NTP client's configuration. If you see this information on your NTP server, then it refers to how the NTP server is configured as client to retrieve updates from an NTP server. Two keys to check here are:
  • SpecialPollInterval – The poll interval at which the NTP server (acting as NTP client) will request another update to its time reference.
  • SpecialPollTimeRemaining – Contains the time reference's FQDN and the remaining time until the next update.
• NtpServer – Contains the NTP server's configuration. Make sure the Enabled key is set to 1.
• VMICTimeProvider – Contains the Hyper-V Time Sync Service's configuration. Make sure that Enabled key is set to 0.

Disable the Hyper-V Time Sync Service

If your environment allows for it, you can disable the Hyper-V Time Sync Service from providing samples to the W32TM service. If you do so, W32TM will get updates only from its domain time source as configured above. To disable the Hyper-V Time Sync Service, run the following command from an elevated command prompt:
reg add HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider /v Enabled /t reg_dword /d 0

Configure a PDC to Sync with an External NTP Server

For this, first you need to find a public NTP server that is close to you. You can check the following URL to find it:
http://support.ntp.org/bin/view/Servers/WebHome

NOTE: There are several other publicly available sources, so make sure you choose the one that is more convenient to you.

When you choose the public NTP time reference, run the following commands on the PDC:

1. w32tm /config /manualpeerlist:"public NTP server's FQDN,0x1" /syncfromflags:MANUAL /reliable:yes
2. w32tm /config /update
3. w32tm /resync
4. w32tm /resync /rediscover

You can then run the following two commands to verify the changes were applied correctly:

1. w32tm /query /status
2. w32tm /query /source

Configure a DC to Sync with the PDC

To configure additional DCs to sync with the PDC to obtain NTP information, you need to execute the following commands:

1. w32tm /config /syncfromflags:DOMHIER /update
2. net stop w32time & net start w32time
3. w32tm /resync /force
4. w32tm /query /source