• VMware

    Learn about VMware virtualization for its products like vsphere ESX and ESXi, vCenter Server, VMware View, VMware P2V and many more

  • Linux

    Step by step configuration tutorials for many of the Linux services like DNS, DHCP, FTP, Samba4 etc including many tips and tricks in Red Hat Linux.

  • Database

    Learn installation and configuration of databases like Oracle, My SQL, Postgresql, etc including many other related tutorials in Linux.

  • How to configure NTP client to sync with NTP server during system startup (boot) in (RHEL 7 / CentOS 7) Linux

    While working on NTP server on my setup I realised that in Red Hat Enterprise Linux when we a service restart for NTPD or chronyd, the local time does not syncs with the NTP server immediately and it takes some time before syncing the local clock

    So we have to manually sync the clock with NTP server using ntpdate

    But what if a machine went for a reboot with incorrect date and time?

    I also wanted to validate this scenario and it turns out NTPD in Red Hat 7 is very poor in handling this compared to SuSE Enterprise Linux 11.

    To validate this

    I changed the date of my setup
    server2:~ # date --set "01 Jan 2018"
    Mon Jan  1 00:00:00 IST 2018

    Logged a message to understand the time of reboot
    server2:~ # logger rebooting

    My NTP server is configured properly and is enabled
    server2:~ # systemctl is-enabled ntpd
    enabled

    Next reboot the blade to see if it takes the new date and time
    server2:~ # reboot
    PolicyKit daemon disconnected from the bus.
    We are no longer a registered authentication agent.

    From the syslog we see during the shutdown stage the logs were stored with wrong time
    Jan  1 00:00:05 server2 deepak: rebooting
    Jan  1 00:00:21 server2 systemd: Stopped Dump dmesg to /var/log/dmesg.
    Jan  1 00:00:21 server2 systemd: Stopping Dump dmesg to /var/log/dmesg...

    Fortunately our hwclock had the correct date and time hence once the system started up it had proper time
    Jan  1 00:00:21 server2 systemd: Stopping custom security scripts at start-up...
    Jan  1 00:00:21 server2 systemd: Stopping OpenSSH server daemon...
    Jan  1 00:00:21 server2 systemd: Stopping (null)...
    Feb 20 15:34:50 server2 kernel: microcode: microcode updated early to revision 0x3a, date = 2017-01-30
    Feb 20 15:34:50 server2 kernel: Initializing cgroup subsys cpuset
    Feb 20 15:34:50 server2 kernel: Initializing cgroup subsys cpu
    Feb 20 15:34:50 server2 kernel: Initializing cgroup subsys cpuacct

    So let us change the date and time of our hwclock
    server2:~ # hwclock --set --date  "01 jan 2018"

    server2:~ # hwclock ;date
    Mon 01 Jan 2018 12:00:14 AM IST  -0.547321 seconds
    Tue Feb 20 15:41:16 IST 2018

    server2:~ # logger rebooting again

    server2:~ # reboot

    From our syslog as we see the node went for shutdown with proper time since our date had correct time (only hwclock was changed)
    Feb 20 15:41:07 server2 deepak: rebooting again
    Feb 20 15:41:17 server2 systemd: Stopping custom firewall configuration using iptables...
    Feb 20 15:41:17 server2 systemd: Unmounting RPC Pipe File System...
    Feb 20 15:41:17 server2 systemd: Stopping system-systemd\x2dfsck.slice.
    Feb 20 15:41:17 server2 systemd: Stopped Stop Read-Ahead Data Collection 10s After Complete

    Next once again as we see the system attempts to startup, hwclock time is used for logging purpose which for our case was incorrectly set before reboot
    Feb 20 15:41:17 server2 systemd: Stopping Pro-active monitoring utility for unix systems...
    Feb 20 15:41:17 server2 systemd: Stopping LSB: LinuxICCM Scanner...
    Feb 20 15:41:17 server2 systemd: Stopping Self Monitoring and Reporting Technology (SMART) Daemon...
    Jan  1 00:02:57 server2 journal: Runtime journal is using 8.0M (max allowed 4.0G, trying to leave 4.0G free of 62.8G available â current limit 4.0G).
    Jan  1 00:02:57 server2 kernel: microcode: microcode updated early to revision 0x3a, date = 2017-01-30
    Jan  1 00:02:57 server2 kernel: Initializing cgroup subsys cpuset
    Jan  1 00:02:57 server2 kernel: Initializing cgroup subsys cpu
    Jan  1 00:02:57 server2 kernel: Initializing cgroup subsys cpuacct

    But NTPD never syncs the local clock date and time with NTP server until our node is completely UP

    Once the node is up after few minutes I observed the local time syncing with NTP
    Jan  1 00:04:04 server2 rsyslogd: action 'action 0' resumed (module 'builtin:ompipe') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
    Jan  1 00:04:04 server2 rsyslogd: action 'action 0' resumed (module 'builtin:ompipe') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
    Jan  1 00:04:04 server2 systemd: Started Stop Read-Ahead Data Collection.
    Feb 20 15:47:32 server2 systemd: Time has been changed
    Feb 20 15:47:32 server2 rsyslogd: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
    Feb 20 15:48:19 server2 su: (to root) deepak on pts/0

    So you see NTPD is really very slow at this and not very good.
    But actually we have few options with NTPD which can be used to tweak this behaviour

    We can use ntpdate to forcefully sync the localclock using NTP server before NTPD starts

    In Red Hat 7 we have below unit file for ntpdate.service
    # systemctl cat ntpdate
    # /usr/lib/systemd/system/ntpdate.service
    [Unit]
    Description=Set time via NTP
    After=syslog.target network.target nss-lookup.target
    Before=time-sync.target
    Wants=time-sync.target

    [Service]
    Type=oneshot
    ExecStart=/usr/libexec/ntpdate-wrapper
    RemainAfterExit=yes

    [Install]
    WantedBy=multi-user.target

    By default this service is disabled, so enable this service
    # systemctl enable ntpdate.service
    Created symlink from /etc/systemd/system/multi-user.target.wants/ntpdate.service to /usr/lib/systemd/system/ntpdate.service.

    Next modify the NTP server which ntpdate would sync in /etc/ntp/step-tickers
    server2:~ # cat /etc/ntp/step-tickers
    # List of NTP servers used by the ntpdate service.

    192.168.1.100

    Next lets again change the hwclock date and time
    server2:~ # hwclock --set --date  "01 jan 2018"

    Add a logger message to monitor the logs
    server2:~ # logger "rebooting to test ntpdate"

    Lastly reboot the blade
    server2:~ # reboot

    Now lets monitor our syslog once my node is UP
    Here the node went for reboot
    Feb 20 15:54:03 server2 deepak: rebooting to test ntpdate
    Feb 20 15:54:06 server2 systemd: Stopping custom firewall configuration using iptables...

    Here my system started to boot
    Feb 20 15:54:06 server2 systemd: Stopping Serial Getty on ttyS0...
    Feb 20 15:54:06 server2 systemd: Stopping Getty on tty1...
    Jan  1 00:02:56 server2 journal: Runtime journal is using 8.0M (max allowed 4.0G, trying to leave 4.0G free of 62.8G available â current limit 4.0G).
    Jan  1 00:02:56 server2 kernel: microcode: microcode updated early to revision 0x3a, date = 2017-01-30
    Jan  1 00:02:56 server2 kernel: Initializing cgroup subsys cpuset
    Jan  1 00:02:56 server2 kernel: Initializing cgroup subsys cpu
    Jan  1 00:02:56 server2 kernel: Initializing cgroup subsys cpuacct

    Now since we enabled our ntpdate service I observed that during the system startup ntpdate does a force sync with NTP server
    Jan  1 00:03:16 server2 hp-ams[2476]: amsHelper Started . .
    Jan  1 00:03:16 server2 systemd: Started HP Agentless Management Service daemon.
    Jan  1 00:03:16 server2 cntdb.security: - Setup B&R chain ...
    Feb 20 15:57:11 server2 systemd: Time has been changed
    Feb 20 15:57:11 server2 ntpdate[2612]: step time server 192.168.1.100 offset 4377234.731233 sec
    Feb 20 15:57:11 server2 systemd: Started Set time via NTP.

    But again I do not like such dependency and my expectation is that the very first message which gets logged from hwclock must be proper so we have to make sure that before system goes down the hwclock is in sync with local clock





    IMPORTANT NOTE: With Red Hat Enterprise 7 when the system clock is synchronized by the Network Time Protocol (NTP) or Precision Time Protocol (PTP), the kernel automatically synchronizes the hardware clock to the system clock every 11 minutes.
    But still considering worst case scenario the hwclock was not set properly with system clock and system went for a reboot with wrong hwclock time?

    My solution for this was to create a systemd unit file which will be called before shutdown and during startup to make sure
    1. First local time syncs with NTP server
    2. then hwclock is synced with local clock
    So with this during reboot our node will always come up with proper time
    server1:~ # systemctl cat set_my_clock.service
    # /usr/lib/systemd/system/set_my_clock.service
    [Unit]
    Description=Syncing system and hardware clock

    [Service]
    Type=oneshot
    ExecStart=/etc/init.d/set_my_clock start
    ExecStop=/etc/init.d/set_my_clock stop
    RemainAfterExit=true

    [Install]
    WantedBy=basic.target

    NOTE: You can also create this file under "/etc/systemd/system" as that is the recommended location from Red Hat

    Next create the main script which will do the work
    NOTE: I have enabled debugging to monitor the execution during reboot, you can remove the set -x
    # cat /etc/init.d/set_my_clock
    #!/bin/bash
    set -x
    case "$1" in

      start|stop)
        echo "Set Sys time according to Hardware Clock";
    # This will force sync the local system clock with NTP server
        /sbin/ntpdate -u 192.168.1.00 192.168.1.101;

    # This will sync our hwclock with the system clock time
        /sbin/hwclock --systohc;
        ;;

       *)
             echo "Usage: $0 {start|stop}"
             exit 1
          ;;
    esac

    echo "done !"
    exit 0

    Enable this script
    # systemctl enable set_my_clock.service

    It is time to validate our fix
    Let's change the hwclock date and time
    server1 # hwclock --set --date "05 Feb 2018"

    server1 # date --set "01 Jan 2018"
    Mon Jan  1 00:00:00 IST 2018

    Lets have a logger message to monitor the logs
    server1:~ # logger rebooting

    Reboot the node
    server1:~ # reboot

    Here the shutdown started
    Jan  1 00:00:13 server1 deepak: rebooting
    Jan  1 00:00:15 server1 systemd: Stopping Availability of block devices...
    Jan  1 00:00:15 server1 systemd: Stopped Stop Read-Ahead Data Collection 10s After Completed Startup.

    Our script was executed properly during system shutdown but then as you see it is unable to change the current time of the syslog
    Jan  1 00:00:15 server1 set_my_clock: + case "$1" in
    Jan  1 00:00:15 server1 set_my_clock: + echo 'Set Sys time according to Hardware Clock'
    Jan  1 00:00:15 server1 set_my_clock: Set Sys time according to Hardware Clock
    Jan  1 00:00:15 server1 set_my_clock: + /sbin/ntpdate -u 192.168.1.100 192.168.1.101

    although the system is coming up with proper time so we know our hwclock was synced properly with system clock above
    Feb 20 16:41:54 server1 kernel: Initializing cgroup subsys cpuset
    Feb 20 16:41:54 server1 kernel: Initializing cgroup subsys cpu
    Feb 20 16:41:54 server1 kernel: Initializing cgroup subsys cpuacct

    Again during startup
    Feb 20 16:42:02 server1 systemd: Starting Syncing system and hardware clock...
    Feb 20 16:42:02 server1 set_my_clock: + case "$1" in
    Feb 20 16:42:02 server1 set_my_clock: + echo 'Set Sys time according to Hardware Clock'
    Feb 20 16:42:02 server1 set_my_clock: Set Sys time according to Hardware Clock
    Feb 20 16:42:02 server1 set_my_clock: + /sbin/ntpdate -u 192.168.1.100 192.168.1.101
    Feb 20 16:42:02 server1 set_my_clock: 20 Feb 16:42:02 ntpdate[1118]: no servers can be used, exiting
    Feb 20 16:42:02 server1 set_my_clock: + /sbin/hwclock --systohc

    So with this now our system will come up with proper date and time and we need not be dependent on NTP service daemon for system startup.

    I hope the article was useful.

    Deepak Prasad

    Deepak Prasad is a techie and an author who is still trying to survive in this IT generation with very little knowledge he has on Linux/Unix, VMware, SAN Storage, Automation, networking etc

    You can follow him on Facebook or Google+

    Do you also have something to share here?

    Become an author and be a part of our GoLinuxHub Team, Click here for more information
    How to configure NTP client to sync with NTP server during system startup (boot) in (RHEL 7 / CentOS 7) Linux How to configure NTP client to sync with NTP server during system startup (boot) in (RHEL 7 / CentOS 7) Linux Reviewed by Deepak Prasad on Tuesday, February 20, 2018 Rating: 5

    No comments:

    Powered by Blogger.