Remote OS fingerprinting is carried out by noting the way the remote system responds to specifically crafted TCP packets. These can range from examining the default TCP window size in a packet, to measuring the amount of data in ICMP packets, and even gauging TCP initial sequence numbers. Similar to port scanning, there are several method to successfully fingerprint an OS. Querying the services running on a target machine is often the simplest way for OS fingerprinting.
Active stack fingerprint is based on the principle that an operating system's IP stack has its own unique way of responding to specially crafted TCP packets. This is due to the different interpretation that vendor abide with while implementing the TCP/IP stack on the particular OS. In active fingerprinting, a variety of malformed packets are sent to the remote host, and the responses compared to a database.
In Nmap, active stack fingerprint is done through eight tests:
- A TCP packet with the SYN and ECN -Echo flags enabled is sent to an open TCP port.
- A TCP packet with no flags enabled is sent to an open TCP port.
- A TCP packet with the URG, PSH, SYN and FIN flags enabled is sent to an open TCP port.
- A TCP packet with the ACK flag enabled is sent to an open TCP port.
- A TCP packet with the SYN flag enabled is sent to a closed TCP port.
- A TCP packet with the ACK flag enabled is sent to a closed TCP port.
- A TCP packet with the URG, PSH and FIN flag enabled is sent to a closed TCP port.
- A UDP packet is sent to a closed UDP port. The objective is to extract an ICMP port unreachable message back from the target machine.
Most operating system increment a system-wide IPID value for each packet they send. Others such as OpenBSD, use a random IPID and some system (like Linux) use an IPID of 0 in many cases where the "Don't Fragment" bit is not set. Windows does not put the IPID in network byte order, so it increments by 256 for each packet. Another number that can be sequenced for OS detection purposes is the TCP timestamp option values. Some system do not support the feature; others increment the value at frequencies of 2HZ, 100HZ or 1000HZ and still other return 0.