Why aren't telnet bots finishing the three-way handshake?
They are performing port scans, not trying to establish a connection.
This specific scanning technique is called a SYN scan. The idea is that you don't have to complete the entire TCP handshake to find out if a port is open. A SYN scan is often preferred to establishing a full connection because it's faster and less noticeable. From the nmap guide:
SYN scan is the default and most popular scan option for good reasons. It can be performed quickly, scanning thousands of ports per second on a fast network not hampered by restrictive firewalls. It is also relatively unobtrusive and stealthy since it never completes TCP connections. [...]
This technique is often referred to as half-open scanning, because you don't open a full TCP connection. You send a SYN packet, as if you are going to open a real connection and then wait for a response. A SYN/ACK indicates the port is listening (open), while a RST (reset) is indicative of a non-listener. If no response is received after several retransmissions, the port is marked as filtered.
You can try to reproduce this behavior by attempting a SYN scan yourself using nmap and checking the Wireshark results:
nmap -sS yourhost
The bots can't possibly tell that it's not a real telnet service before they even finish connecting?
Correct, the disadvantage of this technique is that they can't really fingerprint the service. All they know is that the server has accepted a connection at that port.