Python Reading Until Null Character From Telnet

I am telneting to my server, which answers to me with messages and at the end of each message is appended hex00 (null character) which cannot be read. I tried searching through and

Solution 1:

I stumbled in this same problem when trying to get data from an RS232-TCP/IP Converter using telnet - the telnetlib would suppress every 0x00 from the message. As Fredrik Johansson well answered, it is the way telnetlib was implemented.

One solution would be to override the process_rawq() function from telnetlib's Telnet class that doesn't eat all the null characters:

import telnetlib
from telnetlib import IAC, DO, DONT, WILL, WONT, SE, NOOPT

def_process_rawq(self):
    """Alteração da implementação desta função necessária pois telnetlib suprime 0x00 e \021 dos dados lidos
    """
    buf = ['', '']
    try:
        while self.rawq:
            c = self.rawq_getchar()
            ifnot self.iacseq:
#                if c == theNULL:#                    continue#                if c == "\021":#                    continueif c != IAC:
                    buf[self.sb] = buf[self.sb] + c
                    continueelse:
                    self.iacseq += c
            eliflen(self.iacseq) == 1:
                # 'IAC: IAC CMD [OPTION only for WILL/WONT/DO/DONT]'if c in (DO, DONT, WILL, WONT):
                    self.iacseq += c
                    continue

                self.iacseq = ''if c == IAC:
                    buf[self.sb] = buf[self.sb] + c
                else:
                    if c == SB: # SB ... SE start.
                        self.sb = 1
                        self.sbdataq = ''elif c == SE:
                        self.sb = 0
                        self.sbdataq = self.sbdataq + buf[1]
                        buf[1] = ''if self.option_callback:
                        # Callback is supposed to look into# the sbdataq
                        self.option_callback(self.sock, c, NOOPT)
                    else:
                        # We can't offer automatic processing of# suboptions. Alas, we should not get any# unless we did a WILL/DO before.
                        self.msg('IAC %d not recognized' % ord(c))
            eliflen(self.iacseq) == 2:
                cmd = self.iacseq[1]
                self.iacseq = ''
                opt = c
                if cmd in (DO, DONT):
                    self.msg('IAC %s %d',
                        cmd == DO and'DO'or'DONT', ord(opt))
                    if self.option_callback:
                        self.option_callback(self.sock, cmd, opt)
                    else:
                        self.sock.sendall(IAC + WONT + opt)
                elif cmd in (WILL, WONT):
                    self.msg('IAC %s %d',
                        cmd == WILL and'WILL'or'WONT', ord(opt))
                    if self.option_callback:
                        self.option_callback(self.sock, cmd, opt)
                    else:
                        self.sock.sendall(IAC + DONT + opt)
    except EOFError: # raised by self.rawq_getchar()
        self.iacseq = ''# Reset on EOF
        self.sb = 0pass
    self.cookedq = self.cookedq + buf[0]
    self.sbdataq = self.sbdataq + buf[1]
telnetlib.Telnet.process_rawq = _process_rawq

then override the Telnet class' method:

telnetlib.Telnet.process_rawq = _process_rawq

This solved the problem for me.

Solution 2:

This code (http://www.opensource.apple.com/source/python/python-3/python/Lib/telnetlib.py) seems to just ignore null characters. Is that really correct behavior?

defprocess_rawq(self):
    """Transfer from raw queue to cooked queue.

    Set self.eof when connection is closed.  Don't block unless in
    the midst of an IAC sequence.

    """
    buf = ''try:
        while self.rawq:
            c = self.rawq_getchar()
            if c == theNULL:
                continue
:
:

process_rawq is then in turn called by e.g. read_until

defread_until(self, match, timeout=None):
    """Read until a given string is encountered or until timeout.

    When no match is found, return whatever is available instead,
    possibly the empty string.  Raise EOFError if the connection
    is closed and no cooked data is available.

    """
    n = len(match)
    self.process_rawq()
    :
    :

I also want to receive the null character. In my particular case it marks the end of a multiline message.

So the answer seems to be that this is expected behavior as the library code is written.

FWIW https://support.microsoft.com/en-us/kb/231866 states:

Communication is established using TCP/IP and is based on a Network Virtual Terminal (NVT). On the client, the Telnet program is responsible for translating incoming NVT codes to codes understood by the client's display device as well as for translating client-generated keyboard codes into outgoing NVT codes.

The NVT uses 7-bit codes for characters. The display device, referred to as a printer in the RFC, is only required to display the standard printing ASCII characters represented by 7-bit codes and to recognize and process certain control codes. The 7-bit characters are transmitted as 8-bit bytes with the most significant bit set to zero. An end-of-line is transmitted as a carriage return (CR) followed by a line feed (LF). If you want to transmit an actual carriage return, this is transmitted as a carriage return followed by a NUL (all bits zero) character.

and

Name     Code   DecimalValueFunctionNULL   NUL 0No operation