UnicodeDecodeError

Q: How do I find out which encoding the data really uses?

Look at the bad bytes: a single byte 0xe9, 0xe8 or 0xfc in otherwise ASCII text is Latin-1/cp1252 (é, è, ü); ff fe or fe ff at the start is UTF-16. If the source can tell you (HTTP Content-Type charset, XML declaration, file format spec), trust it over guessing. Third-party packages such as charset-normalizer can guess when nothing else helps.

Q: Why do I get 'charmap' codec can't decode byte on Windows?

open() without encoding= uses the locale encoding, which on many Windows systems is cp1252. cp1252 leaves five byte values undefined (0x81, 0x8d, 0x8f, 0x90, 0x9d), and UTF-8 files often contain them, so reading a UTF-8 file fails with 'charmap' codec can't decode byte 0x9d. Pass encoding='utf-8' to open(), or run Python in UTF-8 mode (python -X utf8).

Q: What is the difference between errors='replace', 'ignore' and 'backslashreplace'?

replace puts U+FFFD (�) in place of each bad sequence, ignore drops it, backslashreplace inserts the byte values as \xNN text. errors='surrogateescape' maps each bad byte to a lone surrogate so the text can be encoded back to the identical bytes. The default, 'strict', raises UnicodeDecodeError.

Q: Is UnicodeDecodeError a subclass of ValueError?

Yes: UnicodeDecodeError → UnicodeError → ValueError. except ValueError catches it, which also means a broad except ValueError around parsing code will swallow decoding problems — put except UnicodeDecodeError first if you need to treat it differently.

The bytes are not valid in the codec you picked — almost always the data was written in a different encoding than you assumed. Positions are byte offsets, not character indexes.

InheritsBaseException›Exception›ValueError›UnicodeError›UnicodeDecodeError

Type / value exceptionPython 3 (all)Live demo

UnicodeDecodeError(encodingencoding — Codec name as shown in the message, e.g. 'utf-8'. Stored in e.encoding.type: str · required, objectobject — The whole input being decoded (any bytes-like object is stored as bytes). Stored in e.object.type: bytes · required, startstart — Index of the first bad byte in object.type: int · required, endend — Index just after the last bad byte. end - start == 1 gives the "byte 0x.. in position N" wording; a longer range gives "bytes in position S-E".type: int · required, reasonreason — Why decoding failed, e.g. 'invalid start byte'.type: str · required)

Raised by

b.decode('utf-8'), str(b, 'ascii'), open(p, encoding='utf-8').read()

Message

'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte

Quick fix

decode with the real encoding, or errors='replace'

Watch out

position is a byte offset into e.object, not an index into the text

Demo

Live evaluation

The classic wrong-codec bug: text saved as Latin-1 (one byte per accented letter) and read back as UTF-8.

Try:

Inputs

textstrtext to round-trip

Code

'hello'.encode('latin-1').decode('utf-8')

Result

'hello'

In Trigger, é is the single byte 0xe9 in Latin-1. UTF-8 reads 0xe9 as the start of a 3-byte sequence: at the end of the data that is unexpected end of data, followed by a space it is an invalid continuation byte. '5 €' fails one step earlier — € does not exist in Latin-1, so encode() raises UnicodeEncodeError. In Handle, the cut-off emoji f0 9f 98 is one bad sequence: one U+FFFD, three escaped bytes.

Constructor

Name	Type	Required	Description
encoding	str	yes	Codec name as shown in the message, e.g. 'utf-8'. Stored in e.encoding.
object	bytes	yes	The whole input being decoded (any bytes-like object is stored as bytes). Stored in e.object.
start	int	yes	Index of the first bad byte in object.
end	int	yes	Index just after the last bad byte. end - start == 1 gives the "byte 0x.. in position N" wording; a longer range gives "bytes in position S-E".
reason	str	yes	Why decoding failed, e.g. 'invalid start byte'.

Attributes

Attribute	Type	Meaning
encoding	str	The codec that failed, e.g. 'utf-8' — or 'charmap' for table-based codecs such as cp1252.
object	bytes	The complete input that was being decoded.
start	int	Byte offset of the first invalid byte. object[start:end] is the offending slice.
end	int	Byte offset just past the invalid sequence.
reason	str	Short explanation: 'invalid start byte', 'invalid continuation byte', 'unexpected end of data', 'ordinal not in range(128)', …
args	tuple	All five constructor arguments, in order.

Common patterns

Always pass encoding= to open()

Without it, open() uses the platform's locale encoding (often cp1252 on Windows, UTF-8 elsewhere), so the same file can decode on one machine and fail on another.

with open(path, encoding='utf-8') as f:
    text = f.read()

Try UTF-8, fall back to a legacy codec

Valid UTF-8 rarely happens by accident, so try it first. cp1252 accepts almost any byte, which makes it a reasonable fallback for Western European legacy files.

def read_text(data: bytes) -> str:
    try:
        return data.decode('utf-8')
    except UnicodeDecodeError:
        return data.decode('cp1252', errors='replace')

Show where decoding failed

The attributes pinpoint the bad bytes — print a little context around them.

try:
    text = data.decode('utf-8')
except UnicodeDecodeError as e:
    context = e.object[max(0, e.start - 10):e.end + 10]
    raise ValueError(f'bad UTF-8 at byte {e.start}: {context!r}') from e

Round-trip unknown bytes with surrogateescape

errors=surrogateescape decodes every invalid byte to a lone surrogate and encodes it back to the same byte — lossless for file names and pass-through data.

text = data.decode('utf-8', errors='surrogateescape')
assert text.encode('utf-8', errors='surrogateescape') == data

Examples

1. Latin-1 byte read as UTF-8

b'caf\xe9'.decode('utf-8')

Returns

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data

2. Decode with the right codec

b'caf\xe9'.decode('latin-1')

Returns

'café'

3. Inspect the attributes

try: b'ab\xffcd'.decode('utf-8') except UnicodeDecodeError as e: info = (e.encoding, e.start, e.end, e.reason, e.object[e.start:e.end]) info

Returns

('utf-8', 2, 3, 'invalid start byte', b'\xff')

4. Positions count bytes

data = 'día'.encode('utf-8') + b'\xff' try: data.decode('utf-8') except UnicodeDecodeError as e: pos = (len('día'), e.start) pos

Returns

(3, 4)

5. Truncated multi-byte char

'😀'.encode('utf-8')[:3].decode('utf-8')

Returns

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-2: unexpected end of data

6. UTF-16 data read as UTF-8

'\ufeffhi'.encode('utf-16-le').decode('utf-8')

Returns

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

7. Reading a file

with open('menu.txt', 'wb') as f: f.write(b'caf\xe9 au lait') with open('menu.txt', encoding='utf-8') as f: f.read()

Returns

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte

8. cp1252 has holes: 'charmap'

b'\x9d'.decode('cp1252')

Returns

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 0: character maps to <undefined> decoding with 'cp1252' codec failed

Pitfalls

1. errors='ignore' silently loses letters

Suppressing the error does not fix the encoding mismatch — it deletes the characters that proved it. Find the real codec instead.

errors='ignore'

b'caf\xe9'.decode('utf-8', errors='ignore')

'caf'

The real codec

b'caf\xe9'.decode('cp1252')

'café'

2. Decoding a stream chunk by chunk

A chunk boundary can split a multi-byte character; decoding each chunk separately then fails even though the data is valid. An incremental decoder keeps the partial bytes for the next call.

chunk.decode()

data = 'é'.encode('utf-8')
data[:1].decode('utf-8') + data[1:].decode('utf-8')

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 0: unexpected end of data

Incremental decoder

import codecs
data = 'é'.encode('utf-8')
dec = codecs.getincrementaldecoder('utf-8')()
dec.decode(data[:1]) + dec.decode(data[1:])

'é'

3. str(bytes) does not decode

Trying to dodge the error with str() gives the repr of the bytes object — b'…' ends up inside your text.

str(data)

str(b'caf\xc3\xa9')

"b'caf\\xc3\\xa9'"

data.decode()

b'caf\xc3\xa9'.decode('utf-8')

'café'

When to use

Use it

Catch it where outside bytes (files, sockets, subprocess output) become text
Use its start/end/object attributes to report exactly which bytes are bad
Raise it from a custom codec (with all five arguments)

Reach for something else

Hiding it with errors='ignore' when the real problem is the wrong codec
Catching it just to retry with random encodings — latin-1 never fails, so it 'succeeds' with garbage
Decoding at all when the data is binary (images, zip, pickle) — keep it as bytes

Notes

CPython impl

Objects/unicodeobject.c + Objects/stringlib/codecs.h — the UTF-8 decoder reports the longest valid prefix of a broken sequence as one error (start..end)

Catch via

except UnicodeError (all three unicode errors) or except ValueError

charmap

Table-based codecs (cp1252, cp437, …) report their encoding as 'charmap' and the reason as 'character maps to <undefined>'; when called via bytes.decode() the traceback adds a note naming the real codec (decoding with 'cp1252' codec failed)

BOM

Data starting with ff fe or fe ff is UTF-16; ef bb bf is a UTF-8 BOM — decode with 'utf-8-sig' to drop it

FAQ

What does 'utf-8' codec can't decode byte 0x.. in position N mean? ▾

Byte number N (counting from 0) of the input is not valid UTF-8 at that point. The reason tells you more: invalid start byte means a byte that can never begin a character (0x80-0xc1, 0xf5-0xff — 0xff and 0xfe usually mean UTF-16 data); invalid continuation byte means a multi-byte sequence was cut short by an ordinary byte, the usual sign of Latin-1/cp1252 text; unexpected end of data means the input stops in the middle of a character.

How do I find out which encoding the data really uses? ▾

Why do I get 'charmap' codec can't decode byte on Windows? ▾

What is the difference between errors='replace', 'ignore' and 'backslashreplace'? ▾

Is UnicodeDecodeError a subclass of ValueError? ▾

UnicodeDecodeError

Demo

Constructor

Attributes

Common patterns

Examples

Pitfalls

When to use

Notes

Related

FAQ