Python: Removing Particular Character (u"\u2610") From String

I have been wrestling with decoding and encoding in Python, and I can't quite figure out how to resolve my problem. I am looping over xml text files (sample) that are apparently co

Solution 1:

The problem is that you're mixing unicode and str. Whenever you do that, Python has to convert one to the other, which is does by using sys.getdefaultencoding(), which is usually ASCII, which is almost never what you want.*

If the exception comes from this line:

noboxes = noextrawhitespace.replace(u"\u2610", "")

… the fix is simple… except that you have to know whether noextrawhitespace is supposed to be a unicode object or a UTF-8-encoding str object). If the former, it's this:

noboxes = noextrawhitespace.replace(u"\u2610", u"")

If the latter, it's this:

noboxes = noextrawhitespace.replace(u"\u2610".encode('utf-8'), "")

But really, you have to get all of the strings consistent in your code; mixing the two up is going to cause problems in more places than this one.

---

Since I don't have your XML files to test, I wrote my own:

<xml>
    <text>abc&#9744;def</text>
</xml>

Then, I added these two lines to the bottom of your code (and a bit to the top to just open my file instead of globbing for whatever):

noboxes = noextrawhitespace.replace(u"\u2610".encode('utf-8'), "")
print noboxes

The output is now:

[<text>abc☐def</text>]
[<text>abc☐def</text>]
[<text>abcdef</text>]

So, I think that's what you want here.

---

* Sure sometimes you want ASCII… but those aren't usually the times when you have unicode objects…

Solution 2:

Give this a try:

noextrawhitespace.replace("\\u2610", "") 

I think you are just missing that extra '\'

This might also work.

print(noextrawhitespace.decode('unicode_escape').encode('ascii','ignore'))

Solution 3:

Reading your sample, the following are the non-ASCII characters in the document:

0x2223 DIVIDES
0x2022 BULLET
0x3009 RIGHT ANGLE BRACKET
0x25aa BLACK SMALL SQUARE
0x25ca LOZENGE
0x3008 LEFT ANGLE BRACKET
0x2014 EM DASH
0x2026 HORIZONTAL ELLIPSIS

\u2223 is the actual character in question in line 3682, and it is being used as a soft hyphen. The others are used in markup for tagging illegible characters, such as:

<GAP DESC="illegible" RESP="oxf" EXTENT="4+ letters" DISP="\u2022\u2022\u2022\u2022\u2026"/>

Here's some code to do what your code is attempting. Make sure to process in Unicode:

from bs4 import BeautifulSoup
import re

with open('k000039.000.xml') as f:
    soup = BeautifulSoup(f)  # BS figures out the encoding

text = u''.join(soup.strings)      # strings is a generator for just the text bits.
text = re.sub(ur'\s+',ur' ',text)  # Simplify all white space.
text = text.replace(u'\u2223',u'') # Get rid of the DIVIDES character.
print text

Output:

[[truncated]] reckon my self a Bridegroom too. Buckle. I doubt Kickey won't find him such. [Aside.] Mrs. Sago. Well,—poor Keckky's bound to good Behaviour, or she had lost quite her Puddy's Favour. Shall I for this repine at Fortune?—No. I'm glad at Heart that I'm forgiven so. Some Neighbours Wives have but too lately shown, When Spouse had left 'em all their Friends were flown. Then all you Wives that wou'd avoid my Fate. Remain contented with your present State FINIS.