十八、自然语言处理

自然语言处理（NLP）是使用计算机分析文本数据的方法。

这是维基百科上的自然语言处理。

NLTK 是用于文本分析的主要 Python 模块。

NLTK 组织网站在，他们有一整本教程在这里。

NLTK

NLTK 为超过 50 种语料库和词汇资源（如 WordNet）提供了易于使用的界面，以及一套用于分类，分词，词干提取，标注，解析和语义推理的文本处理库，用于工业级 NLP 库的包装器，和活跃的讨论论坛。

在这个笔记本中，我们将使用 NLTK 包中的一些有用功能来完成一些基本的文本分析。

要处理文本数据，通常需要使用语料库 - 文本数据集进行比较。 NLTK 有许多这样的数据集可用，但默认情况下不会安装它们（因为它们的完整集合会非常大）。下面我们将下载其中一些数据集。

  2. # 请返回此单元格，取消注释，然后运行此代码。
  3. # 这段代码赋予 python 写入磁盘的权限（如果它还没有这样做的权限）。
  4. import ssl
  6. try:
  7.     _create_unverified_https_context = ssl._create_unverified_context
  8. except AttributeError:
  9.     pass
  10. else:
  11.     ssl._create_default_https_context = _create_unverified_https_context
  14. # 从 NLTK 下载一些有用的数据文件
  15. nltk.download('punkt')
  16. nltk.download('stopwords')
  17. nltk.download('averaged_perceptron_tagger')
  18. nltk.download('maxent_ne_chunker')
  19. nltk.download('words')
  20. nltk.download('treebank')
  22. '''
  23. [nltk_data] Downloading package punkt to /Users/tom/nltk_data...
  24. [nltk_data]   Package punkt is already up-to-date!
  25. [nltk_data] Downloading package stopwords to /Users/tom/nltk_data...
  26. [nltk_data]   Package stopwords is already up-to-date!
  27. [nltk_data] Downloading package averaged_perceptron_tagger to
  28. [nltk_data]     /Users/tom/nltk_data...
  29. [nltk_data]   Package averaged_perceptron_tagger is already up-to-
  30. [nltk_data]       date!
  31. [nltk_data] Downloading package maxent_ne_chunker to
  32. [nltk_data]     /Users/tom/nltk_data...
  33. [nltk_data]   Package maxent_ne_chunker is already up-to-date!
  34. [nltk_data] Downloading package words to /Users/tom/nltk_data...
  35. [nltk_data]   Package words is already up-to-date!
  36. [nltk_data] Downloading package treebank to /Users/tom/nltk_data...
  37. [nltk_data]   Package treebank is already up-to-date!
  39. True
  40. '''
  43. # 设置一些要测试的数据的测试句子
  44. sentence = "UC San Diego is a great place to study cognitive science."

分词是将文本数据拆分为“标记”的过程，这些标记是有意义的数据片段。

分词的更多信息在。

词可以在不同的级别完成 - 例如，你可以将文本分词为句子，和/或分词为单词。

  1. # 在单词级别对我们的句子进行分词
  2. tokens = nltk.word_tokenize(sentence)
  5. # 查看单词分词后的数据
  6. print(tokens)
  8. # ['UC', 'San', 'Diego', 'is', 'a', 'great', 'place', 'to', 'study', 'cognitive', 'science', '.']

这里是维基百科上的词性标注。

  1. # 对我们的句子进行词性标注
  2. tags = nltk.pos_tag(tokens)
  5. # 检查我们的数据的 POS 标签
  6. print(tags)
  8. # [('UC', 'NNP'), ('San', 'NNP'), ('Diego', 'NNP'), ('is', 'VBZ'), ('a', 'DT'), ('great', 'JJ'), ('place', 'NN'), ('to', 'TO'), ('study', 'VB'), ('cognitive', 'JJ'), ('science', 'NN'), ('.', '.')]
  11. # 查看描述所有缩写含义的文档
  12. nltk.help.upenn_tagset()
  14. '''
  15. $: dollar
  16.     $ -$ --$ A$ C$ HK$ M$ NZ$ S$ U.S.$ US$
  17. '': closing quotation mark
  18.     ' ''
  19. (: opening parenthesis
  20.     ( [ {
  21. ): closing parenthesis
  22.     ) ] }
  23. ,: comma
  24.     ,
  25. --: dash
  26.     --
  27. .: sentence terminator
  28.     . ! ?
  29. :: colon or ellipsis
  30.     : ; ...
  31. CC: conjunction, coordinating
  32.     & 'n and both but either et for less minus neither nor or plus so
  33.     therefore times v. versus vs. whether yet
  34. CD: numeral, cardinal
  35.     mid-1890 nine-thirty forty-two one-tenth ten million 0.5 one forty-
  36.     seven 1987 twenty '79 zero two 78-degrees eighty-four IX '60s .025
  37.     fifteen 271,124 dozen quintillion DM2,000 ...
  38. DT: determiner
  39.     all an another any both del each either every half la many much nary
  40.     neither no some such that the them these this those
  41. EX: existential there
  42.     there
  43. FW: foreign word
  44.     lutihaw alai je jour objets salutaris fille quibusdam pas trop Monte
  45.     terram fiche oui corporis ...
  46. IN: preposition or conjunction, subordinating
  47.     astride among uppon whether out inside pro despite on by throughout
  48.     below within for towards near behind atop around if like until below
  50. JJ: adjective or numeral, ordinal
  51.     third ill-mannered pre-war regrettable oiled calamitous first separable
  52.     ectoplasmic battery-powered participatory fourth still-to-be-named
  53.     multilingual multi-disciplinary ...
  54. JJR: adjective, comparative
  55.     bleaker braver breezier briefer brighter brisker broader bumper busier
  56.     calmer cheaper choosier cleaner clearer closer colder commoner costlier
  57.     cozier creamier crunchier cuter ...
  58. JJS: adjective, superlative
  59.     calmest cheapest choicest classiest cleanest clearest closest commonest
  60.     corniest costliest crassest creepiest crudest cutest darkest deadliest
  61.     dearest deepest densest dinkiest ...
  62. LS: list item marker
  63.     A A. B B. C C. D E F First G H I J K One SP-44001 SP-44002 SP-44005
  64.     SP-44007 Second Third Three Two * a b c d first five four one six three
  65.     two
  66. MD: modal auxiliary
  67.     can cannot could couldn't dare may might must need ought shall should
  68.     shouldn't will would
  69. NN: noun, common, singular or mass
  70.     common-carrier cabbage knuckle-duster Casino afghan shed thermostat
  71.     investment slide humour falloff slick wind hyena override subhumanity
  72.     machinist ...
  73. NNP: noun, proper, singular
  74.     Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos
  75.     Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA
  76.     Shannon A.K.C. Meltex Liverpool ...
  77. NNPS: noun, proper, plural
  78.     Americans Americas Amharas Amityvilles Amusements Anarcho-Syndicalists
  79.     Andalusians Andes Andruses Angels Animals Anthony Antilles Antiques
  80.     Apache Apaches Apocrypha ...
  81. NNS: noun, common, plural
  82.     undergraduates scotches bric-a-brac products bodyguards facets coasts
  83.     divestitures storehouses designs clubs fragrances averages
  84.     subjectivists apprehensions muses factory-jobs ...
  85. PDT: pre-determiner
  86.     all both half many quite such sure this
  87. POS: genitive marker
  88.     ' 's
  89. PRP: pronoun, personal
  90.     hers herself him himself hisself it itself me myself one oneself ours
  91.     ourselves ownself self she thee theirs them themselves they thou thy us
  92. PRP$: pronoun, possessive
  93.     her his mine my our ours their thy your
  94. RB: adverb
  95.     occasionally unabatingly maddeningly adventurously professedly
  96.     stirringly prominently technologically magisterially predominately
  97.     swiftly fiscally pitilessly ...
  98. RBR: adverb, comparative
  99.     further gloomier grander graver greater grimmer harder harsher
  100.     healthier heavier higher however larger later leaner lengthier less-
  101.     perfectly lesser lonelier longer louder lower more ...
  102. RBS: adverb, superlative
  103.     best biggest bluntest earliest farthest first furthest hardest
  104.     heartiest highest largest least less most nearest second tightest worst
  105. RP: particle
  106.     aboard about across along apart around aside at away back before behind
  107.     by crop down ever fast for forth from go high i.e. in into just later
  108.     low more off on open out over per pie raising start teeth that through
  109.     under unto up up-pp upon whole with you
  110. SYM: symbol
  111.     % & ' '' ''. ) ). * + ,. < = > @ A[fj] U.S U.S.S.R * ** ***
  112. TO: "to" as preposition or infinitive marker
  113.     to
  114. UH: interjection
  115.     Goodbye Goody Gosh Wow Jeepers Jee-sus Hubba Hey Kee-reist Oops amen
  116.     huh howdy uh dammit whammo shucks heck anyways whodunnit honey golly
  117.     man baby diddle hush sonuvabitch ...
  118. VB: verb, base form
  119.     ask assemble assess assign assume atone attention avoid bake balkanize
  120.     bank begin behold believe bend benefit bevel beware bless boil bomb
  121.     boost brace break bring broil brush build ...
  122. VBD: verb, past tense
  123.     dipped pleaded swiped regummed soaked tidied convened halted registered
  124.     cushioned exacted snubbed strode aimed adopted belied figgered
  125.     speculated wore appreciated contemplated ...
  126. VBG: verb, present participle or gerund
  127.     telegraphing stirring focusing angering judging stalling lactating
  128.     hankerin' alleging veering capping approaching traveling besieging
  129.     encrypting interrupting erasing wincing ...
  130. VBN: verb, past participle
  131.     multihulled dilapidated aerosolized chaired languished panelized used
  132.     experimented flourished imitated reunifed factored condensed sheared
  133.     unsettled primed dubbed desired ...
  134. VBP: verb, present tense, not 3rd person singular
  135.     predominate wrap resort sue twist spill cure lengthen brush terminate
  136.     appear tend stray glisten obtain comprise detest tease attract
  137.     emphasize mold postpone sever return wag ...
  138. VBZ: verb, present tense, 3rd person singular
  139.     bases reconstructs marks mixes displeases seals carps weaves snatches
  140.     slumps stretches authorizes smolders pictures emerges stockpiles
  141.     seduces fizzes uses bolsters slaps speaks pleads ...
  142. WDT: WH-determiner
  143.     that what whatever which whichever
  144. WP: WH-pronoun
  145.     that what whatever whatsoever which who whom whosoever
  146.     whose
  147. WRB: Wh-adverb
  148.     how however whence whenever where whereby whereever wherein whereof why
  150. '''

命名实体识别旨在用与相关的实体类型标记单词。

这里是上的命名实体识别。

“停止词”是一种语言中最常见的词语，我们经常希望在文本分析之前将其过滤掉。

这里是维基百科上的停止词。

  1. # 查看英语中的停止词语料库
  2. print(nltk.corpus.stopwords.words('english'))
  4. # ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]

文本编码

NLP的关键组件之一是决定如何编码文本数据。

常见的编码是：

  • 词袋（BoW）
    • 文本被编码为单词和频率的集合
  • 词频-逆向文件频率（TF-IDF）
    • TF-IDF是一种加权，用于存储单词和语料库中的共性关系。

我们将浏览 BoW 和 TF-IDF 文本编码的示例。

  1. # 导入
  2. %matplotlib inline
  4. # 标准 Python 有一些有用的字符串工具
  5. import string
  7. # 集合是标准 Python 的一部分，包含一些有用的数据对象
  8. from collections import Counter
  10. import numpy as np
  11. import matplotlib.pyplot as plt
  13. # Scikit-learn 有一些有用的 NLP 工具，例如 TFIDF 向量化器
  14. from sklearn.feature_extraction.text import TfidfVectorizer

我们将要查看的数据是 BookCorpus 数据集的一小部分。原始数据集可在此处找到：。

原始数据集是从超过 11,000 本书中收集的，并且已经在句子和单词级别上进行了分词。这里提供和使用的小子集包含前 10,000 个句子。

  1. # 加载数据
  2. with open('files/book10k.txt', 'r') as f:
  3.     sents = f.readlines()
  6. # 查看数据 - 打印出第一个和最后一个句子，作为示例
  7. print(sents[0])
  8. print(sents[-1])
  10. '''
  11. the half-ling book one in the fall of igneeria series kaylee soderburg copyright 2013 kaylee soderburg all rights reserved .
  13. alejo was sure the fact that he was nervously repeating mass along with five wrinkly , age-encrusted spanish women meant that stalin was rethinking whether he was going to pay the price .
  14. '''
  17. # 预处理：从句子中删除所有额外的空格
  18. sents = [sent.strip() for sent in sents]

我们首先看一下文档中的单词频率，然后打印出频率最高的前 10 个单词。

如果你滚动上面的单词列表，你可能会注意到的一点是，它仍然包含标点符号。我们删除那些。

  1. # 'string' 模块（标准库）有一个有用的标点符号列表
  2. print(string.punctuation)
  4. # !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
  7. # 从计数对象中删除所有标点符号标记
  8. for punc in string.punctuation:
  9.     if punc in counts:
  10.         counts.pop(punc)
  13. # 获得前 10 个最常用的单词
  14. top10 = counts.most_common(10)
  17. # 提取顶部单词，并计数
  18. top10_words = [it[0] for it in top10]
  19. top10_counts = [it[1] for it in top10]
  22. # 绘制文本中最常用单词的条形图
  23. plt.barh(top10_words, top10_counts)
  24. plt.title('Term Frequency');
  25. plt.xlabel('Frequency');

正如我们所看到的，文档中出现了the，was，a等等。

对于弄清楚这些文档的内容，或者作为使用和理解这些文本数据的方式，这些经常出现的单词并不是很有用。

  1. # 丢弃所有停止词
  2. for stop in nltk.corpus.stopwords.words('english'):
  3.     if stop in counts:
  4.         counts.pop(stop)
  7. # 获取删除停止词的数据中，前 20 个最常用单词
  8. top20 = counts.most_common(20)
  11. # 绘制文本中最常用单词的条形图
  12. plt.barh([it[0] for it in top20], [it[1] for it in top20])
  13. plt.title('Term Frequency');
  14. plt.xlabel('Frequency');

这看起来可能更相关/有用。我们可以继续探索这个 BoW 模型，但现在让我们转向，并使用 TFIDF 进行探索。

  1. # 初始化 TFIDF 对象
  2. tfidf = TfidfVectorizer(analyzer='word',
  3.                         sublinear_tf=True,
  4.                         max_features=5000,
  5.                         tokenizer=nltk.word_tokenize)
  8. # 将 TFIDF 转换应用于我们的数据
  9. # 请注意，这会接受句子并对其进行分词，然后应用 TFIDF
  10. tfidf_books = tfidf.fit_transform(sents).toarray()

TfidfVectorizer 将计算每个单词的逆文档频率（IDF）。

然后 TFIDF 计算为TF * IDF，用于降低频繁出现的单词的权重。该 TFIDF 存储在tfidf_books变量中，该变量是一个n_documents x n_words矩阵，用于以 TFIDF 表示来编码文档。

让我们首先为前 10 个最常出现的单词（来自第一次分析）中的每一个绘制 IDF。

png

我们将该绘图与以下绘图进行比较，该绘图显示 IDF 最高的前 10 个单词。

  1. # 获得 IDF 得分最高的单词
  2. inds = np.argsort(tfidf.idf_)[::-1][:10]
  3. top_IDF_tokens = [list(tfidf.vocabulary_)[ind] for ind in inds]
  4. top_IDF_scores = tfidf.idf_[inds]
  7. # 绘制 IDF 得分最高的单词
  8. plt.barh(top_IDF_tokens, top_IDF_scores)
  9. plt.xlabel('IDF Score');

正如我们所看到的，与更罕见的单词相比，文档中经常出现的单词获得的 IDF 分数非常低。

在 TF-IDF 之后，我们成功地减少了文档中频繁出现的单词的权重。这允许我们通过最独特的单词来表示文档，这可以是表示文本数据的更有用的方式。