2023年7月26日发(作者：)

python实现统计⽂本当中单词数量title: python实现统计⽂本当中单词数量date: 2018-6-30 15:12:43categories: Pythontags:

- python关于⽤实现统计⽂本当中单词数量这个功能，代码进⾏⼀步⼀步的升级。我做个回顾，或许以后还能写出更符合标准的代码。1 刚看完《python编程：从⼊门到实践》的时候写的代码学习python的时候在《python编程：从⼊门到实践书中第10章中学习了分析⽂本，当时写出了统计⼀个单词出现的频率：# 10-10 常见单词def row_count(filename): try: with open(filename) as f_obj: content = f_() except FileNotFoundError: msg = "The file " + filename + " does not exist." print(msg) else: content = e(',', ' ') content = e('.', ' ') content = e('-', ' ') content = ().lower() words = () # 统计row单词出现在⽂本中的次数 number = ('row') print('row : %d' % number)filename = 'Heart of 'row_count(filename)运⾏结果为：row : 9这个代码只是实现⼀个单词的出现次数的统计。并且还有⼀些问题。⽐如还有[a、(b这样的标点符号出现在单词中。2 写完⼀个单词的统计，⼜扩展了对所有单词都进⾏统计并排序当时写完课后作业时，想到了能不能对所有单词都进⾏统计并进⾏排序呢，于是上⽹查了⼀些资料，写出了下⾯的代码：from operator import itemgetterdef words_list(filename): try: with open(filename) as f_obj: content = f_() except FileNotFoundError: msg = "The file " + filename + " does not exist." print(msg) else: content = e(',', ' ') content = e('.', ' ') content = e('!', ' ') content = e('-', ' ') content = e('_', ' ') content = e('(', ' ') content = e(')', ' ') content = () words = [() for word in ()] return wordsdef count_results(filename): words_count = {} words = words_list(filename) words_count = words_ys(words) for word in words_(): number = (word) words_count[word] = number words_count = sorted(words_(), key=itemgetter(1), reverse=True) return words_countif __name__ == '__main__': filename = 'Heart of ' words_count = count_results(filename) for word, word_count in words_count[:10]: print('{0:<10} : {1}'.format(word, word_count))运⾏结果为：the : 2440of : 1492a : 1205and : 1045i : 1039to : 967was : 671in : 668he : 563had : 503还是有问题，还是有标点符号没有彻底清除，类似这种[a]、_a_。3 学完正则表达式以后使⽤正则表达式进⾏分割单词学会使⽤正则以后，试着⽤正则来清除标点符号：import refrom operator import itemgetterdef words_list(filename): try: with open(filename) as f_obj: content = f_() except FileNotFoundError: msg = "The file " + filename + " does not exist." print(msg) else: WORD_RE = e(r'W+') words = WORD_(()) return wordsdef count_results(filename): words_count = {} words = words_list(filename) words_count = words_ys(words) for word in words_(): number = (word) words_count[word] = number words_count = sorted(words_(), key=itemgetter(1), reverse=True) return words_countif __name__ == '__main__': filename = 'Heart of ' words_count = count_results(filename) for word, word_count in words_count[:10]: print('{0:<10} : {1}'.format(word, word_count))统计结果为：the : 2468of : 1496a : 1209i : 1153and : 1062to : 974in : 673was : 672he : 596it : 515还是有问题，w+包含[A-Za-z0-9_]，所以包含'_'的单词也会被统计出来，⽐如说‘_that_’也会被统计，并且区别于that。⾸先想到就是先把下划线(_)替换为空格即可，不过也可以把_that_和that当做是2种单词。4 还想到了⼀个不需要count的解法今天看另⼀本书《流畅的python》，在学习字典和集合的时候想到了另⼀种解法：def count_results(filename): try: with open(filename) as f_obj: content = f_() except FileNotFoundError: msg = "The file " + filename + " does not exist." print(msg) else: WORD_RE = e(r'w+') words_count = {} for match in WORD_er(()): word = () # occurrences = words_(word, []) # (word) # words_count[word] = occurrences words_ault(word, []).append(word) words_count = {word:len(value) for word, value in words_()} words_count = sorted(words_(), key=itemgetter(1), reverse=True) return words_countif __name__ == '__main__': filename = 'Heart of ' words_count = count_results(filename) for word, word_count in words_count[:10]: print('{0:<10} : {1}'.format(word, word_count))the : 2468of : 1496a : 1209i : 1153and : 1062to : 974in : 673was : 672he : 596it : 515这个⽅法⽐使⽤count的⽅法速度要快上不少。5 count⽅法和len⽅法速度⽐较import refrom operator import itemgetterdef words_list(filename): try: with open(filename) as f_obj: content = f_() except FileNotFoundError: msg = "The file " + filename + " does not exist." print(msg) else: WORD_RE = e(r'W+') words = WORD_(()) return wordsdef count_results(filename): words_count = {} words = words_list(filename) words = words_list(filename) words_count = words_ys(words) for word in words_(): number = (word) words_count[word] = number words_count = sorted(words_(), key=itemgetter(1), reverse=True) return words_countdef count_words(filename): try: with open(filename) as f_obj: content = f_() except FileNotFoundError: msg = "The file " + filename + " does not exist." print(msg) else: WORD_RE = e(r'w+') words_count = {} for match in WORD_er(()): word = () # occurrences = words_(word, []) # (word) # words_count[word] = occurrences words_ault(word, []).append(word) words_count = {word: len(value) for word, value in words_()} words_count = sorted(words_(), key=itemgetter(1), reverse=True) return words_countif __name__ == '__main__': import timeit def test_count_results(): filename = 'Heart of ' words_count = count_results(filename) return words_count def test_count_words(): filename = 'Heart of ' words_count = count_words(filename) return words_count time_1 = ('test_count_results()', setup="from __main__ import test_count_results") time_2 = ('test_count_words()', setup="from __main__ import test_count_words") print(time_(number=10)) print(time_(number=10))输出结果为：54.6624.5711结果发现差距还是挺⼤的。说⼀个我认为的原因：count需要遍历列表，遍历列表是很消耗时间的，时间复杂度为O(n)；⽽使⽤len()⽅法时，CPython会直接从内存来读取属性值，当然要快很多。以下摘⾃《流畅的python》第⼀章如何使⽤特殊⽅法⾸先明确⼀点，特殊⽅法的存在是为了被Python解释器调⽤的，你⾃⼰并不需要调⽤它们。也就是说没有my_object.__len()__这种写法，⽽应该使⽤len(my_object)。在执⾏len(my_object)的时候，如果my_object是⼀个⾃定义类的对象，那么Python会⾃⼰去调⽤其中由你实现的__len__⽅法。然⽽如果是Python内置的类型，⽐如列表（list）、字符串（str）、字节系列（bytearray）等，那么CPython会抄个近路，__len__实际上会直接返回PyVarObject⾥的ob_size属性。PyVarObject是表⽰内存中长度可变的内置对象的C语⾔结构体。直接读取这个值⽐调⽤⼀个⽅法要快很多。⽬前知识掌握的不太多，还不能很肯定。说不定以后还有其他的改进，可能未完待续。6 使⽤计数器（Counter）⽅法来实现（⽬前为⽌个⼈觉得最佳解法）好吧，今天在学习《流畅的python》时，⼜发现了⽬前为⽌我认为最佳解决⽅式。r这个映射类型会给键准备⼀个整体计数器。每次更新⼀个键的时候会增加这个计数器。所以这个类型可以⽤来给可散列表对象计数，或者是当成多重集合来⽤——多重集合就是集合⾥的元素可以出现不⽌⼀次。Counter实现了+和-运算符⽤来合并记录，还有像most_common([n])这类很有⽤的⽅法。most_common([n])会按照次序返回映射⾥最常见的n个键和他们的计数，详情参阅⽂档（）。看⼀下使⽤Counter类来实现对单词进⾏统计的代码：import reimport collectionsdef words_list(filename): try: with open(filename) as f_obj: content = f_() except FileNotFoundError: msg = "The file " + filename + " does not exist." print(msg) else: WORD_RE = e(r'W+') words = WORD_(()) return words

def words_counter(filename): try: with open(filename) as f_obj: content = f_() except FileNotFoundError: msg = "The file " + filename + " does not exist." print(msg) else: words = words_list(filename) words_count = r(words) return words_countif __name__ != '__main__': filename = 'Heart of ' words_count = words_counter(filename) # 注意这个显⽰前10需要使⽤words__common(10)⽅法 for word, word_count in words__common(10): print('{0:<20} : {1}'.format(word, word_count))结果为：the : 2468of : 1496a : 1209i : 1153and : 1062to : 974in : 673was : 672he : 596it : 515连排序（sorted）都省了，理论上应该更快。我们来测试⼀下速度。if __name__ == '__main__': import timeit def test_count_results(): filename = 'Heart of ' words_count = count_results(filename) return words_count def test_count_words(): filename = 'Heart of ' words_count = count_words(filename) return words_count def test_words_counter(): filename = 'Heart of ' words_count = words_counter(filename) return words_count time_1 = ('test_count_results()', setup="from __main__ import test_count_results") time_2 = ('test_count_words()', setup="from __main__ import test_count_words") time_3 = ('test_words_counter()', setup="from __main__ import test_words_counter") print(time_(number=10)) print(time_(number=10)) print(time_(number=10))结果为：36.7650.3799860.2645591842058579果然速度更快了。还能未完待续吗？