导入sogou输入法的细胞词库

Posted on July 16, 2010 by yongsun

首先请下载这个导入工具，解压缩到某个目录中。从pinyin.sogou.com下载细胞词库（scel格式），保存在某个目录中。然后在terminal中执行下面的操作：

$ python import_sogou_celldict.py PATH_TO_CELL_DICT.scel

感谢中文社区其他开发者和项目reverse engineering的工作，

98 thoughts on “导入sogou输入法的细胞词库”

yongsun on August 22, 2010 at 12:48 pm said:

@o_0! 先下载scel格式的词库，然后依文中的步骤执行脚本导入即可 ...
Houge on August 22, 2010 at 4:30 pm said:

@yongsun 谢谢您的答复，您能帮我导入一次，发封邮件附在邮件中可以吗？身边实在没有其他的系统了。邮箱地址您应该能从回复那里看到。我只是需要那个词库：http://pinyin.sogou.com/dict/cell.php?id=15125
如果您能帮我，小生将不胜感激
dunjian on August 23, 2010 at 4:56 am said:

@yongsun 不好意思，我重新下载了code和词库，不过还是有同样的问题。看他的提示，似乎是UnicodeEncodeError，不过我不知道这是啥咚咚，需要在哪修改，还请大侠指点:-)
yongsun on August 23, 2010 at 8:00 am said:

@dunjian, 您能告诉我您使用的是什么系统，以及python解释器的版本么？python --version
dunjian on August 23, 2010 at 9:39 am said:

系统是：OS X 10.6.4
Python的版本是：Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
P.S. 这个Python是我通过fink安装的。
yongsun on August 23, 2010 at 9:57 am said:

@dunjian，您可以试一试系统自带的python，10.6上的python版本就是2.6 ...
Kandu on August 23, 2010 at 1:45 pm said:

@Houge，寫好了一個程式，可以將 sunpinyin 的系統詞庫從 UTF-32 轉換為 UTF-8 編碼，然後改 importer.py 里的 UTF-32 至 UTF-8 並改讀入文件為轉換出的文件即可導入。
Kandu on August 23, 2010 at 1:47 pm said:

@Houge 此程式是依照編碼來改，所以不依賴外部庫，在 mac 上應該可以執行。 irc 聯繫吧
dunjian on August 25, 2010 at 4:50 am said:

谢谢yongsun，不过用了系统自带的python还是存在同样的问题
dunjian on August 25, 2010 at 5:18 am said:

对了，还有一点是我用的是英文的系统，并且用textedit打开细胞词库时，里面都是乱码
yongsun on August 25, 2010 at 10:38 am said:

@dunjian，奇怪，我在10.6下可以很正常的导入啊？是否可以先把fink的路径在$PATH中去掉再试试看？另，scel是二进制格式，不能直接用textedit打开 ...
dunjian on August 25, 2010 at 12:05 pm said:

yongsun，实在太麻烦你了。我试了把fink的路径去掉了，还是存在同样的问题。
Traceback (most recent call last):
File “import_sogou_celldict.py”, line 81, in
main()
File “import_sogou_celldict.py”, line 78, in main
import_to_sunpinyin_user_dict (generator)
File “/Users/Xinfeng/Desktop/sunpinyin_importer/importer.py”, line 50, in import_to_sunpinyin_user_dict
print “[%s] is too long or too short for sunpinyin userdict” % utf8str
UnicodeEncodeError: ‘ascii’ codec can’t encode characters in position 1-9: ordinal not in range(128)
yongsun on August 25, 2010 at 2:01 pm said:

@dunjian，我也不明白到底是什么原因了
lin on August 29, 2010 at 4:45 am said:

您好，非常感谢您开发了这么好的软件，用mac打字从来没这么舒服过，我也发现晚了，呵呵。但是“贇”字只有繁体，没有简体，这是我的名字，打繁体跟护照上不对应，有点麻烦。您能帮忙看一下不？不甚感激~~~^--^
yongsun on August 30, 2010 at 9:01 am said:

@lin，已经在open-gram上fix了，稍后发布新版词库的二进制文件 ...
yongsun on August 30, 2010 at 10:34 pm said:

@lin, 请下载http://code.google.com/p/sunpinyin/downloads/detail?name=pydict_sc.bin.le，并替换系统中的/Library/Input\ Methods/SunPinyin.app/Contents/Resources/pydict_sc.bin，并杀死sunpinyin的进程 ...
乌斑兔小白 on September 12, 2010 at 9:53 am said:

导入词库的时候有个提示信息:
[周孝闵帝宇文觉] is too long or too short for sunpinyin userdict
导入诗词词库的时候满屏都是这写信息,是不是我们的拼音词库不支持稍微长一点的句子啊?谢谢!
乌斑兔小白 on September 12, 2010 at 10:03 am said:

不过,在使用sunpinyin的过程中,如果输入了诗词,那么下次输入声母就可以找到诗词了,看来系统可以储存自定义长句,可是导入搜狗词库的长句子却出了问题,真是奇怪啊!
yongsun on September 12, 2010 at 9:42 pm said:

@乌斑兔小白，用户词典目前有上限，不过还有history cache可以弥补 ...
csta87 on September 20, 2010 at 5:32 pm said:

求助啊~我的系统是10.5.8 导入失败了，请问是什么原因呢？
Traceback (most recent call last):
File "import_sogou_celldict.py", line 81, in
main()
File "import_sogou_celldict.py", line 78, in main
import_to_sunpinyin_user_dict (generator)
File "/Users/csta87/Downloads/sunpinyin_importer/importer.py", line 58, in import_to_sunpinyin_user_dict
sysdict = load_system_dict()
File "/Users/csta87/Downloads/sunpinyin_importer/importer.py", line 47, in load_system_dict
for w in str.decode('UTF-32').split(''):
LookupError: unknown encoding: UTF-32
yongsun on September 20, 2010 at 10:05 pm said:

@csta87，貌似是python的版本有问题，不行先到10.6或者linux上进行导入，然后再copy回来就好了。您是否使用了第三方的python，例如macports的？
是 on September 22, 2010 at 11:46 am said:

ssssss
george on October 14, 2010 at 6:29 pm said:

george@george-laptop:~/Downloads/sunpinyin_importer$ python import_sogou_celldict.py PATH_TO_CELL_DICT.scel
Traceback (most recent call last):
File "import_sogou_celldict.py", line 81, in
main()
File "import_sogou_celldict.py", line 78, in main
import_to_sunpinyin_user_dict (generator)
File "/home/george/Downloads/sunpinyin_importer/importer.py", line 71, in import_to_sunpinyin_user_dict
for (pystr, utf8str) in records:
File "import_sogou_celldict.py", line 22, in get_word_from_sogou_cell_dict
f = open (fname, 'rb')
IOError: [Errno 2] No such file or directory: 'PATH_TO_CELL_DICT.scel'
能告诉我这个该怎么办吗？
yongsun on October 17, 2010 at 12:59 am said:

@george，您需要用实际的文件名替换"PATH_TO_CELL_DICT.scel"…
yokoqo on October 19, 2010 at 11:50 pm said:

老板,怎么老是都是提示 "****" is too long or too short for sunpinyin userdict?
这个是正常的吗?
yongsun on October 20, 2010 at 9:28 am said:

@yokoqo，目前我们的用户词典只支持最长6个音节的词条，今后会加入用户短语库来弥补
buptguo on November 29, 2010 at 6:29 pm said:

导入的时候还有词的长短限制的吗？为什么我导入好多too long or too short for sunpinyin userdict
yongsun on December 6, 2010 at 9:58 am said:

@buptguo，是的，最长可以导入6个音节的词条 ...
StayReal on December 8, 2010 at 8:19 pm said:

要不要那么囧啊= =....我兴致勃勃下了5个细胞词库...全部都是 is too long or too short for sunpinyin userdict
我才刚装的sun啊= =..而且还没切换导sun下打字过.
yongsun on December 9, 2010 at 9:37 am said:

@StayReal, you may already have some words imported, however the import script does not print a message for those successfully imported words ...
sk on December 18, 2010 at 10:33 pm said:

-bash: /Users/and1mantou/Downloads/1.scel: Permission denied
Kai-ShengmatoMacBook-Pro:sunpinyin_importer and1mantou$
怎么回事呢？总是拒绝
yongsun on December 20, 2010 at 10:13 am said:

@sk, 你是在shell下直接运行1.scel这个文件？另外，建议你直接用fit的最新版本吧，已经包含sunpinyin引擎了，而且有词库管理工具
Rabbit_52 on January 7, 2011 at 9:18 pm said:

very nice! 长句用不上罢了，能导入我已经很满意了！谢谢孙！:)
chu on February 5, 2011 at 11:10 am said:

导入词库的时候每一个词条都提示：xxx is too long or too short for sunpinyin userdict
环境：
mac os x 10.6.5
SunPinyin-MacOS-2.0.2.1008
Python 2.6.1
yongsun on February 7, 2011 at 7:07 pm said:

@chu，不必担心，只有过长或过短时才提示该消息…
Isalahberlin on March 14, 2011 at 6:11 pm said:

$ python import_sogou_celldict.py PATH_TO_CELL_DICT.scel
——————————————————————————————
您好lz：
我如果有多个scel词库的话，比如100个，我如何一次性的在shell里面把他们导入到sunpinyin中去呢？上面的这个命令该如何更改呢？一个一个的导入实在是太麻烦了，或者说，能不能有一个将多个scel词库合并为一个词库的方法呢？
Isalahberlin on March 14, 2011 at 6:12 pm said:

忘了说了，这些词库都是在一个文件夹里面的。
yongsun on March 15, 2011 at 9:09 am said:

@Isalahberlin，您可以用一个简单的shell命令来做这件事请，是您放词库的目录，假定import_sogou_celldict.py在当前目录中，
$ find -name "*.scel" -exec python import_sogou_celldict.py {} \; -print
Isalahberlin on March 16, 2011 at 2:41 pm said:

这些导入的脚本很好用，但我还有三个问题，请大家解答：
1.我在用sogou词库导入的py脚本的时候，出现了下面的报错信息，我到原词库里面看了，原来的词条和其他正常的词条并没有什么特别（这只是一部分，还有很多）:
[ong'neng'jian] has un-recognized syllables, ignoring this record!
[i'chan] has un-recognized syllables, ignoring this record!
[i'sai'ya'bo'lin] has un-recognized syllables, ignoring this record!
[hi'wan'fan] has un-recognized syllables, ignoring this record!
[iu'mei'xie] has un-recognized syllables, ignoring this record!
[u'an'zhao] has un-recognized syllables, ignoring this record!
[iang'hang] has un-recognized syllables, ignoring this record!
[i'li'qie'fu] has un-recognized syllables, ignoring this record!
原词库里面的词条:
gong'neng'jian 功能键
zi'chan 自产
yi'sai'ya'bo'lin 以塞亚伯林
chi'wan'fan 吃完饭
jiu'mei'xie 就没写
bu'an'zhao 不按照
liang'hang 两行
di'li'qie'fu 蒂里切夫
很明显，在报错里面，把这些词条的首字母给切掉了，那么为什么这些词条会提示说里面有无法识别的音节呢?而其他的词条却没事呢?这些词条看起来和其他没并被切的词条没有任何区别啊？
2.关于词库重复导入的问题，如果我忘了先前已经导入了某个词库，又一次把这个词库导入到userdict里面去了，这样会不会导致userdict里面增加垃圾数据?无端的增加userdict的大小?
3.我发现一个很诡异的现象，我的mac里面，userdict这个文件的大小有的时候的变化会让人捉摸不定，比如我在导入30个词库之前，他的大小还是100M，可以我导入了30个词库之后，他的大小反而变小了，一下子变成70M了---------这是怎么回事儿?
Isalahberlin on March 16, 2011 at 2:42 pm said:

另外关于我上上个提问，谢谢老大的回答，我已经解决了。。。。。。。。。
Isalahberlin on March 16, 2011 at 2:47 pm said:

FIT，还是别提了，你们怎么选了这么一群猪一样的合作伙伴？太难用了，很怀疑那些人是不是专业出身的，用着sunpinyin的核心，词频让他们玩儿成那样，不管多生僻的词都放在第一个。。。软件响应慢的要死，弄个输入法还动不动就风火轮cpu占用90%。。。。。。他们那词库导入工具垃圾的要死，比lz的这些py差太远了。
Isalahberlin on March 18, 2011 at 10:32 am said:

这些导入的脚本很好用，但我还有三个问题，请大家解答：
1.我在用sogou词库导入的py脚本的时候，出现了下面的报错信息，我到原词库里面看了，原来的词条和其他正常的词条并没有什么特别（这只是一部分，还有很多）:
[ong'neng'jian] has un-recognized syllables, ignoring this record!
[i'chan] has un-recognized syllables, ignoring this record!
[i'sai'ya'bo'lin] has un-recognized syllables, ignoring this record!
[hi'wan'fan] has un-recognized syllables, ignoring this record!
[iu'mei'xie] has un-recognized syllables, ignoring this record!
[u'an'zhao] has un-recognized syllables, ignoring this record!
[iang'hang] has un-recognized syllables, ignoring this record!
[i'li'qie'fu] has un-recognized syllables, ignoring this record!
原词库里面的词条:
gong’neng’jian 功能键
zi’chan 自产
yi’sai’ya’bo’lin 以塞亚伯林
chi’wan’fan 吃完饭
jiu’mei’xie 就没写
bu’an’zhao 不按照
liang’hang 两行
di’li’qie’fu 蒂里切夫
很明显，在报错里面，把这些词条的首字母给切掉了，那么为什么这些词条会提示说里面有无法识别的音节呢?而其他的词条却没事呢?这些词条看起来和其他没并被切的词条没有任何区别啊？
2.关于词库重复导入的问题，如果我忘了先前已经导入了某个词库，又一次把这个词库导入到userdict里面去了，这样会不会导致userdict里面增加垃圾数据?无端的增加userdict的大小?
3.我发现一个很诡异的现象，我的mac里面，userdict这个文件的大小有的时候的变化会让人捉摸不定，比如我在导入30个词库之前，他的大小还是100M，可以我导入了30个词库之后，他的大小反而变小了，一下子变成70M了———这是怎么回事儿?
Wao on April 19, 2011 at 7:09 am said:

要是支持搜狗主词库的导入就好了~
yongsun on April 19, 2011 at 1:42 pm said:

@Isalahberlin 有可能是脚本的错误，你能把导入的词库文件（url）告诉我么？
marissa on July 21, 2011 at 10:51 am said:

Traceback (most recent call last):
File "import_sogou_celldict.py", line 81, in
main()
File "import_sogou_celldict.py", line 78, in main
import_to_sunpinyin_user_dict (generator)
File "/home/marissa/work/seg/sunpinyin_importer/importer.py", line 55, in import_to_sunpinyin_user_dict
userdict_path = userdict_path if userdict_path else get_userdict_path()
File "/home/marissa/work/seg/sunpinyin_importer/importer.py", line 23, in get_userdict_path
raise "Can not detect sunpinyin's userdict!"
TypeError: exceptions must be old-style classes or derived from BaseException, not str
yongsun on July 22, 2011 at 8:26 pm said:

@marissa，您是在linux上使用ibus-sunpinyin还是fcitx-sunpinyin？貌似是没有在您的HOME目录下找到sunpinyin的用户词典…
大茶几の骨骸 on November 19, 2011 at 12:18 am said:

请问遇到重复的词是做覆盖处理吗?
yongsun on November 21, 2011 at 9:23 am said:

@大茶几の骨骸，遇到重复的词应该会跳过

素心如何天上月

Yong Sun's Blog

导入sogou输入法的细胞词库

98 thoughts on “导入sogou输入法的细胞词库”

Leave a Reply Cancel reply