百度NLP中文分词插件
更新时间：2024-06-12
analysis-baidu-nlp是百度智能云Elasticsearch（简称ES）团队自主研发的中文分词插件，该插件在中文分词上的性能与准确率均处于业界领先水平。
背景
analysis-baidu-nlp 基于百度NLP内部自主研发的DeepCRF模型，该模型凝聚了百度在中文搜索领域十几年的技术积累，其模型性能与准确率均处于业界领先地位
提供基础粒度和短语粒度两种分词结果，以供不同的应用需求，短语粒度是以基础粒度的分词进行智能组合的结果。
注意：
- 词典模型会在第一次使用时加载到JVM的堆外内存，我们推荐所用的套餐节点内存8G以上。
- 目前NLP中文分词插件支持6.5.3、6.8.14、7.4.2、7.10.2版本的实例，不支持NLP中文分词插件的集群，请提交工单，BES团队会协助升级集群，升级方式参见ES版本升级。
分词粒度
analysis-baidu-nlp主要提供两种粒度的Analyzer：
- 基础粒度模型(bd-nlp-basic)
- 短语粒度模型(bd-nlp-phrase)
两种Analyzer内部集成大小写过滤器、停用词过滤器，开箱即用。
同名提供两种Tokenizer：
- 基础模型粒度(bd-nlp-basic)
- 短语大粒度模型(bd-nlp-phrase)
两种粒度Tokenizer只提供最原始的切词结果，用户可根据自己的应用需求添加自定义的停用词过滤以及一些复杂的过滤器。停用词过滤可以参考如下示例，添加了and、or和the停用词：
Plain Text
1PUT /test_index
2{
3 "settings": {
4 "analysis": {
5 "filter": {
6 "my_stop": {
7 "type": "stop",
8 "stopwords": ["and", "is", "the"]
9 }
10 }
11 }
12 }
13}与ik在基础粒度和短语粒度切词的对比
基础粒度对比
对 “维修基金” 进行基础最大粒度切词效果对比
- bd-nlp-basic 切词
Plain Text
1POST /_analyze
2{
3 "text": "维修基金",
4 "analyzer": "bd-nlp-basic"
5}分词结果：
Plain Text
1{
2 "tokens": [
3 {
4 "token": "维修",
5 "start_offset": 0,
6 "end_offset": 2,
7 "type": "WORD",
8 "position": 0
9 },
10 {
11 "token": "基金",
12 "start_offset": 2,
13 "end_offset": 4,
14 "type": "WORD",
15 "position": 1
16 }
17 ]
18}- ik_max_word 切词
Plain Text
1POST _analyze
2{
3 "analyzer": "ik_max_word",
4 "text": "维修基金"
5}切词结果：
Plain Text
1{
2 "tokens": [
3 {
4 "token": "维修基金",
5 "start_offset": 0,
6 "end_offset": 4,
7 "type": "CN_WORD",
8 "position": 0
9 },
10 {
11 "token": "维修",
12 "start_offset": 0,
13 "end_offset": 2,
14 "type": "CN_WORD",
15 "position": 1
16 },
17 {
18 "token": "维",
19 "start_offset": 0,
20 "end_offset": 1,
21 "type": "CN_WORD",
22 "position": 2
23 },
24 {
25 "token": "修",
26 "start_offset": 1,
27 "end_offset": 2,
28 "type": "CN_CHAR",
29 "position": 3
30 },
31 {
32 "token": "基金",
33 "start_offset": 2,
34 "end_offset": 4,
35 "type": "CN_WORD",
36 "position": 4
37 },
38 {
39 "token": "基",
40 "start_offset": 2,
41 "end_offset": 3,
42 "type": "CN_WORD",
43 "position": 5
44 },
45 {
46 "token": "金",
47 "start_offset": 3,
48 "end_offset": 4,
49 "type": "CN_CHAR",
50 "position": 6
51 }
52 ]
53}对 “清明节，又称踏青节、行清节、三月节、祭祖节等” 进行短语切测效果对比
- bd-nlp-basic 短语切词
Plain Text
1POST /_analyze
2{
3 "text": "清明节，又称踏青节、行清节、三月节、祭祖节等",
4 "analyzer": "bd-nlp-phrase"
5}短语切分结果：
Plain Text
1{
2 "tokens": [
3 {
4 "token": "清明节",
5 "start_offset": 0,
6 "end_offset": 3,
7 "type": "WORD",
8 "position": 0
9 },
10 {
11 "token": "又称",
12 "start_offset": 4,
13 "end_offset": 6,
14 "type": "WORD",
15 "position": 2
16 },
17 {
18 "token": "踏青节",
19 "start_offset": 6,
20 "end_offset": 9,
21 "type": "WORD",
22 "position": 3
23 },
24 {
25 "token": "行清节",
26 "start_offset": 10,
27 "end_offset": 13,
28 "type": "WORD",
29 "position": 5
30 },
31 {
32 "token": "三月节",
33 "start_offset": 14,
34 "end_offset": 17,
35 "type": "WORD",
36 "position": 7
37 },
38 {
39 "token": "祭祖",
40 "start_offset": 18,
41 "end_offset": 20,
42 "type": "WORD",
43 "position": 9
44 },
45 {
46 "token": "节",
47 "start_offset": 20,
48 "end_offset": 21,
49 "type": "WORD",
50 "position": 10
51 }
52 ]
53}- ik_smart 智能切词
Plain Text
1POST _analyze
2{
3 "analyzer": "ik_smart",
4 "text": "清明节，又称踏青节、行清节、三月节、祭祖节等"
5}切词结果：
Plain Text
1{
2 "tokens": [
3 {
4 "token": "清明节",
5 "start_offset": 0,
6 "end_offset": 3,
7 "type": "CN_WORD",
8 "position": 0
9 },
10 {
11 "token": "又称",
12 "start_offset": 4,
13 "end_offset": 6,
14 "type": "CN_WORD",
15 "position": 1
16 },
17 {
18 "token": "踏青",
19 "start_offset": 6,
20 "end_offset": 8,
21 "type": "CN_WORD",
22 "position": 2
23 },
24 {
25 "token": "节",
26 "start_offset": 8,
27 "end_offset": 9,
28 "type": "CN_WORD",
29 "position": 3
30 },
31 {
32 "token": "行",
33 "start_offset": 10,
34 "end_offset": 11,
35 "type": "CN_WORD",
36 "position": 4
37 },
38 {
39 "token": "清",
40 "start_offset": 11,
41 "end_offset": 12,
42 "type": "CN_CHAR",
43 "position": 5
44 },
45 {
46 "token": "节",
47 "start_offset": 12,
48 "end_offset": 13,
49 "type": "CN_WORD",
50 "position": 6
51 },
52 {
53 "token": "三月",
54 "start_offset": 14,
55 "end_offset": 16,
56 "type": "CN_WORD",
57 "position": 7
58 },
59 {
60 "token": "节",
61 "start_offset": 16,
62 "end_offset": 17,
63 "type": "COUNT",
64 "position": 8
65 },
66 {
67 "token": "祭祖",
68 "start_offset": 18,
69 "end_offset": 20,
70 "type": "CN_WORD",
71 "position": 9
72 },
73 {
74 "token": "节",
75 "start_offset": 20,
76 "end_offset": 21,
77 "type": "CN_WORD",
78 "position": 10
79 }
80 ]
81}Analyze API使用
基础模型粒度分词
Plain Text
1POST /_analyze
2{
3 "analyzer": "bd-nlp-basic",
4 "text": "去年我们和他们展开了炉际竞赛，第一回合赢了，第二回合和第三回合却败下阵来。"
5}分词结果：
Plain Text
1{
2 "tokens": [
3 {
4 "token": "去年",
5 "start_offset": 0,
6 "end_offset": 2,
7 "type": "WORD",
8 "position": 0
9 },
10 {
11 "token": "我们",
12 "start_offset": 2,
13 "end_offset": 4,
14 "type": "WORD",
15 "position": 1
16 },
17 {
18 "token": "和",
19 "start_offset": 4,
20 "end_offset": 5,
21 "type": "WORD",
22 "position": 2
23 },
24 {
25 "token": "他们",
26 "start_offset": 5,
27 "end_offset": 7,
28 "type": "WORD",
29 "position": 3
30 },
31 {
32 "token": "展开",
33 "start_offset": 7,
34 "end_offset": 9,
35 "type": "WORD",
36 "position": 4
37 },
38 {
39 "token": "炉际",
40 "start_offset": 10,
41 "end_offset": 12,
42 "type": "WORD",
43 "position": 6
44 },
45 {
46 "token": "竞赛",
47 "start_offset": 12,
48 "end_offset": 14,
49 "type": "WORD",
50 "position": 7
51 },
52 {
53 "token": "第一",
54 "start_offset": 15,
55 "end_offset": 17,
56 "type": "WORD",
57 "position": 9
58 },
59 {
60 "token": "回合",
61 "start_offset": 17,
62 "end_offset": 19,
63 "type": "WORD",
64 "position": 10
65 },
66 {
67 "token": "赢",
68 "start_offset": 19,
69 "end_offset": 20,
70 "type": "WORD",
71 "position": 11
72 },
73 {
74 "token": "第二",
75 "start_offset": 22,
76 "end_offset": 24,
77 "type": "WORD",
78 "position": 14
79 },
80 {
81 "token": "回合",
82 "start_offset": 24,
83 "end_offset": 26,
84 "type": "WORD",
85 "position": 15
86 },
87 {
88 "token": "和",
89 "start_offset": 26,
90 "end_offset": 27,
91 "type": "WORD",
92 "position": 16
93 },
94 {
95 "token": "第三",
96 "start_offset": 27,
97 "end_offset": 29,
98 "type": "WORD",
99 "position": 17
100 },
101 {
102 "token": "回合",
103 "start_offset": 29,
104 "end_offset": 31,
105 "type": "WORD",
106 "position": 18
107 },
108 {
109 "token": "败",
110 "start_offset": 32,
111 "end_offset": 33,
112 "type": "WORD",
113 "position": 20
114 },
115 {
116 "token": "下",
117 "start_offset": 33,
118 "end_offset": 34,
119 "type": "WORD",
120 "position": 21
121 },
122 {
123 "token": "阵",
124 "start_offset": 34,
125 "end_offset": 35,
126 "type": "WORD",
127 "position": 22
128 },
129 {
130 "token": "来",
131 "start_offset": 35,
132 "end_offset": 36,
133 "type": "WORD",
134 "position": 23
135 }
136 ]
137}短语模型大粒度分词
Plain Text
1POST /_analyze
2{
3 "analyzer": "bd-nlp-phrase",
4 "text": "去年我们和他们展开了炉际竞赛，第一回合赢了，第二回合和第三回合却败下阵来。"
5}分词结果：
Plain Text
1{
2 "tokens": [
3 {
4 "token": "去年",
5 "start_offset": 0,
6 "end_offset": 2,
7 "type": "WORD",
8 "position": 0
9 },
10 {
11 "token": "我们",
12 "start_offset": 2,
13 "end_offset": 4,
14 "type": "WORD",
15 "position": 1
16 },
17 {
18 "token": "和",
19 "start_offset": 4,
20 "end_offset": 5,
21 "type": "WORD",
22 "position": 2
23 },
24 {
25 "token": "他们",
26 "start_offset": 5,
27 "end_offset": 7,
28 "type": "WORD",
29 "position": 3
30 },
31 {
32 "token": "展开",
33 "start_offset": 7,
34 "end_offset": 9,
35 "type": "WORD",
36 "position": 4
37 },
38 {
39 "token": "炉际竞赛",
40 "start_offset": 10,
41 "end_offset": 14,
42 "type": "WORD",
43 "position": 6
44 },
45 {
46 "token": "第一回合",
47 "start_offset": 15,
48 "end_offset": 19,
49 "type": "WORD",
50 "position": 8
51 },
52 {
53 "token": "赢",
54 "start_offset": 19,
55 "end_offset": 20,
56 "type": "WORD",
57 "position": 9
58 },
59 {
60 "token": "第二回合",
61 "start_offset": 22,
62 "end_offset": 26,
63 "type": "WORD",
64 "position": 12
65 },
66 {
67 "token": "和",
68 "start_offset": 26,
69 "end_offset": 27,
70 "type": "WORD",
71 "position": 13
72 },
73 {
74 "token": "第三",
75 "start_offset": 27,
76 "end_offset": 29,
77 "type": "WORD",
78 "position": 14
79 },
80 {
81 "token": "回合",
82 "start_offset": 29,
83 "end_offset": 31,
84 "type": "WORD",
85 "position": 15
86 },
87 {
88 "token": "败",
89 "start_offset": 32,
90 "end_offset": 33,
91 "type": "WORD",
92 "position": 17
93 },
94 {
95 "token": "下",
96 "start_offset": 33,
97 "end_offset": 34,
98 "type": "WORD",
99 "position": 18
100 },
101 {
102 "token": "阵",
103 "start_offset": 34,
104 "end_offset": 35,
105 "type": "WORD",
106 "position": 19
107 },
108 {
109 "token": "来",
110 "start_offset": 35,
111 "end_offset": 36,
112 "type": "WORD",
113 "position": 20
114 }
115 ]
116}索引指定Analyzer
Plain Text
1PUT test
2{
3 "mappings": {
4 "doc": {
5 "properties": {
6 "k1": {
7 "type": "text",
8 "analyzer": "bd-nlp-basic" // 使用基础粒度模型
9 },
10 "k2": {
11 "type": "text",
12 "analyzer": "bd-nlp-phrase" // 使用短语粒度模型
13 }
14 }
15 }
16 },
17 "settings": {
18 "index": {
19 "number_of_shards": "1",
20 "number_of_replicas": "0"
21 }
22 }
23}索引指定Tokenizer
Plain Text
1PUT /test
2{
3 "settings":{
4 "analysis":{
5 "analyzer":{
6 "my_analyzer":{
7 "tokenizer":"bd-nlp-basic", // 自定义一个analyzer
8 "filter":[
9 "lowercase" // 添加应用需要的过滤器
10 ]
11 }
12 }
13 }
14 },
15 "mappings":{
16 "properties":{
17 "k2":{
18 "type":"text",
19 "analyzer":"my_analyzer" // 将自定义analyzer应用到对应字段上
20 }
21 }
22 }
23}准确率与召回率
百度内部大数据集测试结果：
| 模型 | 准确率 | 召回率 | F值 |
|---|---|---|---|
| analysis-baidu-nlp | 98.8% | 98.9% | 98.8% |