Wzzz
/
NLP
Not watched
1
0
Fork 0
Code
Releases 0 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
1 Commit
1 Branch
17 MB
1 Download
 
 
 
 
Tree: 45915b2ba1
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '45915b2ba1'
${ noResults }
NLP/Models+K-Means/nltk_data/tokenizers/punkt_tab
WangYizhou 45915b2ba1 “提交项目” 6 months ago
..
czech “提交项目” 6 months ago
danish “提交项目” 6 months ago
dutch “提交项目” 6 months ago
english “提交项目” 6 months ago
estonian “提交项目” 6 months ago
finnish “提交项目” 6 months ago
french “提交项目” 6 months ago
german “提交项目” 6 months ago
greek “提交项目” 6 months ago
italian “提交项目” 6 months ago
norwegian “提交项目” 6 months ago
polish “提交项目” 6 months ago
portuguese “提交项目” 6 months ago
russian “提交项目” 6 months ago
slovene “提交项目” 6 months ago
spanish “提交项目” 6 months ago
swedish “提交项目” 6 months ago
turkish “提交项目” 6 months ago
README “提交项目” 6 months ago

README 

Pretrained Punkt Models -- Jan Strunk (New version trained after issues 313 and 514 had been corrected)

Most models were prepared using the test corpora from Kiss and Strunk (2006). Additional models have
been contributed by various people using NLTK for sentence boundary detection.

For information about how to use these models, please confer the tokenization HOWTO:
http://nltk.googlecode.com/svn/trunk/doc/howto/tokenize.html
and chapter 3.8 of the NLTK book:
http://nltk.googlecode.com/svn/trunk/doc/book/ch03.html#sec-segmentation

There are pretrained tokenizers for the following languages:

File                Language            Source                             Contents                Size of training corpus(in tokens)           Model contributed by
=======================================================================================================================================================================
czech.pickle        Czech               Multilingual Corpus 1 (ECI)        Lidove Noviny                   ~345,000                             Jan Strunk / Tibor Kiss
                                                                           Literarni Noviny
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
danish.pickle       Danish              Avisdata CD-Rom Ver. 1.1. 1995     Berlingske Tidende              ~550,000                             Jan Strunk / Tibor Kiss
                                        (Berlingske Avisdata, Copenhagen)  Weekend Avisen
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
dutch.pickle        Dutch               Multilingual Corpus 1 (ECI)        De Limburger                    ~340,000                             Jan Strunk / Tibor Kiss
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
english.pickle      English             Penn Treebank (LDC)                Wall Street Journal             ~469,000                             Jan Strunk / Tibor Kiss
                    (American)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
estonian.pickle     Estonian            University of Tartu, Estonia       Eesti Ekspress                  ~359,000                             Jan Strunk / Tibor Kiss
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
finnish.pickle      Finnish             Finnish Parole Corpus, Finnish     Books and major national        ~364,000                             Jan Strunk / Tibor Kiss
                                        Text Bank (Suomen Kielen           newspapers
                                        Tekstipankki)
                                        Finnish Center for IT Science
                                        (CSC)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
french.pickle       French              Multilingual Corpus 1 (ECI)        Le Monde                        ~370,000                             Jan Strunk / Tibor Kiss
                    (European)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
german.pickle       German              Neue Zürcher Zeitung AG            Neue Zürcher Zeitung            ~847,000                             Jan Strunk / Tibor Kiss
                    (Switzerland)       CD-ROM
                    (Uses "ss"
                     instead of "ß")
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
greek.pickle        Greek               Efstathios Stamatatos              To Vima (TO BHMA)               ~227,000                             Jan Strunk / Tibor Kiss
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
italian.pickle      Italian             Multilingual Corpus 1 (ECI)        La Stampa, Il Mattino           ~312,000                             Jan Strunk / Tibor Kiss
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
norwegian.pickle    Norwegian           Centre for Humanities              Bergens Tidende                 ~479,000                             Jan Strunk / Tibor Kiss
                    (Bokmål and         Information Technologies,
                     Nynorsk)           Bergen
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
polish.pickle       Polish              Polish National Corpus             Literature, newspapers, etc.  ~1,000,000                             Krzysztof Langner
                                        (http://www.nkjp.pl/)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
portuguese.pickle   Portuguese          CETENFolha Corpus                  Folha de São Paulo              ~321,000                             Jan Strunk / Tibor Kiss
                    (Brazilian)         (Linguateca)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
slovene.pickle      Slovene             TRACTOR                            Delo                            ~354,000                             Jan Strunk / Tibor Kiss
                                        Slovene Academy for Arts
                                        and Sciences
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
spanish.pickle      Spanish             Multilingual Corpus 1 (ECI)        Sur                             ~353,000                             Jan Strunk / Tibor Kiss
                    (European)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
swedish.pickle      Swedish             Multilingual Corpus 1 (ECI)        Dagens Nyheter                  ~339,000                             Jan Strunk / Tibor Kiss
                                                                           (and some other texts)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
turkish.pickle      Turkish             METU Turkish Corpus                Milliyet                        ~333,000                             Jan Strunk / Tibor Kiss
                                        (Türkçe Derlem Projesi)
                                        University of Ankara
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

The corpora contained about 400,000 tokens on average and mostly consisted of newspaper text converted to
Unicode using the codecs module.

Kiss, Tibor and Strunk, Jan (2006): Unsupervised Multilingual Sentence Boundary Detection.
Computational Linguistics 32: 485-525.

---- Training Code ----

# import punkt
import nltk.tokenize.punkt

# Make a new Tokenizer
tokenizer = nltk.tokenize.punkt.PunktSentenceTokenizer()

# Read in training corpus (one example: Slovene)
import codecs
text = codecs.open("slovene.plain","Ur","iso-8859-2").read()

# Train tokenizer
tokenizer.train(text)

# Dump pickled tokenizer
import pickle
out = open("slovene.pickle","wb")
pickle.dump(tokenizer, out)
out.close()

---------
Delete task

Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.

cancel
confirm

在信息安全领域，漏洞评估和管理是关键任务之一。本作品探讨了如何利用预训练文本大模型来评估和研判漏洞的严重等级，具体基于通用漏洞评分系统。传统漏洞评分方法依赖于手动分析和专家评审。而基于自然语言处理文本大模型通过其深度学习能力，可以自动化地处理和分析大量的安全相关文本数据，从而提高漏洞评估的效率和准确性。结合词干提取、词性还原能够更好地发挥自然语言处理文本大模型的预测能力与准确度。

CSV SVG Jupyter Notebook other