Search Results for author:
Found 8 papers, 2 papers with code
OpenKorPOS: Democratizing Korean Tokenization with Voting-Based Open Corpus Annotation
As this problem originates from the conventional scheme used when creating a POS tagging corpus, we propose an improvement to the existing scheme, which makes it friendlier to generative tasks.
PatchBERT: Just-in-Time, Out-of-Vocabulary Patching
Large scale pre-trained language models have shown groundbreaking performance improvements for transfer learning in the domain of natural language processing.
Two Counterexamples to Tokenization and the Noiseless Channel
In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), R\'enyi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest R\'enyi efficiency of the unigram distribution should be chosen.
Learning How to Translate North Korean through South Korean
Training a model using North Korean data is the most straightforward approach to solving this problem, but there is insufficient data to train NMT models.
StyleKQC: A Style-Variant Paraphrase Corpus for Korean Questions and Commands
Paraphrasing is often performed with less concern for controlled style conversion.
Open Korean Corpora: A Practical Report
Korean is often referred to as a low-resource language in the research community.
Jamo Pair Encoding: Subcharacter Representation-based Extreme Korean Vocabulary Compression for Efficient Subword Tokenization
In the context of multilingual language model pre-training, vocabulary size for languages with a broad set of potential characters is an unsolved problem.
Machines Getting with the Program: Understanding Intent Arguments of Non-Canonical Directives
Modern dialog managers face the challenge of having to fulfill human-level conversational skills as part of common user expectations, including but not limited to discourse with no clear objective.
