暨南学报(哲学社会科学版)

作者：范俊军, 刘贤娴

单位：暨南大学文学院。

关键词：少数民族语言,民族文献,文本识别,OCR,数据构建,数字人文

我国少数民族语言文献数量庞大,文字种类繁多,内容涵盖政治、经济、法律、历史、文学、艺术、宗教、天文、地理、医药等领域,是中华民族文化知识的重要组成部分。构建各民族文献文本数据,使之应用于自然语言处理和人工智能,能有效促进中华优秀传统知识创新性传承,促进知识社会化,是对各民族语言古文献和现代书报刊进行文字识别和文本转换数据构建的基础。国内早期OCR技术虽然解决了几种主要少数民族文字识别的问题,但因字符为非Unicode基本集编码而弃用。当前OCR技术已能较好识别蒙、藏、维、哈、朝等文种文献,但在处理我国汉文与少数民族文字混排图像文本时仍然效果不佳。因此应推进少数民族语言文献OCR识别技术创新。我国少数民族语言文献现行活态文字有十多种,其中非拉丁字系的文字有11种,OCR技术应重点解决这类少数民族语言字系的抄本、刻版和铅字印刷文本,以及汉文与民族文字混排文本的识别问题,研发开放的多功能工具和平台。在此基础上,进一步开展少数民族语言文献文本大规模数据构建,以促进我国语言科学研究和自然语言处理的创新发展。"/>template{display:none;}.mag-rich-xref-fn { cursor: pointer;}//长视频$(document).ready(function(){if($("#showLongArticleVideo") && $("#showLongArticleVideo").length && $("#showLongArticleVideo").length>0){mag_ajax_update({ele_id:'showLongArticleVideo',url:mag_currentQikanUrl() + "/CN/article/showLongArticleVideo.do?id="+$("#articleId").val()});}});$(document).ready(function(){window.metaData = {"journal":{"issn":"1000-5072","qiKanMingCheng_CN":"暨南学报（哲学社会科学版）","id":2,"qiKanMingCheng_EN":"Jinan Journal"},"article":{"keywordList_cn":["少数民族语言","民族文献","文本识别","OCR","数据构建","数字人文"],"juan":"46","zhaiyao_cn":"我国少数民族语言文献数量庞大,文字种类繁多,内容涵盖政治、经济、法律、历史、文学、艺术、宗教、天文、地理、医药等领域,是中华民族文化知识的重要组成部分。构建各民族文献文本数据,使之应用于自然语言处理和人工智能,能有效促进中华优秀传统知识创新性传承,促进知识社会化,是对各民族语言古文献和现代书报刊进行文字识别和文本转换数据构建的基础。国内早期OCR技术虽然解决了几种主要少数民族文字识别的问题,但因字符为非Unicode基本集编码而弃用。当前OCR技术已能较好识别蒙、藏、维、哈、朝等文种文献,但在处理我国汉文与少数民族文字混排图像文本时仍然效果不佳。因此应推进少数民族语言文献OCR识别技术创新。我国少数民族语言文献现行活态文字有十多种,其中非拉丁字系的文字有11种,OCR技术应重点解决这类少数民族语言字系的抄本、刻版和铅字印刷文本,以及汉文与民族文字混排文本的识别问题,研发开放的多功能工具和平台。在此基础上,进一步开展少数民族语言文献文本大规模数据构建,以促进我国语言科学研究和自然语言处理的创新发展。","endNoteUrl_en":"https://jnxb.jnu.edu.cn/skb/EN/article/getTxtFile.do?fileType=EndNote&id=6998","reference":"","bibtexUrl_cn":"https://jnxb.jnu.edu.cn/skb/CN/article/getTxtFile.do?fileType=BibTeX&id=6998","abstractUrl_en":"https://jnxb.jnu.edu.cn/skb/EN/10.11778/j.jnxb.20241011","qi":"6","id":6998,"nian":2024,"bianHao":"1720771493171-33693479","zuoZheEn_L":"FAN Junjun, LIU Xianxian","juanUrl_en":"https://jnxb.jnu.edu.cn/skb/EN/Y2024","shouCiFaBuRiQi":"2024-07-12","qiShiYe":"31","qiUrl_cn":"https://jnxb.jnu.edu.cn/skb/CN/Y2024/V46/I6","lanMu_cn":"语言学·数字人文","pdfSize":"1297","zuoZhe_CN":"范俊军, 刘贤娴","risUrl_cn":"https://jnxb.jnu.edu.cn/skb/CN/article/getTxtFile.do?fileType=Ris&id=6998","title_cn":"我国民族语言文献文本数字化识别问题——基于OCR及其工具","doi":"10.11778/j.jnxb.20241011","jieShuYe":"45","keywordList_en":["ethno-languages","ethnic documents","text recognition","OCR","data construction","Digital humanities"],"endNoteUrl_cn":"https://jnxb.jnu.edu.cn/skb/CN/article/getTxtFile.do?fileType=EndNote&id=6998","zhaiyao_en":"China has over 130 minority languages and more than 10 minority scripts. These have preserved a wealth of ethnic language documents, including a large number of ancient manuscripts and modern printed documents. These records capture the long-standing civilization of the Chinese nation and the knowledge and practices of various ethnic groups in their production and daily life. The content covers a wide range of areas, including politics, economics, law, history, literature, art, religion, medicine, astronomy, and geography, reflecting the exchange, integration, and innovation of various ethnic cultures. Fully utilizing contemporary data science and artificial intelligence (AI) to innovate text recognition technology for minority language documents and achieving the digitization of massive amounts of literature is of great historical and cultural significance, and practical political significance. This effort is crucial for the scientific protection of Chinese minority language document resources and the inheritance of excellent Chinese traditional knowledge and cultural spirit.Collecting and organizing minority language books, newspapers, and manuscripts for large-scale text recognition and digitization is a crucial source for building natural language processing (NLP) and AI datasets. The digitization of minority language documents involves two fundamental tasks: (1) compiling and cataloging various documents to create indexed data; and (2) performing optical character recognition (OCR) on the content of these documents to convert them into computer-processable text files. The recognition of minority language text is the prerequisite for the digitization of document content, while OCR text recognition is key to constructing large-scale corpora and knowledge text data in China's native languages. Efficient document OCR recognition technology has broad applications. It enables publishers to transition from passively receiving manuscripts to actively creating knowledge content, maximizing content production potential. Additionally, it facilitates the extraction of text data such as characters, words, sentences, and paragraphs from vast social language landscapes, addressing the issue of entity name corpora in NLP tasks. Through OCR technology, a large number of language examples and corpora in linguistic works can be automatically extracted and annotated. This enables the large-scale integration and utilization of discrete corpora in mixed-language documents of various minority languages and Chinese, driving a data-oriented shift in linguistic research in China.The text recognition of minority language documents involves two key technologies: the accuracy of single-language recognition, and the differentiation and recognition of texts in documents with mixed scripts. Currently, the OCR technology in China performs well in recognizing documents in Mongolian, Tibetan, Uighur, Kazakh, Korean, and other languages, but it performs poorly at the application level in recognizing mixed-language documents. Therefore, the R&D of text recognition technology for minority language documents in China currently focuses on four main tasks: (1) Solving the text recognition of ancient documents in minority languages; (2) Addressing the recognition and extraction of mixed scripts involving multiple minority languages, the coexistence of Chinese characters and minority scripts, and various linguistic works in minority languages; (3) Advancing the OCR recognition and simultaneous digitization of single-language documents in minority languages; (4) Rapidly developing various tools and integrated platforms for the text recognition of minority language documents in China. To accomplish these tasks, interdisciplinary research that combines linguistics and contemporary AI science is necessary.","bibtexUrl_en":"https://jnxb.jnu.edu.cn/skb/EN/article/getTxtFile.do?fileType=BibTeX&id=6998","abstractUrl_cn":"https://jnxb.jnu.edu.cn/skb/CN/10.11778/j.jnxb.20241011","zuoZheCn_L":"范俊军, 刘贤娴","juanUrl_cn":"https://jnxb.jnu.edu.cn/skb/CN/Y2024","lanMu_en":"","qiUrl_en":"https://jnxb.jnu.edu.cn/skb/EN/Y2024/V46/I6","zuoZhe_EN":"FAN Junjun, LIU Xianxian","risUrl_en":"https://jnxb.jnu.edu.cn/skb/EN/article/getTxtFile.do?fileType=Ris&id=6998","title_en":"Digital Recognition of Minority Language Documents in China—Based on OCR and its ways","hasPdf":"true"},"authorNotes_cn":["范俊军、刘贤娴,暨南大学文学院。"]};if(window.metaData && (!window.metaData.authorNotesCommon_cn && !window.metaData.authorNotesCorresp_cn && window.metaData.authorNotes_cn)){window.metaData.authorNotesCommon_cn = window.metaData.authorNotes_cn;}if(window.metaData && (!window.metaData.authorNotesCommon_en && !window.metaData.authorNotesCorresp_en && window.metaData.authorNotes_en)){window.metaData.authorNotesCommon_en = window.metaData.authorNotes_en;}var _nlmdtdXml = $("#article_nlmdtdXml").val();var magDir = _nlmdtdXml.replace(/\/[^\/]+$/,'');window.metaData.magDir = magDir;new mag_vue({ el: '#metaVue', //dataUrl: mag_currentQikanUrl() + '/EN/article/getRichHtmlJson.do?articleId='+$("#articleId").val(), data:window.metaData, loading:true, doTextFun:function(text){ return doDataJsonText(text); }, doAfterFun:function(selector,jsonData){doMagVueAfter(selector,jsonData);}});var pars = {attachType:'PDF',id:$("#articleId").val(),json:true};jQuery.post($("#mag-common-contextPath").val()+"/CN/article/showArticleFile.do?" + new Date().getTime(), pars, function(data){var json=null;if(data.match(/^\[json\].*/g)){json=eval("("+data.replace(/^\[json\]/g,"")+")");}if(json && json.status == 1){//有权限}else{//无权限$("#no-access-div").show();}});mag_ajax({url:mag_currentQikanUrl() + '/CN/article/getRichHtmlJson.do',data:{articleId:$("#articleId").val(),token:'8a55c7e3dd2840209c3674d56a420d2b',referer:document.referrer},dataType:"json",async:"false",success:function(json){if(json.richHtmlJson && json.richHtmlJson.article){if(json.trans){var _nlmdtdXmlCN = $("#article_nlmdtdXmlCN").val();var magDirCN = _nlmdtdXmlCN.replace(/\/[^\/]+$/,'');json.richHtmlJson.magDir=magDirCN;}else{json.richHtmlJson.magDir=magDir;}new mag_vue({ el: '#bodyVue', data:json.richHtmlJson, doTextFun:function(text){ return doDataJsonText(text); }, doAfterFun:function(selector,jsonData){doMagVueAfter(selector,jsonData);}});}}});//altmetricif($.trim($("#article_doi").val())!=''){$.ajax({url:'https://api.altmetric.com/v1/doi/'+$.trim($("#article_doi").val())+'?callback=altmetric_callback',//url:'https://api.altmetric.com/v1/doi/10.1007/s13238-016-0264-7?callback=altmetric_callback',dataType:'jsonp'});}mag_rich_menu(jQuery);$(function(){ $('#autoMenu').autoMenu();})$("#autoMenu").niceScroll({cursorcolor: "#ccc", //滚动条的颜色cursoropacitymax: 0, //滚动条的透明度，从0-1 touchbehavior: false, //使光标拖动滚动像在台式电脑触摸设备 true滚动条拖动不可用 cursorwidth: "6px", //滚动条的宽度 单位默认px cursorborder: "0", // 游标边框css定义 cursorborderradius: "3px", //滚动条两头的圆角 autohidemode: true, //是否隐藏滚动条 true的时候默认不显示滚动条，当鼠标经过的时候显示滚动条 zindex: "auto", //给滚动条设置z-index值 railvalign:'defaul',railpadding: {top: 0,right: -6,left: 0,bottom: 0}, //滚动条的位置});window.autoMenuFlag=true;$("#relatedArticles").niceScroll({cursorcolor: "#ccc", //滚动条的颜色cursoropacitymax: 0, //滚动条的透明度，从0-1 touchbehavior: false, //使光标拖动滚动像在台式电脑触摸设备 true滚动条拖动不可用 cursorwidth: "6px", //滚动条的宽度 单位默认px cursorborder: "0", // 游标边框css定义 cursorborderradius: "3px", //滚动条两头的圆角 autohidemode: true, //是否隐藏滚动条 true的时候默认不显示滚动条，当鼠标经过的时候显示滚动条 zindex: "auto", //给滚动条设置z-index值 railvalign:'defaul',railpadding: {top: 0,right: -6,left: 0,bottom: 0}, //滚动条的位置});//公式序号 居右jQuery(".mag-xml-disp-formula").each(function() {var _this = jQuery(this);var _label= _this.children("label");if(typeof(_label.html())!="undefined"){_this.children("label").remove();_this.wrap("");_this.parent().parent().append(""+_label.html()+"");}});$('body').click(function(){$('.daochu_div').hide();$('.qipao').hide();})$('.qipao').click(function(event){event.stopPropagation();})/*参考文献显示隐藏*/$('.biaotijiantoush1').click(function(event){if($(this).attr('class')=='glyphicon glyphicon-triangle-top biaotijiantoush biaotijiantoush1'){$(this).parents('.mag_main_zhengwen_left_div_ckwx_table_benwenyiny').next('.mag_main_zhengwen_left_div_ckwx_table_ckwx_xiangqing').stop().slideUp();$(this).attr('class','glyphicon glyphicon-triangle-bottom biaotijiantoush biaotijiantoush1');}else if($(this).attr('class')=='glyphicon glyphicon-triangle-bottom biaotijiantoush biaotijiantoush1'){$(this).parents('.mag_main_zhengwen_left_div_ckwx_table_benwenyiny').next('.mag_main_zhengwen_left_div_ckwx_table_ckwx_xiangqing').stop().slideDown();$(this).attr('class','glyphicon glyphicon-triangle-top biaotijiantoush biaotijiantoush1');}event.stopPropagation();});//加载补充材料mag_ajax_update({ele_id:'showArticle_suppls',url:mag_currentQikanUrl() + '/CN/article/showArticleSuppls.do',data:{articleId:$("#articleId").val()}});//加载专题mag_ajax_update({ele_id:'showArticle_subjects',url:mag_currentQikanUrl() + '/CN/subject/showSubjectListByArticle.do',data:{articleId:$("#articleId").val(),forwardJsp:'showArticle'}});//渲染公式mag_mathJax();});function altmetric_callback(json){if(json && json.score){if(json.score - parseInt(json.score) > 0){$("#altmetric-score").html(parseInt(json.score)+1);}else{$("#altmetric-score").html(parseInt(json.score));}$(".main_content_center_right_duoweidu_div").css("width","25%");$("#altmetric-a").show();$("#altmetric-a").attr("href",json.details_url);}}function richClickAuthor(name){var searchSQL = "((("+mag_advSearchValReplace(name)+"[Author]) AND 2J[Journal]) AND year[Order])";mag_request({url:mag_currentQikanUrl() + '/CN/searchresult',data:{searchSQL:searchSQL}});}function richClickKeyword(keyword){var searchSQL = "((("+mag_advSearchValReplace(keyword)+"[Keyword]) AND 2J[Journal]) AND year[Order])";mag_request({url:mag_currentQikanUrl() + '/CN/searchresult',data:{searchSQL:searchSQL}});}function mag_share(){$("#bShare_more_a").click();}function loadRelatedArticles(){if($("#showRelatedArticles") && $("#showRelatedArticles").length>0){//加载过了}else{mag_loading();mag_ajax_update({ele_id:'relatedArticles-content',url:mag_currentQikanUrl() + '/CN/article/relatedArticle.do',data:{id:$("#articleId").val(), forwardJsp:'showArticle'},complete:function(){mag_layer_close();}});}}function showMetrics(){mag_window_location(mag_currentQikanUrl() + '/CN/metrics/'+$("#articleId").val());}function piaofuFn(a,fnId){var content=''+$("#mag-rich-fn-"+fnId).html()+'';piaofu(a,content);}function huoqukuanduimg(){//}function doDataJsonText(text){var str = text;//var _nlmdtdXml = $("#article_nlmdtdXml").val();//var magDir = _nlmdtdXml.replace(/\/[^\/]+$/,'');str = str.replace(/";ss += "";var shuomingTitle = "";var shuomingContent = '';if(fig.titleCn && fig.titleCn!=''){shuomingTitle = fig.titleCn;if(fig.titleContentCn && fig.titleContentCn!=''){shuomingContent += fig.titleContentCn;}}if(shuomingTitle==''){if(fig.titleEn && fig.titleEn!=''){shuomingTitle = fig.titleEn;}if(fig.titleContentEn && fig.titleContentEn!=''){shuomingContent += fig.titleContentEn;}}else{if(fig.titleEn && fig.titleEn!=''){shuomingContent += ""+fig.titleEn+"

来源：2024年第6期

《暨南学报(哲学社会科学版)》期刊编辑部

查看暨南学报(哲学社会科学版)杂志2024年第6期