Challenges in Web Crawling for Data Collection

Georgi Cholakov 1), Emil Doychev 1),
Svetla Koeva 2)
1) Plovdiv University „Paisii Hilendarski“ Faculty of Mathematics and Informatics
2) Institute for Bulgarian Language „Prof. Lyubomir Andreychin“ - Bulgarian Academy of Sciences

https://doi.org/10.53656/math2024-1-1-cha

Abstract. The article presents the challenges of implementing a System for data retrieval and visualisation from the Internet by crawling language resources from the Hugging Face repository and extracting the associated data. The data in the system is updated at regular intervals to track the dynamics of language resource creation for different time periods. The article presents: a) the analysis of the available data and its structure; b) the chosen method for crawling the pages and extracting the data. The shared experience of overcoming the specific challenges can serve to solve similar problems related to the extraction of data from the Internet, a task that often has to be solved in various projects (including school projects).
Keywords: web crawling; automatic data extraction; linguistic datasets

Свързани статии:

Two-Parametric Problem for Optimal Distribution of Resources Developing a Positive Mindset to Entrepreneurship and Social Activity through the Innovative Learning System (InnoSchool) RESULTS OF THE EXTERNAL ASSESSMENT IN BIOLOGY AND HEALTH EDUCATION – VIII GRADE (2018/2019 SCHOOL YEAR) Is there a Place for a “New Art” in the Classroom? Is the 1:1 Model an Alternative