Traditional crawler or AI-assisted crawler? How to choose?

Foreword

In the field of data fetching, traditional crawlers and AI-assisted crawlers have their own advantages. The traditional crawler crawls data based on rules, which is suitable for websites with stable structure and clear rules. However, with the frequent changes and complications of the website structure, traditional crawlers have gradually exposed their limitations. In contrast, AI-assisted crawlers use artificial intelligence technology to intelligently analyze web pages and adapt to changes, showing higher flexibility and accuracy. So, in the face of different grasping needs, how should we choose? This article will deeply discuss the characteristics and advantages and disadvantages of traditional crawlers and AI-assisted crawlers, so as to provide you with decision-making reference.
What are the traditional crawlers and AI-assisted crawlers respectively

Traditional crawler

Traditional crawlers mainly rely on fixed rules or patterns to grab web data. They typically locate and extract the required information by identifying specific elements in a web page, such as class names, tags, or structures. However, the limitations of this approach are obvious. Once a site is updated to change the original class name, label, or structure, traditional crawlers will fail because they cannot recognize the new elements, resulting in data fetching failures or errors.

AI-assisted crawler

Ai-assisted crawlers intelligently analyze and understand the content of web pages to more accurately locate and extract the required information. Through technologies such as natural language processing, they are able to understand the semantic information of the web page, so that they can more precisely locate the required data, and even after the website is updated, AI-assisted crawlers can continue to effectively crawl data.

Example

The crawler uses x-crawl. The crawled websites are all real. To avoid disputes, https://www.example.com is used instead.

Traditional crawler to obtain movie information of movie rankings through specific elements in the web page
Crawler + AI, the crawler is paired with AI to obtain movie information from the movie rankings

Traditional crawler

Traditional crawler, obtains movie information of movie rankings through specific elements in the webpage

import { createCrawl } from 'x-crawl' //Create a crawler application
const crawlApp = createCrawl() // crawlPage is used to crawl pages
crawlApp.crawlPage('https://www.example.com').then(async (res) => { const { page, browser } = res.data // Wait for the element to appear on the page await page.waitForSelector('#wrapper #content .article') const filmHandleList = await page.$$('#wrapper #content .article table') const pendingTask = [] for (const filmHandle of filmHandleList) { // Cover link (picture) const picturePending = filmHandle.$eval('td img', (img) => img.src) //Movie name(name) const namePending = filmHandle.$eval( 'td:nth-child(2) a', (el) => el.innerText.split(' / ')[0] ) // Introduction (info) const infoPending = filmHandle.$eval( 'td:nth-child(2) .pl', (el) => el.textContent ) // score const scorePending = filmHandle.$eval( 'td:nth-child(2) .star .rating_nums', (el) => el.textContent ) // Number of comments (commentsNumber) const commentsNumberPending = filmHandle.$eval( 'td:nth-child(2) .star .pl', (el) => el.textContent?.replace(/\(|\)/g, '') ) pendingTask.push([ namePending, picturePending, infoPending, scorePending, commentsNumberPending ]) } const filmInfoResult = [] let i = 0 for (const item of pendingTask) { Promise.all(item).then((res) => { // filmInfo is a movie information object, the order is determined before const filmInfo = [ 'name', 'picture', 'info', 'score', 'commentsNumber' ].reduce((pre, key, i) => { pre[key] = res[i] return pre }, {}) //Save each movie information filmInfoResult.push(filmInfo) //Last processing if (pendingTask.length === ++i) { browser.close() // Arrange, decide whether it is more or less based on the quantity const filmResult = { element: filmInfoResult, type: filmInfoResult.length > 1 ? 'multiple' : 'single' } console.log(filmResult) } }) }
})