blob: ea362f37421a5f5c8c0e1da6ca02b506bef1b389 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
* crawler
** sqlite
urls:
|---------+----------+--------|
| id: num | url: str | status |
|---------+----------+--------|
status: one of pending/done/failed
links:
|--------------+------------|
| from: url_id | to: url_id |
|--------------+------------|
queue
| url1 |
| url2 |
| url3 |
thread:
+ lock queue
+ dequeue url_a
+ unlock queue
+ set url_a status to pending
+ fetch page
+ filter out urls
+ for each url_b in urls:
+ insert url_b into sqlite or ignore (if already exists)
+ insert link (url_a -> url_b) into sqlite
+ enqueue url_b if it did not exist in sqlite (lock -> enqueue -> unlock)
+ set url_a status to done
|
