* crawler
** sqlite
urls:
|---------+----------+--------|
| id: num | url: str | status |
|---------+----------+--------|
status: one of pending/done/failed

links:
|--------------+------------|
| from: url_id | to: url_id |
|--------------+------------|


queue
| url1 |
| url2 |
| url3 |

thread:
+ lock queue
+ dequeue url_a
+ unlock queue
+ set url_a status to pending
+ fetch page
+ filter out urls
+ for each url_b in urls:
  + insert url_b into sqlite or ignore (if already exists)
  + insert link (url_a -> url_b) into sqlite
  + enqueue url_b if it did not exist in sqlite (lock -> enqueue -> unlock)
+ set url_a status to done