More web crawling capabilities
Registered by
Dimitris Kalamaras
SocNetV should be able to crawl web pages (ie mailing list archives or facebook open graph) to simplify the work of network analyst/researcher.
Blueprint information
- Status:
- Complete
- Approver:
- Dimitris Kalamaras
- Priority:
- Essential
- Drafter:
- Dimitris Kalamaras
- Direction:
- Approved
- Assignee:
- Dimitris Kalamaras
- Definition:
- Superseded
- Series goal:
- Accepted for trunk
- Implementation:
-
Implemented
- Milestone target:
- None
- Started by
- Dimitris Kalamaras
- Completed by
- Dimitris Kalamaras
Whiteboard
Initial web crawler code has been added to SocNetV v. 0.70 and revamped in v1.6
The new code is good and pretty quick, but it could offer some other nice features :
1. Support for Mailing Lists emails crawling
2. Support for Facebook profile crawling
Ideas:
A. Can we add a menu item for the user to open the link in a browser? :)
B. Crawler should be able to count how many times a word appears in a page
Crawler code is there, but we need minor modifications to
a) form to add a textfield
b) code to count the used-defined
(?)
