Subsequent, you'll want to choose how you want the crawler to deal with several Net systems. You can find an ongoing discussion with regards to the intelligence of online search engine crawlers. It isn't totally clear If they're full-blown headless browsers or just glorified curl scripts (or anything between). https://www.youtube.com/watch?v=EVeTP0aV8gY