scraperjs

Installing

npm install scraperjs

grunt test

var scraperjs = require('scraperjs');
scraperjs.StaticScraper.create('https://news.ycombinator.com/')
	.scrape(function($) {
		return $(".title a").map(function() {
			return $(this).text();
		}).get();
	})
	.then(function(news) {
		console.log(news);
	})

Dynamic Scraper

var scraperjs = require('scraperjs');
scraperjs.DynamicScraper.create('https://news.ycombinator.com/')
	.scrape(function($) {
		return $(".title a").map(function() {
			return $(this).text();
		}).get();
	})
	.then(function(news) {
		console.log(news);
	})

Show me the way! (aka Routes)

Example

var scraperjs = require('scraperjs'),
	router = new scraperjs.Router();

router
	.otherwise(function(url) {
	console.log("Url '"+url+"' couldn't be routed.");
});

var path = {};

router.on('https?://(www.)?youtube.com/watch/:id')
	.createStatic()
	.scrape(function($) {
		return $("a").map(function() {
			return $(this).attr("href");
		}).get();
	})
	.then(function(links, utils) {
		path[utils.params.id] = links
	})

router.route("https://www.youtube.com/watch/YE7VzlLtp-4", function() {
	console.log("i'm done");
});

StaticScraper, DynamicScraper and ScraperPromise

var scraperPromise = scraperjs.StaticScraper.create() // or DynamicScraper

• ``onStatusCode(code:number, callback:function(utils:Object))``, executes the callback when the status code is equal to the code,
• ``onStatusCode(callback:function(code:number, utils:Object))``, executes the callback when receives the status code. The callback receives the current status code,
• ``delay(time:number, callback:function(last:?, utils:Object))``, delays the execution of the chain by time (in milliseconds),
• ``timeout(time:number, callback:function(last:?, utils:Object))``, executes the callback function after time (in milliseconds),
• ``then(lastResult:?, callback:function(last:?, utils:Object))``, executes the callback after the last promise,
• ``async(callback:function(last:?, done:function(result:?, err:?), utils))`, executes the callback, stopping the promise chain, resuming it when the `done`` function is called. You can provide a result to be passed down the promise chain, or an error to trigger the catch promise,
• ``catch(callback:function(error:Error, utils:Object))``, executes the callback when there was an error, errors block the execution of the chain even if the promise was not defined,
• ``done(callback:function(last:?, utils:Object))``, executes the callback at the end of the promise chain, this is always executed, even if there was an error,
• ``get(url:string)``, makes a simple HTTP GET request to the url. This promise should be used only once per scraper.
• ``request(options:Object)`, makes a (possibly) more complex HTTP request, scraperjs uses the [request](https://github.com/mikeal/request) module, and this method is a simple wrapper of `request.request()``. This promise should be used only once per scraper.
• ``scrape(scrapeFn:function(...?), callback:function(result:?, utils:Object)=, ...?)`, scrapes the page. It executes the scrapeFn and passes it's result to the callback. When using the StaticScraper, the scrapeFn receives a jQuery function that is used to scrape the page. When using the DynamicScraper, the scrapeFn doesn't receive anything and can only return a [JSON-serializable](https://github.com/sgentle/phantomjs-node/wiki#evaluating-pages) type. Optionally an arbitrary number of arguments can be passed to the scraping function. The callback may be omitted, if so, the result of the scraping may be accessed with the ` then ` promise or ` utils.lastReturn `` in the next promise.

DynamicScraper.create()
	.get("http://news.ycombinator.com")
	.then(function(_, utils) {
		utils.stop();
		// utils.params.paramName
	});

The `` utils `` object

• `` .lastResult ``, value returned in the last promise
• `` .stop() ``, function to stop the promise chain,
• `` .url ``, url provided to do the scraper,
• `` .params ``, object with the parameters defined in the router matching pattern.

A more powerful DynamicScraper.

var ds = scraperjs.DynamicScraper
	.create('http://news.ycombinator.com')
	.async(function(_, done, utils) {
		utils.scraper.inject(__dirname+'/path/to/code.js', function(err) {
			// in this case if there was an error won't fire catch promise.
			if(err) {
				done(err);
			} else {
				done();
			}
		});
	})
	.scrape(function() {
		return functionInTheCodeInjected();
	})
	.then(function(result) {
		console.log(result);
	});

Router

var router = new scraperjs.Router(options);

• `` firstMatch ``, a boolean, if true the routing will stop once the first path is matched, the default is false.

• ``on(path:string|RegExp|function(url:string))`, makes the promise for the match url or regular expression, alternatively you can use a function to accept or not a passed url. The promises `get` or `request` and `createStatic` or `createDynamic`` are expected after the on promise.
• ``get()``, makes so that the page matched will be requested with a simple HTTP request,
• ``request(options:Object)``, makes so that the page matched will be requested with a possible more complex HTTP request, , scraperjs uses the request module, and this method is a simple wrapper of request.request(),
• ``createStatic()`, associates a static scraper to use to scrape the matched page, this returns ScraperPromise, so any promise made from now on will be made over a ScraperPromise of a StaticScraper. Also the `done`` promise of the scraper will not be available.
• ``createDynamic()`, associates a dynamic scraper to use to scrape the matched page, this returns ScraperPromise, so any promise made from now on will be made over a ScraperPromise of a DynamicScraper. Also the `done`` promise of the scraper will not be available.
• ``route(url:string, callback:function(boolean))``, routes an url through all matched paths, calls the callback when it's executed, true is passed if the route was successful, false otherwise.
• ``use(scraperInstance:ScraperPromise)``, uses a ScraperPromise already instantiated.
• ``otherwise(callback:function(url:string))``, executes the callback function if the routing url didn't match any path.
• ``catch(callback:function(url:string, error:Error))`, executes the callback when an error occurred on the routing scope, not on any scraper, for that situations you should use the `catch`` promise of the scraper.

Notes

• Scraperjs always fetches the document with request, and then when using a DynamicScraper, leverages phantom's setContent() to set the body of the page object. This will result in subtly different processing of web pages compared to directly loading a URL in PhantomJS.

More

• ``async``, for flow control
• ``request``, to make HTTP requests, again, if you want more complex requests see it's documentation
• ``phantom`](https://github.com/sgentle/phantomjs-node) + [`phantomjs``, phantom is an awesome module that links node to phantom, used in the DynamicScraper
• ``cheerio``, light and fast DOM manipulation, used to implement the StaticScraper
• ``jquery``, to include jquery in the DynamicScraper
• although ``Routes.js`` is great, scraperjs doesn't use it to maintain it's "interface layout", but the code to transform the path given on the on promise to regular expressions is from them