net.sf.jmatchparser.util
Class PrefetchDownloader

java.lang.Object
  net.sf.jmatchparser.util.PrefetchDownloader

public class PrefetchDownloader
extends Object

Utility class that makes it easy to download files in the background as they are detected (over slow links) while still doing all of the parsing in a single thread. The main idea is, that whenever you encounter a file you have to parse later, you add it to the prefetcher by calling the download(String, String) method. All documents added to the prefetcher are loaded in a configurable number of background threads (and stored into the CachingDownloader's cache. When ready to parse the new document, you can request the result by calling waitFor(String), waiting for it if needed.

Constructor Summary

PrefetchDownloader(CachingDownloader cd, int threadcount, boolean cacheResults)
Create a new prefetch downloader.

Method Summary

void	download(String url, String cacheName) Add a document to download
void	download(String url, String postData, String cacheName) Add a document to download, including POST data.
void	shutdown() Wait until all prefetching is finished and shut down the prefetch threads.
InputStream	waitFor(String cacheName) Wait until a given document has been prefetched.
void	waitPrefetch() Wait until all prefetching is finished.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PrefetchDownloader

public PrefetchDownloader(CachingDownloader cd,
                          int threadcount,
                          boolean cacheResults)

Create a new prefetch downloader.

Parameters:

cd - Caching downloader to use

threadcount - Number of prefetch threads to start

cacheResults - Whether to cache the results in the prefetcher. If not, they are only written to the cache and can be retrieved from there

Method Detail

download

public void download(String url,
                     String cacheName)

Add a document to download

Parameters:

url - URL of the document

cacheName - Name of the document in the cache

See Also:

CachingDownloader.download(String, String)

download

public void download(String url,
                     String postData,
                     String cacheName)

Add a document to download, including POST data.

Parameters:

url - URL of the document

postData - postData to post

cacheName - Name of the document in the cache

See Also:

CachingDownloader.download(String, String, String)

waitPrefetch

public void waitPrefetch()
                  throws InterruptedException

Wait until all prefetching is finished. This is useful if the results do not have to be parsed, just downloaded, to make sure to wait until everything requested by now has been downloaded.

Throws:

InterruptedException

waitFor

public InputStream waitFor(String cacheName)
                    throws InterruptedException

Wait until a given document has been prefetched.

Parameters:

cacheName - Name of the document in the cache

Returns:

The document content as a stream

Throws:

InterruptedException

shutdown

public void shutdown()
              throws InterruptedException

Wait until all prefetching is finished and shut down the prefetch threads.

Throws:

InterruptedException