Description: This is the documentation for ubh, the Usenet Binary Harvester.
Keywords: ubh usenet binary harvester perl net::nntp nntp libnet MIME MIME::Parser

# ubh - The Usenet Binary Harvester

- NAME
- SYNOPSIS
- DESCRIPTION
- OPTIONS
- INSTALLATION

- Win32 / ActivePerl
- Unix
- Mac OS X
- MacPerl

- SETUP
- USAGE EXAMPLES
- WEBPAGE
- COPYRIGHT and LICENSE

# NAME

`ubh` - The Usenet Binary Harvester

# SYNOPSIS

`ubh -c file (-S|-M) -A -b file -C -d -D -g [(-a|-s)]|[(-f|-l) num] -i [(-I|-X) pattern] -L -n -O [yes|no|skip] -u -w -y -z`

# DESCRIPTION

`ubh` is the Usenet Binary Harvester, a Perl program which automatically discovers, downloads, and decodes single and multi-part binary Usenet postings.

`ubh` decodes single and multi-part uuencoded binaries.

`ubh` decodes single and multiple part MIME binaries. It also combines and decodes multi-part MIME `message/partial` binaries.

`ubh` decodes single and multiple part yEnc encoded binaries.

You can specify search filters to select articles to download via Perl regular expression syntax.

`ubh` uses a standard `.newsrc` file to control which groups to process, and uses the `.newsrc` to keep track of articles already processed.

`ubh` caches downloaded subject headers in the `.ubhcache` file.

`ubh` automatically eliminates crossposted articles by marking them as read in your `.newsrc`.

`ubh` can connect to multiple servers, combining the available single and multi-part subjects from those servers, and can assemble multi-part binaries from parts which are spread across those servers.

By default `ubh` will only consider articles with a well-formed `Subject` header. A well-formed `Subject` header is one that contains a file name with an extension which matches the extension filter. Multi-part `Subject` headers will be recognized if they contain a part/total designator of the form `[`m`/`n`]` or `(`m`/`n`)`, where m is the part number and n is the total number of parts. Part numbers begin with `1` and may or may not contain leading zeroes. A part number of `0` is ignored.

`ubh` provides an interactive article preselection option to allow you to preview the `Subject:` headers for multi-part binaries, and specify which binaries you wish to download.

`ubh` runs equally well under Unix-based Perl, Active Perl on Win32 platforms, and Mac OS X.

`ubh` requires `Net:NNTP` (from the `libnet` distribution, `News:Newsrc` (which itself requires `Set::IntSpan`), `MIME::Parser` (from the `MIME-tools` distribution, which itself requires `MIME::Base64`, `IO::Stringy`, and the `MailTools` distribution), and `String::CRC32`. See the section INSTALLATION for details.

# OPTIONS

These options apply to single part article processing:

Option | Description
--- | ---
-S | Process only single part articles.
-g | This option ("g" for "greedy") will download and process each unread article even if the subject does not contain a filename which matches the single part extension filter.

These options apply to multi-part article processing:

Option | Description
--- | ---
-M | Process only multi-part articles.
-i | Interactive preselection of multi-part articles.

These options apply to both single part and multi-part article processing:

Option | Description
--- | ---
-A | Enables disk-based article assembly. This will download the articles to disk (instead of RAM) prior to decoding.
-b file | Batch file processing. file should contain a list of article subjects (from the .log files; see -s), one per line, to be downloaded. Mutually exclusive with -I and -X.
-C | Cleans up filenames. Without -C, `ubh` will perform certain mandatory character substitutions. This option performs further cleanup of the file names by replacing all non-alphanumeric characters in a filename with the EVILCHAR. Kills all non-alphanumeric leading characters. This will eliminate spaces in filnames, as well as all other undesireable (and possibly illegal) characters.
-d | Diagnostic mode. Downloads and writes all unread articles in raw form. This occurs prior to single and multi-part filtering. It's very useful to look at the raw articles to see why they are failing to be selected for downloading. Helpful for reverse-engineering new or bizarre encoding formats. You can also use this to perform your own post-processing directly on the raw articles. Articles are output using the article ID as the file name with the extension `.dump`.
-D | Dump mode. Downloads and writes all selected single and complete multi-part article to disk, instead of decoding. Very useful to look at the raw articles to see why they are failing to be unencoded. Helpful for reverse-engineering new or bizarre encoding formats. You can also use this to perform your own post-processing directly on the raw articles.
-c file | Use file as configuration file, instead of the default. On Win32 platforms, the default is `ubhrc`. On Unix platforms, the default is `.ubhrc`. By default, `ubh` will look in your home directory for the `.ubhrc`/`ubhrc` file.
-G group | By default `ubh` will process every subscribed group in the `.newsrc`. This option specifies the name of one group to be processed. Note that group must exist in the `.newsrc` and must be subscribed.
-a | Process all articles, but disregard the newsrc (ie, consider all articles even if they are marked as read in the newsrc, and do not catch up the group at the end of processing of the group).
-f num | Process the first num articles. Updates newsrc.
-l num | Process the last num articles. Updates newsrc.
-s | Log all subjects to `subjects.log`. Log multi-part subjects to `multiparts.log`. Doesn't download anything. Disregards newsrc. If `FORCEDIR` is in effect, the names of the file will be prepended with the group name.
-I regexp | Inclusion search filter (double quote on command line). regexp is any valid Perl regular expression.
-X regexp | Exclusion search filter (double quote on command line). regexp is any valid Perl regular expression.
-L | Long filenames - uses the article subject as the filename. This makes life easier because many folks encode their files with terribly vague filenames.
-n | Updates the `.newsrc` every time an article is processed, instead of waiting until the entire group has been processed.
-O opt | Tells `ubh` what to do when it downloads a file and a file by that name already exists. `no` tells ubh to create a unique filename (by prepending the article number to the filename). `yes` tells ubh to overwrite the existing file with the new file. `skip` tells ubh to skip the incoming file and keep the existing file. In the case of multipart uuencoded binaries, `ubh` will download the first part to determine the file name; if a file with the same name already exists, `ubh` will skip the rest of the parts for that binary. The default is `no`.
-u | Prints out a brief usage summary.
-w | Prints out warranty information.
-y | `chmod 0666` on all output files.
-z | Marks articles that don't pass inclusion/exclusion as read. This cleans up the `.newsrc` dramatically.

# INSTALLATION

`ubh` requires `Net:NNTP` (from the `libnet` distribution, `News:Newsrc` (which itself requires `Set::IntSpan`), `MIME::Parser` (from the `MIME-tools` distribution, which itself requires `MIME::Base64`, `IO::Stringy`, and the `MailTools` distribution), and `String::CRC32`. These modules may already be installed on your system. To determine whether these modules have already been installed, use the following commands: `perl -e "use` module `;"` `Net::NNTP` is part of the `libnet` distribution, which provides an entire family of networking modules. If you need to install one or more of these modules, follow the instructions below for your particular platform.

## Win32 / ActivePerl

You may be able to use the Perl Package Manager (PPM) to download and install the required modules. If not, you can use Microsoft's `nmake` utility to install them manually.

## Unix

You will download the modules from your nearest CPAN mirror.

    modules/by-module/Net/libnet-1.0607.tar.gz
    modules/by-module/Set/Set-IntSpan-1.07.tar.gz
    modules/by-module/News/News-Newsrc-1.07.tar.gz
    IO-stringy-1.220.tar.gz
    MIME-Base64-2.12.tar.gz
    MailTools-1.15.tar.gz
    MIME-tools-5.411.tar.gz
    String-CRC32-1.2.tar.gz

Feel free to use more recent versions of these modules, if available. For each of those files, execute the following commands:

    gunzip < module.tar.gz | tar xvf -
    cd module
    perl Makefile.PL
    make
    make test
    make install

For more details, review the installation instructions provided with each module.

## Mac OS X

Use the Unix `.tar.gz` distribution. You need to download (from Apple) "Unix tools for OS X", and then proceed with the module installation as given above.

## MacPerl

Use the Unix `.tar.gz` distribution. Proceed with the module installation as given above.

# SETUP

You specify which groups to process with a standard newsrc file. The format of a newsrc file is one newsgroup name per line, followed by a colon. `ubh` will update this file and place the numbers of articles it has already processed for a given group on that group's line in the newsrc file after the colon. The colon character denotes that you are subscribed to the newsgroup. You can un-subscribe from a group by changing the colon to an exclamation point; this will cause `ubh` to skip over that newsgroup during processing. Here is an example newsrc file:

    alt.binaries.sounds.mp3.1990s: 15558-23146
    alt.binaries.sounds.mp3.1980s: 30139-35021
    alt.binaries.pictures.autos:
    alt.binaries.pictures.cartoons! 63671-64660
    alt.binaries.pictures.hockey: 1-1406

You specify program options with a ubhrc file. On Win32 systems, the default ubhrc file name is `ubhrc`. On Unix systems, the the default ubhrc file name is `.ubhrc`. By default, `ubh` will look in your home directory for the `.ubhrc`/`ubhrc` file. You can specify an alternate configuration file via the `-c` option. Here are the available options. Specify one option per line, OPTION`=`value. Comments begin with `#`.

Keyword | Description
--- | ---
NNTPSERVER | Specifies the name of the news server to connect to. Default is `news`. You can specify more than one server to connect to by using the NNTPSERVER keyword once for each server. If you specify multiple NNTPSERVERs, you must also supply a unique NEWSRCNAME for each server. The order in which your NNTPSERVERs appear in the `ubhrc` determines the priority (from highest to lowest) of what server the articles will be downloaded from.
NNTPRETRIES | Number of times to retry NNTP commands.
NNTPHDRSIP | If the group that `ubh` is processing has more than this number of headers, then `ubh` will request this many headers at a time. Default is 2500.
NEWSRCNAME | The name of the newsrc file. On Win32 systems, the default newsrc file name is `newsrc`. On Unix systems, the default newsrc file name is `.newsrc`.
DATADIR | Directory to store downloaded group subdirectories and downloaded binaries. Default is `data`.
FORCEDIR | Forces output to a specific directory instead of the default `data/`newsgroup-name. Handy for multiple newsgroups which hold similar stuff.
TEMPDIR | Directory to store encoded articles prior to decoding. Default is DATADIR.
CACHEDIR | Directory to store cached article headers (the `.ubhcache` files. Default is DATADIR.
PERMISSION | Permission to apply to created subdirectories. Default is `0777`.
EVILCHAR | Character to use when replacing evil characters in file names. Default is "\_" (literal underscore).
MULTI_EXT | Multi-part article file extensions. Any valid Perl regular expression. See the variable `$multi_ext` in the code for the default set of extensions.
SINGLE_EXT | Single part article file extensions. Any valid Perl regular expression. See the variable `$single_ext` in the code for the default set of extensions.
EXTENSIONS | Sets both `MULTI_EXT` and `SINGLE_EXT`.
ACCOUNT and PASSWORD | News server account and password. You must define both of these if your server requires them. By default, `ubh` will access the server without authentication.
DECODER | Custom external article decoder. Forces -A to be on. You can pass `%d` to indicate the default output directory and `%f` to indicate the input file name. For example:
    DECODER = uudeview -i -a -m +e .mp3.nfo -p %d %f
By default, `ubh` will use MIME::Tools to decode uuencoded and MIME articles. `ubh` will use its own internal yEnc decoder for yEnc enocded articles.
OPT_g OPT_i OPT_A OPT_b OPT_d OPT_D OPT_G OPT_a OPT_f OPT_l OPT_n OPT_r OPT_s OPT_S OPT_M OPT_I OPT_X OPT_C OPT_L OPT_O OPT_z OPT_Z OPT_y | All of the command-line options (except `-u`, `-c`, and `-w`) can be set in the ubhrc file, using `OPT_`x where x is the corresponding command line option letter.

Here is an example ubhrc file. This can be used to harvest single and multi-part binaries in the pictures and multimedia newsgroups.

    NNTPSERVER = binaries.newsfeeds.com
    NEWSRCNAME = newsrc_nfdc
    DATADIR    = data
    MULTI_EXT  = (?i)asf|avi|gif|jpg|mov|mpg|rm
    SINGLE_EXT = (?i)asf|avi|gif|jpg|mov|mpg|rm
    ACCOUNT = bart
    PASSWORD = the+simpsons
    OPT_g = 1

Here is another example ubhrc file. This demonstrates how to search for MP3s by a particular set of artists:

    NNTPSERVER = mp3.newsfeeds.com
    NEWSRCNAME = newsrc_mp3
    DATADIR    = data
    MULTI_EXT  = (?i)mp3
    SINGLE_EXT = dontcare
    OPT_a = 1
    OPT_M = 1
    OPT_I = (?i)oasis|horton.*heat|smiths|morrissey|bjork|foo.*fighters|green.*day

Here is an example ubhrc file which demonstrates how to access multiple servers:

    NNTPSERVER = news-server.san.rr.com
    NNTPHDRSIP = 2000
    NEWSRCNAME = newsrc_news-server.san.rr.com

    NNTPSERVER = binaries2.newsfeeds.com
    ACCOUNT    = johnedward
    PASSWORD   = crossing+over
    NNTPHDRSIP = 1000
    NEWSRCNAME = newsrc_binaries2.newsfeeds.com

# USAGE EXAMPLES

Here are some command-line usage examples:

    ubh -i -M -I (?i)rem|r.e.m.|u2|korn|hanson
    ubh -S -l 1000

# WEBPAGE

http://ubh.sourceforge.net/

# COPYRIGHT and LICENSE

Copyright © 2000-2002 Gerard Lanois
P.O. Box 507264 San Diego, CA 92150-7264

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA