Rewriting with .htaccess

Routing requests

Again, these rules direct all requests to index.php, except specified file types:

RewriteEngine on
RewriteBase /
RewriteRule !\.(js|ico|txt|gif|jpg|png|css)$ index.php

Handling file and directory exceptions

These rules (used immediately prior to the RewriteRule above) exclude real files and directories from the rewriting and lets them pass through unaffected:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

You can also simply allow a specified group of files to pass through unaffected by using this line:

RewriteRule  ^(foo|bar).*  - [L]

In this case, files foo.* and bar.* will be accessed normally.

For more information, see Jayson Minard’s Blueprint for PHP Applications: Bootstrapping.

 

source: http://www.yourhtmlsource.com/sitemanagement/urlrewriting.html

 

URL Rewriting

---

by Ross Shannon

The Apache server’s mod_rewrite module gives you the ability to transparently redirect one URL to another, without the user’s knowledge. This opens up all sorts of possibilities, from simply redirecting old URLs to new addresses, to cleaning up the ‘dirty’ URLs coming from a poor publishing system — giving you URLs that are friendlier to both readers and search engines.

An Introduction to Rewriting

Readable URLs are nice. A well designed website will have a logical file system layout, with smart folder and file names, and as many implementation details left out as possible. In the most well designed sites, readers can guess at filenames with a high level of success.

However, there are some cases when the best possible information design can’t stop your site’s URLs from being nigh-on impossible to use. For instance, you may be using a Content Management System that serves out URLs that look something like

http://www.example.com/viewcatalog.asp?category=hats&prodID=53

This is a horrible URL, but it and its brethren are becoming increasingly prevalent in these days of dynamically-generated pages. There are a number of problems with an URL of this kind:

• It exposes the underlying technology of the website (in this case ASP). This can give potential hackers clues as to what type of data they should send along with the query string to perform a ‘front-door’ attack on the site. Information like this shouldn’t be given away if you can help it.

  Even if you’re not overly concerned with the security of your site, the technology you’re using is at best irrelevant — and at worst a source of confusion — to your readers, so it should be hidden from them if possible.

  Also, if at some point in the future you decide to change the language that your site is based on (to » PHP, for instance); all your old URLs will stop working. This is a pretty serious problem, as anyone who has tackled a full-on site rewrite will attest.

• The URL is littered with awkward punctuation, like the question mark and ampersand. Those & characters, in particular, are problematic because if another webmaster links to this page using that URL, the un-escaped ampersands will mess up their XHTML conformance. They will have to laboriously replace all the ampersands with & character entities, which is often forgotten.
• Some search engines won’t index pages which they think are generated dynamically. They’ll see that question mark in the URL and just turn their asses around.

Luckily, using rewriting, we can clean up this URL to something far more manageable. For example, we could map it to

http://www.example.com/catalog/hats/53/

Much better. This URL is more logical, readable and memorable, and will be picked up by all search engines. The faux-directories are short and descriptive. Importantly, it looks more permanent.

To use mod_rewrite, you supply it with the link text you want the server to match, and the real URLs that these URLs will be redirected to. The URLs to be matched can be straight file addresses, which will match one file, or they can be regular expressions, which will match many files.

Basic Rewriting

Some servers will not have » mod_rewrite enabled by default. As long as the » module is present in the installation, you can enable it simply by starting a .htaccess file with the command

RewriteEngine on

Put this .htaccess file in your root so that rewriting is enabled throughout your site. You only need to write this line once per .htaccess file.

Basic Redirects

We’ll start off with a straight redirect; as if you had moved a file to a new location and want all links to the old location to be forwarded to the new location. Though you shouldn’t really ever » move a file once it has been placed on the web; at least when you simply have to, you can do your best to stop any old links from breaking.

RewriteEngine on
RewriteRule ^old\.html$ new.html

Though this is the simplest example possible, it may throw a few people off. The structure of the ‘old’ URL is the only difficult part in this RewriteRule. There are three special characters in there.

• The caret, ^, signifies the start of an URL, under the current directory. This directory is whatever directory the .htaccess file is in. You’ll start almost all matches with a caret.
• The dollar sign, $, signifies the end of the string to be matched. You should add this in to stop your rules matching the first part of longer URLs.
• The period or dot before the file extension is a special character in regular expressions, and would mean something special if we didn’t escape it with the backslash, which tells Apache to treat it as a normal character.

So, this rule will make your server transparently redirect from old.html to the new.html page. Your reader will have no idea that it happened, and it’s pretty much instantaneous.

Forcing New Requests

Sometimes you do want your readers to know a redirect has occurred, and can do this by forcing a new HTTP request for the new page. This will make the browser load up the new page as if it was the page originally requested, and the location bar will change to show the URL of the new page. All you need to do is turn on the [R] flag, by appending it to the rule:

RewriteRule ^old\.html$ new.html [R]

Using Regular Expressions

Now we get on to the really useful stuff. The power of mod_rewrite comes at the expense of complexity. If this is your first encounter with regular expressions, you may find them to be a tough nut to crack, but the options they afford you are well worth the slog. I’ll be providing plenty of examples to guide you through the basics here.

Using regular expressions you can have your rules matching a set of URLs at a time, and mass-redirect them to their actual pages. Take this rule;

RewriteRule ^products/([0-9][0-9])/$ /productinfo.php?prodID=$1

This will match any URLs that start with ‘products/’, followed by any two digits, followed by a forward slash. For example, this rule will match an URL like products/12/ or products/99/, and redirect it to the PHP page.

The parts in square brackets are called ranges. In this case we’re allowing anything in the range 0-9, which is any digit. Other ranges would be [A-Z], which is any uppercase letter; [a-z], any lowercase letter; and [A-Za-z], any letter in either case.

We have encased the regular expression part of the URL in parentheses, because we want to store whatever value was found here for later use. In this case we’re sending this value to a PHP page as an argument. Once we have a value in parentheses we can use it through what’s called a back-reference. Each of the parts you’ve placed in parentheses are given an index, starting with one. So, the first back-reference is $1, the third is $3 etc.

Thus, once the redirect is done, the page loaded in the readers’ browser will be something like productinfo.php?prodID=12 or something similar. Of course, we’re keeping this true URL secret from the reader, because it likely ain’t the prettiest thing they’ll see all day.

Multiple Redirects

If your site visitor had entered something like products/12, the rule above won’t do a redirect, as the slash at the end is missing. To promote good URL writing, we’ll take care of this by doing a direct redirect to the same URL with the slash appended.

RewriteRule ^products/([0-9][0-9])$ /products/$1/ [R]

Multiple redirects in the same .htaccess file can be applied in sequence, which is what we’re doing here. This rule is added before the one we did above, like so:

RewriteRule ^products/([0-9][0-9])$ /products/$1/ [R]
RewriteRule ^products/([0-9][0-9])/$ /productinfo.php?prodID=$1

Thus, if the user types in the URL products/12, our first rule kicks in, rewriting the URL to include the trailing slash, and doing a new request for products/12/ so the user can see that we likes our trailing slashes around here. Then the second rule has something to match, and transparently redirects this URL to productinfo.php?prodID=12. Slick.

Match Modifiers

You can expand your regular expression patterns by adding some modifier characters, which allow you to match URLs with an indefinite number of characters. In our examples above, we were only allowing two numbers after products. This isn’t the most expandable solution, as if the shop ever grew beyond these initial confines of 99 products and created the URL productinfo.php?prodID=100, our rules would cease to match this URL.

So, instead of hard-coding a set number of digits to look for, we’ll work in some room to grow by allowing any number of characters to be entered. The rule below does just that:

RewriteRule ^products/([0-9]+)$ /products/$1/ [R]

Note the plus sign (+) that has snuck in there. This modifier changes whatever comes directly before it, by saying ‘one or more of the preceding character or range.’ In this case it means that the rule will match any URL that starts with products/ and ends with at least one digit. So this’ll match both products/1 and products/1000.

Other match modifiers that can be used in the same way are the asterisk, *, which means ‘zero or more of the preceding character or range’, and the question mark, ?, which means ‘zero or only one of the preceding character or range.’

Adding Guessable URLs

Using these simple commands you can set up a slew of ‘shortcut URLs’ that you think visitors will likely try to enter to get to pages they know exist on your site. For example, I’d imagine a lot of visitors try jumping straight into our stylesheets section by typing the URL http://www.yourhtmlsource.com/css/. We can catch these cases, and hopefully alert the reader to the correct address by updating their location bar once the redirect is done with these lines:

RewriteRule ^css(/)?$ /stylesheets/ [R]

The simple regular expression in this rule allows it to match the css URL with or without a trailing slash. The question mark means ‘zero or one of the preceding character or range’ — in other words either yourhtmlsource.com/css or yourhtmlsource.com/css/ will both be taken care of by this one rule.

This approach means less confusing 404 errors for your readers, and a site that seems to run a whole lot smoother all ’round.

 

Source: http://roshanbh.com.np/2008/03/url-rewriting-examples-htaccess.html

 

If you are looking for the examples of URL rewriting then this post might be useful for you. In this post, I’ve given five useful examples of URL rewriting using .htacess. If you don’t know something about url rewriting then please check my older post about url rewriting using .htaccess.

Now let’s look at the examples

1)Rewriting product.php?id=12 to product-12.html

It is a simple redirection in which .php extension is hidden from the browser’s address bar and dynamic url (containing “?” character) is converted into a static URL.

RewriteEngine on
RewriteRule ^product-([0-9]+)\.html$ product.php?id=$1

2) Rewriting product.php?id=12 to product/ipod-nano/12.html

SEO expert always suggest to display the main keyword in the URL. In the following URL rewriting technique you can display the name of the product in URL.

RewriteEngine on
RewriteRule ^product/([a-zA-Z0-9_-]+)/([0-9]+)\.html$ product.php?id=$2

3) Redirecting non www URL to www URL

If you type yahoo.com in browser it will be redirected to www.yahoo.com. If you want to do same with your website then put the following code to .htaccess file. What is benefit of this kind of redirection?? Please check the post about SEO friendly redirect (301) redirect in php and .htaccess.

RewriteEngine On
RewriteCond %{HTTP_HOST} ^optimaxwebsolutions\.com$
RewriteRule (.*) http://www.optimaxwebsolutions.com/$1 [R=301,L]

4) Rewriting yoursite.com/user.php?username=xyz to yoursite.com/xyz

Have you checked zorpia.com.If you type http://zorpia.com/roshanbh233 in browser you can see my profile over there. If you want to do the same kind of redirection i.e http://yoursite.com/xyz to http://yoursite.com/user.php?username=xyz then you can add the following code to the .htaccess file.

RewriteEngine On
RewriteRule ^([a-zA-Z0-9_-]+)$ user.php?username=$1
RewriteRule ^([a-zA-Z0-9_-]+)/$ user.php?username=$1

5) Redirecting the domain to a new subfolder of inside public_html.

Suppose the you’ve redeveloped your site and all the new development reside inside the “new” folder of inside root folder.Then the new development of the website can be accessed like “test.com/new”. Now moving these files to the root folder can be a hectic process so you can create the following code inside the .htaccess file and place it under the root folder of the website. In result, www.test.com point out to the files inside “new” folder.

RewriteEngine On
RewriteCond %{HTTP_HOST} ^test\.com$ [OR]
RewriteCond %{HTTP_HOST} ^www\.test\.com$
RewriteCond %{REQUEST_URI} !^/new/
RewriteRule (.*) /new/$1

 

 

.

 

Source: http://roshanbh.com.np/2008/02/hide-php-url-rewriting-htaccess.html

 

Last time I’ve written an article about hiding php file extension where I’ve showed you how you can use .html or .asp extension of file instead of .php extension. But there was one flaw in that technique you have had to change the file extension explicitly but in this post I’m going to show you how to rewrite the URL instead of renaming the file extension Using this technique you will see product.html in the address bar of the browser but the actual file name remains product.php and you don’t need to rename the file extension. Furthermore you can rewrite the URL like product.php?id=5 to product-5.html.

what is the benefits of rewriting URL?

When a search engine visits the dynamic url like product.php?id=5 it does not give much importance to that URL as search engine sees “?” sign treat it as a url which keeps on changing. so we’re converting the dynamic URL like the product.php?id=5 to static url format like product-5.html. We’ll rewrite the url in such a way that in browser’s address bar it will display as a product-5.html but it actually calls the file product.php?id=5. So that why these kind of URL also named as SEO friendly URL.

what is required for URL rewriting ??

To rewrite the URL you must have the mod_rewrite module must be loaded in apache server. And furthermore, FollowSymLinks options also need to be enabled otherwise you may encounter 500 Internal Sever Error. If you don’t know much about mod_rewrite module then please check this post to know how to check and enable mod_rewrite module in apache?

Examples of url rewriting for seo friendly URL

For rewriting the URL, you should create a .htaccess file in the root folder of your web directory. And have to put the following codes as your requirement.

Options +FollowSymlinks
RewriteEngine on
RewriteRule ^(.*)\.htm$ $1.php [nc]

The following example will rewrite the test.php to test.html i.e when a URL like http://localhost/test.htm is called in address bar it calls the file test.php. As you can see the regular expression in first part of the RewriteRule command and $1 represents the first regular expression of the part of the RewriteRule and [nc] means not case sensitive.

Options +FollowSymlinks
RewriteEngine on
RewriteRule ^product-([0-9]+)\.html$ products.php?id=$1

The following example will rewrite the product.php?id=5 to porduct-5.html i.e when a URL like http://localhost/product-5.html calls product.php?id=5 automatically.

• twist ((shell_command) Replace the current process by an instance of the specified
  shell command, after performing the % expansions described in the
  hosts_access(5) manual page.)
• deny (Denys Service/Access)
• banners ((/some/directory) Look for a file in "/some/directory" with the same
  name as the daemon process (for example in.telnetd for the telnet
  service), and copy its contents to the client. Newline characters are replaced
  by carriage-return newline, and % sequences are expanded (see the
  hosts_access(5) manual page).)
• nice ([number] Change the nice value of the process (default 10). Specify a positive
  value to spend more CPU resources on other processes.)
• rfc931 ([timeout_in_seconds] Look up the client user name with the RFC 931 (TAP, IDENT,
  RFC 1413) protocol. This option is silently ignored in case of services based
  on transports other than TCP. It requires that the client system runs an RFC
  931 (IDENT, etc.) -compliant daemon, and may cause noticeable delays with
  connections from non-UNIX clients. The timeout period is optional. If no
  timeout is specified a compile-time defined default value is taken.)
• setenv ((name) (value) Place a (name, value) pair into the process environment. The
  value is subjected to % expansions and may contain whitespace (but
  leading and trailing blanks are stripped off).)
• user ((user[.group]) Ammume the privleges of the user and group)
• allow (Permits Service/Access)
• umask ((umask) Like the umask command that is built into the shell. Should be octal)
• keepalive (Causes the server to periodically send a message to the client. The connection
  is considered broken when the client does not respond. The keepalive option can
  be useful when users turn off their machine while it is still connected to a
  server. The keepalive option is not useful for datagram (UDP) services.)

 

Source: http://webdesign.about.com/cs/apache/a/aainstallapache.htm

 

Before You Begin

Apache is one of the most popular Web servers on the Web right now, and part of its charm is that it’s free. It also has a lot of features that make it very extensible and useful for many different types of Web sites. It is a server that is used for personal Web pages up to enterprise level sites.

This article will discuss how to install Apache on a Linux system. Before we start you should be at least comfortable working in Linux – changing directories, using tar and gunzip, and compiling with make (I’ll discuss where to get binaries if you don’t want to mess with compiling your own). You should also have access to the root account on the server machine.

Download Apache

I recommend downloading the latest stable release. At the time of this writing, that was Apache 2.0. The best place to get Apache is from the Apache HTTP Server download site. Download the sources appropriate to your system. Binary releases are available as well.

Extract the Files

Once you’ve downloaded the files you need to uncompress them and untarring:
  gunzip -d httpd-2_0_NN.tar.gz
  tar xvf httpd-2_0_NN.tar
This creates a new directory under the current directory with the source files.

Configuring

Once you’ve got the files, you need to tell your machine where to find everything by configuring the source files. The easiest way is to accept all the defaults and just type:
  ./configure

Of course, most people don’t want to accept just the default choices. The most important option is the prefix= option. This specifies the directory where the Apache files will be installed. You can also set specific environment variables and modules. Some of the modules I like to have installed are:

• mod_alias – to map different parts of the URL tree
• mod_include – to parse Server Side Includes
• mod_mime – to associate file extensions with its MIME-type
• mod_rewrite – to rewrite URLs on the fly
• mod_speling (sic) – to help your readers who might misspell URLs
• mod_ssl – to allow for strong cryptography using SSL
• mod_userdir – to allow system users to have their own Web page directories

Please keep in mind that these aren’t all the modules I might install on a given system. Read the details about the modules to determine which ones you need.

Build

As with any source installation, you’ll then need to build the installation:
  make
  make install

Customize

Assuming that there were no problems, you are ready to customize your Apache configuration. This really just amounts to editing the httpd.conf file. This file is located in the PREFIX/conf directory. I generally edit it with vi:
  vi PREFIX/conf/httpd.conf
Note: you’ll need to be root to edit this file.

Follow the instructions in this file to edit your configuration the way you want it. More help is available on the Apache Web site.

Test Your Server

Open a Web browser on the same machine and type http://localhost/ in the address box. You should see a page similar to the one in the partial screen shot above. Specifically, it will say in big letters "Seeing this instead of the website you expected?" This is good news, as it means your server installed correctly.

Start Editing/Uploading Pages

Once your server is up and running you can start posting pages. Have fun building your Web site.

Web Design Path

Professional Web Developers

Suggested Reading

How are You Being Served?
Time to Switch from IIS
Become a Hosting Provider

More Apache Help

How to Install Apache on Windows 
More Apache Resources
Other Web Servers

 

Source: http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/custom-guide/ch-httpdconfig.html 

 

In Red Hat Linux 8.0, the Apache HTTP Server was updated to version 2.0, which uses different configuration options. Also starting with Red Hat Linux 8.0, the RPM package was renamed httpd. If you want to migrate an existing configuration file by hand, refer to the migration guide at /usr/share/doc/httpd-<ver>/migration.html or the Red Hat Linux Reference Guide for details.

If you configured the Apache HTTP Server with the HTTP Configuration Tool in previous versions of Red Hat Linux and then performed an upgrade, you can use the application to migrate the configuration file to the new format for version 2.0. Start the HTTP Configuration Tool, make any changes to the configuration, and save it. The configuration file saved will be compatible with version 2.0.

The HTTP Configuration Tool allows you to configure the /etc/httpd/conf/httpd.conf configuration file for the Apache HTTP Server. It does not use the old srm.conf or access.conf configuration files; leave them empty. Through the graphical interface, you can configure directives such as virtual hosts, logging attributes, and maximum number of connections.

Only modules that are shipped with Red Hat Linux can be configured with HTTP Configuration Tool. If additional modules are installed, they can not be configured using this tool.

The httpd and redhat-config-httpd RPM packages need to be installed to use the HTTP Configuration Tool. It also requires the X Window System and root access. To start the application, go to the Main Menu Button => System Settings => Server Settings => HTTP Server or type the command redhat-config-httpd at a shell prompt (for example, in an XTerm or GNOME Terminal).

Caution

Do not edit the /etc/httpd/conf/httpd.conf configuration file by hand if you wish to use this tool. The HTTP Configuration Tool generates this file after you save your changes and exit the program. If you want to add additional modules or configuration options that are not available in HTTP Configuration Tool, you cannot use this tool.

The general steps for configuring the Apache HTTP Server using the HTTP Configuration Tool are as following:

1. Configure the basic settings under the Main tab.

2. Click on the Virtual Hosts tab and configure the default settings.

3. Under the Virtual Hosts tab, configure the Default Virtual Host.

4. If you want to serve more than one URL or virtual host, add the additional virtual hosts.

5. Configure the server settings under the Server tab.

6. Configure the connections settings under the Performance Tuning tab.

7. Copy all necessary files to the DocumentRoot and cgi-bin directories.

8. Exit the application and select to save your settings.

Basic Settings

Use the Main tab to configure the basic server settings.

Figure Basic Settings

Enter a fully qualified domain name that you have the right to use in the Server Name text area. This option corresponds to the ServerName directive in httpd.conf. The ServerName directive sets the hostname of the Web server. It is used when creating redirection URLs. If you do not define a server name, the Web server attempts to resolve it from the IP address of the system. The server name does not have to be the domain name resolved from the IP address of the server. For example, you might want to set the server name to www.example.com when your server’s real DNS name is actually foo.example.com.

Enter the email address of the person who maintains the Web server in the Webmaster email address text area. This option corresponds to the ServerAdmin directive in httpd.conf. If you configure the server’s error pages to contain an email address, this email address will be used so that users can report a problem by sending email to the server’s administrator. The default value is root@localhost.

Use the Available Addresses area to define the ports on which the server will accept incoming requests. This option corresponds to the Listen directive in httpd.conf. By default, Red Hat configures the Apache HTTP Server to listen to port 80 for non-secure Web communications.

Click the Add button to define additional ports on which to accept requests. A window as shown in Figure 19-2 will appear. Either choose the Listen to all addresses option to listen to all IP addresses on the defined port or specify a particular IP address over which the server will accept connections in the Address field. Only specify one IP address per port number. If you want to specify more than one IP address with the same port number, create an entry for each IP address. If at all possible, use an IP address instead of a domain name to prevent a DNS lookup failure. Refer to http://httpd.apache.org/docs-2.0/dns-caveats.html for more information about Issues Regarding DNS and Apache.

Entering an asterisk (*) in the Address field is the same as choosing Listen to all addresses. Clicking the Edit button in the Available Addresses frame shows the same window as the Add button except with the fields populated for the selected entry. To delete an entry, select it and click the Delete button.

Tip

If you set the server to listen to a port under 1024, you must be root to start it. For port 1024 and above, httpd can be started as a regular user.

Default Settings

After defining the Server Name, Webmaster email address, and Available Addresses, click the Virtual Hosts tab and click the Edit Default Settings button. The window shown in Figure 19-3 will appear. Configure the default settings for your Web server in this window. If you add a virtual host, the settings you configure for the virtual host take precedence for that virtual host. For a directive not defined within the virtual host settings, the default value is used.

Site Configuration

The default values for the Directory Page Search List and Error Pages will work for most servers. If you are unsure of these settings, do not modify them.

Figure Site Configuration

The entries listed in the Directory Page Search List define the DirectoryIndex directive. The DirectoryIndex is the default page served by the server when a user requests an index of a directory by specifying a forward slash (/) at the end of the directory name.

For example, when a user requests the page http://www.example.com/this_directory/, they are going to get either the DirectoryIndex page if it exists, or a server-generated directory list. The server will try to find one of the files listed in the DirectoryIndex directive and will return the first one it finds. If it does not find any of these files and if Options Indexes is set for that directory, the server will generate and return a list, in HTML format, of the subdirectories and files in the directory.

Use the Error Code section to configure Apache HTTP Server to redirect the client to a local or external URL in the event of a problem or error. This option corresponds to the ErrorDocument directive. If a problem or error occurs when a client tries to connect to the Apache HTTP Server, the default action is to display the short error message shown in the Error Code column. To override this default configuration, select the error code and click the Edit button. Choose Default to display the default short error message. Choose URL to redirect the client to an external URL and enter a complete URL including the http:// in the Location field. Choose File to redirect the client to an internal URL and enter a file location under the document root for the Web server. The location must begin the a slash (/) and be relative to the Document Root.

For example, to redirect a 404 Not Found error code to a webpage that you created in a file called 404.html, copy 404.html to DocumentRoot/../error/404.html. In this case, DocumentRoot is the Document Root directory that you have defined (the default is /var/www/html/). If the Document Root is left as the default location, the file should be copied to /var/www/error/404.html. Then, choose File as the Behavior for 404 – Not Found error code and enter /error/404.html as the Location.

From the Default Error Page Footer menu, you can choose one of the following options:

• Show footer with email address — Display the default footer at the bottom of all error pages along with the email address of the website maintainer specified by the ServerAdmin directive. Refer to Section 19.3.1.1 General Options for information about configuring the ServerAdmin directive.

• Show footer — Display just the default footer at the bottom of error pages.

• No footer — Do not display a footer at the bottom of error pages.

Logging

By default, the server writes the transfer log to the file /var/log/httpd/access_log and the error log to the /var/log/httpd/error_log file.

The transfer log contains a list of all attempts to access the Web server. It records the IP address of the client that is attempting to connect, the date and time of the attempt, and the file on the Web server that it is trying to retrieve. Enter the name of the path and file in which to store this information. If the path and filename does not start with a slash (/), the path is relative to the server root directory as configured. This option corresponds to the TransferLog directive.

Figure Logging

You can configure a custom log format by checking Use custom logging facilities and entering a custom log string in the Custom Log String field. This configures the LogFormat directive. Refer to http://httpd.apache.org/docs-2.0/mod/mod_log_config.html#formats for details on the format of this directive.

The error log contains a list of any server errors that occur. Enter the name of the path and file in which to store this information. If the path and filename does not start with a slash (/), the path is relative to the server root directory as configured. This option corresponds to the ErrorLog directive.

Use the Log Level menu to set how verbose the error messages in the error logs will be. It can be set (from least verbose to most verbose) to emerg, alert, crit, error, warn, notice, info or debug. This option corresponds to the LogLevel directive.

The value chosen with the Reverse DNS Lookup menu defines the HostnameLookups directive. Choosing No Reverse Lookup sets the value to off. Choosing Reverse Lookup sets the value to on. Choosing Double Reverse Lookup sets the value to double.

If you choose Reverse Lookup, your server will automatically resolve the IP address for each connection which requests a document from your Web server. Resolving the IP address means that your server will make one or more connections to the DNS in order to find out the hostname that corresponds to a particular IP address.

If you choose Double Reverse Lookup, your server will perform a double-reverse DNS. In other words, after a reverse lookup is performed, a forward lookup is performed on the result. At least one of the IP addresses in the forward lookup must match the address from the first reverse lookup.

Generally, you should leave this option set to No Reverse Lookup, because the DNS requests add a load to your server and may slow it down. If your server is busy, the effects of trying to perform these reverse lookups or double reverse lookups may be quite noticeable.

Reverse lookups and double reverse lookups are also an issue for the Internet as a whole. All of the individual connections made to look up each hostname add up. Therefore, for your own Web server’s benefit, as well as for the Internet’s benefit, you should leave this option set to No Reverse Lookup.

Environment Variables

Sometimes it is necessary to modify environment variables for CGI scripts or server-side include (SSI) pages. The Apache HTTP Server can use the mod_env module to configure the environment variables which are passed to CGI scripts and SSI pages. Use the Environment Variables page to configure the directives for this module.

Figure Environment Variables

Use the Set for CGI Scripts section to set an environment variable that is passed to CGI scripts and SSI pages. For example, to set the environment variable MAXNUM to 50, click the Add button inside the Set for CGI Script section as shown in Figure 19-5 and type MAXNUM in the Environment Variable text field and 50 in the Value to set text field. Click OK to add it to the list. The Set for CGI Scripts section configures the SetEnv directive.

Use the Pass to CGI Scripts section to pass the value of an environment variable when the server was first started to CGI scripts. To see this environment variable, type the command env at a shell prompt. Click the Add button inside the Pass to CGI Scripts section and enter the name of the environment variable in the resulting dialog box. Click OK to add it to the list. The Pass to CGI Scripts section configures the PassEnv directive.

If you want to remove an environment variable so that the value is not passed to CGI scripts and SSI pages, use the Unset for CGI Scripts section. Click Add in the Unset for CGI Scripts section, and enter the name of the environment variable to unset. Click OK to add it to the list. This corresponds to the UnsetEnv directive.

To edit any of these environment values, select it from the list and click the corresponding Edit button. To delete any entry from the list, select it and click the cooresponding Delete button.

To learn more about environment variables in Apache HTTP Server, refer to the following:

http://httpd.apache.org/docs-2.0/env.html

Directories

Use the Directories page to configure options for specific directories. This corresponds to the <Directory> directive.

Figure Directories

Click the Edit button in the top right-hand corner to configure the Default Directory Options for all directories that are not specified in the Directory list below it. The options that you choose are listed as the Options directive within the <Directory> directive. You can configure the following options:

• ExecCGI — Allow execution of CGI scripts. CGI scripts are not executed if this option is not chosen.

• FollowSymLinks — Allow symbolic links to be followed.

• Includes — Allow server-side includes.

• IncludesNOEXEC — Allow server-side includes, but disable the #exec and #include commands in CGI scripts.

• Indexes — Display a formatted list of the directory’s contents, if no DirectoryIndex (such as index.html) exists in the requested directory.

• Multiview — Support content-negotiated multiviews; this option is disabled by default.

• SymLinksIfOwnerMatch — Only follow symbolic links if the target file or directory has the same owner as the link.

To specify options for specific directories, click the Add button beside the Directory list box. The window shown in Figure 19-7 appears. Enter the directory to configure in the Directory text field at the bottom of the window. Select the options in the right-hand list, and configure the Order directive with the left-hand side options. The Order directive controls the order in which allow and deny directives are evaluated. In the Allow hosts from and Deny hosts from text field, you can specify one of the following:

• Allow all hosts — Type all to allow access to all hosts.

• Partial domain name — Allow all hosts whose names match or end with the specified string.

• Full IP address — Allow access to a specific IP address.

• A subnet — Such as 192.168.1.0/255.255.255.0

• A network CIDR specification — such as 10.3.0.0/16

Figure Directory Settings

If you check the Let .htaccess files override directory options, the configuration directives in the .htaccess file take precedence.

 

Source: http://www.hypexr.org/linux_mail_server.php 

 

Creating a Linux Mail Server

(Postfix, Procmail, Fetchmail, SpamBayes, Courier-imap, Mutt, SquirrelMail)

What this Document is About

Software Used

Installing the Software

• Installing Postfix
• Installing Fetchmail
• Installing Procmail
• Installing SpamBayes
• Installing Mutt
• Installing courier-imap courier-authlib
• Installing SquirrelMail
• Installing pop-before-smtp

Postfix Configuration

• main.cf

Fetchmail Configuration

Spambayes and Procmail Configuration

• Spam System Initial Setup
• Procmail Configuration for Spambayes

Mutt Configuration

• .muttrc Configuration
• New Mutt Commands

SquirrelMail Configuration

Courier Authentication

Pop-before-smtp Configuration

Additional Simple Procmail Recipes and Mailbox Config

Conclusion

Resources

• Example Files
• Links

Contributions

---

What this Document is About

This document covers how I have set up my Linux box to work as a Mail Server. The details will be specific to Arch Linux, but it will be be general enough for any *nix distribution if you have a knowledge of installing software packages, finding the location of configuration files, and starting/stoping services..

My goals in creating this mail system are to have all of my mail, from various email accounts, stored locally on my machine so I can access it anytime via SSH, the web, or imap. I wanted all of my mail delivered to my local user account and I wanted the spam to be sucked out and launched into space.

The backbone of the mail system is the Mail Transport Agent (MTA). It will handle receiving mail addressed directly to your machine, sending it to the correct users mail box, and sending mail out from the machine. I determined Postfix to be the best choice for my MTA and after some reading found Procmail for sorting mail, SpamBayes for detecting spam, and Fetchmail for grabbing mail from my pop mailboxes.

Bellow is a flow diagram for mail moving through the system:

The SpamAssassin and sa-learn can be ignored. I will describe how to use it with SpamBayes since I was not happy with Spam Assassin’s performance and had made the diagram when I was still using it.

Software Used

• Arch Linux
• Postfix 2.1.5
  Secure, fast, easy to administer drop in replacement for Sendmail. (MTA) Provides smtp-server.
  (Requires sasl installed and support for sasl compiled into Postfix if sending mail through an intermediary as you may need to do if you have a domain name pointing to a dynamic IP address.)
• Cyrus SASL 2.1.21
  SASL authentication daemon.
• Procmail 3.22
  Highly configurable auto mail processing.
• Fetchmail 6.25
  A remote-mail retrieval utility.
• SpamBayes 1.1a1
  a Bayesian anti-spam filter, initially based on the work of Paul Graham. The major difference between this and other, similar projects is the emphasis on testing newer approaches to scoring messages.
• Mutt 1.4.2.1
  A small but very powerful text-based mail client
• Courier-imap 4.0.4
  IMAP / POP3 Server
• Courier-authlib 0.57
  Authentication library for the courier mailserver.
• SquirrelMail cvs 1.5.1
  For checking mail via the web.
  (requires imap-server, perl, and php)
• Pop-before-smtp 1.38
  A simple daemon written in Perl, to allow email relay control based on successful POP or IMAP logins.

Installing the Software

Installing Postfix

A note for those with dynamic IP addresses: Before installing Postfix we must first consider if it will need to make use of Simple Authentication and Security Layer (SASL). If you have a dynamic IP address and are using a service like dyndns.org you will need to have Postfix send your mail through your ISP’s mail server (with authentication) making use of SASL for the connection. This is because many domains that you will send email to will recognize your hostname as pointing to a dynamic IP address and send the mail back. Check if your distributions Postfix package was built with SASL support or if you are compiling Postfix from scratch add SASL with a ./configure option.

If you’ve determined you do not need SASL support in postfix:

$ pacman -Sy postfix

If you do need SASL support we need to install cyrus-sasl and use the Arch Build System (abs) utility for building Postfix. Follow the instructions bellow or follow these Arch Wiki instructions: http://wiki.archlinux.org/index.php/PostFix_Howto_With_SASL

$ pacman -Sy cyrus-sasl
$ abs
$ cd /var/abs
$ mkdir -p local/postfix
$ cp -v extra/daemons/postfix/* local/postfix

Now edit /var/abs/local/postfix/PKGBUILD. Under the build() section change:

make OPT="${CFLAGS}" || return 1

to:

make OPT="${CFLAGS}" \
CCARGS="-DUSE_SASL_AUTH -I/usr/include/sasl/" \
AUXLIBS="-L/usr/lib/sasl2 -lsasl2" || return 1

Now run:

$ makepkg

$ pacman -A postfix-*.pkg.tar.gz

I like to then copy the package into a directory under /var/lib/pacman so I have access to all of my abs built packages later:

$ mkdir /var/lib/pacman/abs_built

$ cp postfix-*.pkg.tar.gz /var/lib/pacman/abs_built

Installing Fetchmail

$ pacman -S fetchmail

Installing Procmail

$ pacman -S procmail

Installing SpamBayes

SpamBayes is not yet in the pacman repository. Download it, extract, and install (substitute version numbers with most current):

$ wget http://dl.sourceforge.net/sourceforge\
/spambayes/spambayes-1.1a1.tar.gz
$ tar zxvf spambayes-1.1a1.tar.gz
$ cd spambayes-1.1a1
$ python setup.py install

Installing Mutt

$ pacman -S mutt

Installing courier-imap and courier-authlib

SquirrelMail as well as other mail clients (evolution, outlook, cell phone, etc.) will use this to connect to the mail box. Courier-imap also depends on courier-authlib, so this command will install that as well.

$ pacman -S courier-imap

Installing SquirrelMail

$ pacman -S squirrelmail

Installing pop-before-smtp

You are going to want to consider using Postfix/TLS or pop-before-smtp if you are interested in users connected via IMAP being able to send mail (using your mail server as a SMTP relay). I recommend using Postfix/TLS but will not cover how to install/configure it in this document.

Pop-before-smtp is not in pacman repositories. Substitute version numbers with the most current.

$ wget http://dl.sourceforge.net/sourceforge/popbsmtp/pop-before-smtp-1.38.tar.gz

$ tar zxvf pop-before-smtp-*.tar.gz

$ cd pop-before-smtp-*

Read the README in this directory and then read README.QUICKSTART in the contrib directory for where to copy the necessary files to.

Postfix Configuration

main.cf

Now change to the ‘/etc/postfix’ directory and open up main.cf for editing.

The domain name for my machine is through dyndns.org. Set your myhostname and my domain to something like this:

myhostname = example.com mydomain = example.com

Next we are going to want to set Postfix up for using qmail-style delivery with Maildir (where each email is stored in its own file). This is for courier-imap which SquirrelMail will be using to access the mail and will have other advantages. Don’t forget the ending forward slash.

home_mailbox = Maildir/

Set it up so that incoming mail is sent through procmail and so we can alias names/users to mailboxes.

mailbox_command = /usr/bin/procmail -a "$DOMAIN"

alias_maps = hash:/etc/postfix/aliases

If you have a dynamic IP or have trouble with mail being returned from large ISP’s or mail systems you are going to have to take a few extra steps. We will set it up so that mail will go through a relay host. In this example I will use my ISP’s (SBC) smtp server as an example. Make these additional changes to main.cf

relayhost = [smtp.sbcglobal.yahoo.com]

# authentication for sbc yahoo

smtp_sasl_auth_enable = yes

smtp_sasl_security_options = noanonymous

smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

That wraps up our changes to main.cf. We just need to create the hash aliases databases that we defined above. Add these lines to the top of /etc/postfix/aliases (replacing hypexr with your user account of course):

root: hypexr

spam: hypexr
ham:  hypexr

Generate the database. From /etc/postfix dir:

$ postalias aliases

Set up smtp server and authentication for SBC mail relay. sasl_passwd:

smtp.sbc.mail.yahoo4.akadns.net    my_username@sbcglobal.net:my_passw

Note: Server name can change from time to time. There is a CNAME that points to this name but CNAMEs do not work here.

Generate the database:

$ ./postmap sasl_passwd

Start the postfix server and we are done with postfix!!!!

$ /etc/rc.d/postfix start

Having problems getting it to start? Check out the Postfix log file usually in /var/log/mail or /var/log/maillog.

Fetchmail Configuration

We are going to run our fetchmail daemon from user root. So create the file /root/.fetchmailrc:

set postmaster "postmaster"

set bouncemail

set no spambounce

set properties ""

set daemon 60

poll pop.example.com with proto POP3

      user ‘user_name’ there with password ‘my_password’ is ‘local_user_to_deliver_to’ here options flush

poll pop.gmail.com with proto POP3 port 995

      user ‘my_gmail_un@gmail.com’ there with ssl with password ‘my_password’ is ‘hypexr’ here options keep

This tells fetchmail to check every minute for mail at pop.mypop.com and also at my gmail account. The flush option at the end of the mypop entry tells it to remove the mail after it is downloaded. I do not want the mail removed from my Gmail account so I use the keep option. Google also uses ssl, hence the "with ssl". Fetchmail comes with its configuration utility fetchmailconf which makes setting up your .fetchmailrc file easy.

I don’t believe that Arch Linux has a rc script for fetchmail. The one that I made (/etc/rc.d/fetchmail) looks like this (stat_busy, stat_fail, stat_done are specific to Arch Linux. You could take these out):

#!/bin/sh

. /etc/rc.conf
. /etc/rc.d/functions

DAEMON_NAME="fetchmail"
DAEMON_CONF="/root/.fetchmailrc"
DAEMON_PATH="/usr/bin/fetchmail"

PID=`pidof -o %PPID ${DAEMON_NAME}`

case "$1" in
start)
   stat_busy "Starting ${DAEMON_NAME}"
   [ -z "$PID" ] && ${DAEMON_PATH} -f ${DAEMON_CONF} -d 60 &> /dev/null
   if [ $? -gt 0 ]; then
      stat_fail
   else
      add_daemon ${DAEMON_NAME}
      stat_done
   fi
   ;;
stop)
   stat_busy "Stopping ${DAEMON_NAME}"
   [ ! -z "$PID" ]  && kill "$PID" &> /dev/null
   if [ $? -gt 0 ]; then
      stat_fail
   else
      rm_daemon ${DAEMON_NAME}
      stat_done
   fi
   ;;
restart)
   # calling 'stop' and 'start' without the $0 fails...
   $0 stop
   sleep 3
   $0 start
   ;;
*)
   echo "usage: $0 {start|stop|restart}"
esac
exit 0

And start the fetchmail daemon:

$ /etc/rc.d/fetchmail start

Or if you don’t want to create/modify an rc script you can start the fetchmail daemon like:

$ /usr/bin/fetchmail -d 60

Spambayes and Procmail Configuration

We are going to be using Spambayes to train and determine if mail is spam and Procmail to sort the mail into folders when they enter our mail system. These two go together as far as Procmail is going to use Spambayes to help sort the mail.

I had used SpamAssassin for a year and had not been very happy with its performance. I have only been using Spambayes for a few weeks and have been pleasantly surprised by how well it has worked and how quickly it adapts. In fairness I had been training SpamAssassin on all of my spam, the spam it had caught and the spam it had missed as well. It was also an older version and I was not using all of the features. With that being said if you are curious give it a try. Many other excellent programs exist. A quick search on Google will turn them up.

When Spambayes is installed it is going to move several files that start with sb_ into your /usr/bin directory. We are going to be interested in sb_mboxtrain.py and sb_filter.py. The former trains the spam filter and the later classifies incoming spam. There are two methods I tried for the initial training:

1. Since I already had thousand of spam and ham (non spam) messages saved up I first tried initially training on this older mail. The results were okay but it was miss sorting mail. These problems would have sorted them selves out with the continued training of the misclassified mail but it would have also took a little while because the database had already been trained so much.
2. The second method I used was extremely effective. I started out with no training on initial messages. I simply let Spambayes work with no information. I then sent the spam and ham messages that ended up in the wrong mailbox to a designated folder. I then run the Spambayes training program on these miss classified messages.

The results from the second method have been amazing. Without even having to manually classify very many messages Spambayes has been performing at superior levels. Check out Spambayes Wiki for many other training methods. I will discuss easy methods of sorting miss classifications in the Mutt and Squirrel Mail sections.

Spam System Initial Setup

We are going to want Maildir folders to hold our regular mail, mail classified as spam, mail classified as unsure, missed spam, and missed ham.

If the standard Maildir directory (~/Maildir) doesn’t already exist lets create it with the necessary sub directories to make it a mail box. Run these commands from your user account who will be receiving the mail.

$ mkdir ~/Maildir

$ mkdir ~/Maildir/{new,cur,tmp}

And now the other 4 directories that we are going to use for dealing with spam (Making the directories hidden is going to be done for integration with Squirrelmail):

$ mkdir ~/Maildir/{.MissedSpam,.MissedHam,.CaughtSpam,.Unsure}

$ mkdir ~/Maildir/.MissedSpam/{new,cur,tmp}

$ mkdir ~/Maildir/.MissedHam/{new,cur,tmp}

$ mkdir ~/Maildir/.CaughtSpam/{new,cur,tmp}

$ mkdir ~/Maildir/.Unsure/{new,cur,tmp}

The messages that we are going to be training as spam are going to be in ~/Maildir/.MissedSpam and ham will be in ~/Maildir/.MissedHam. Lets create the script and cron job that will take care of training the Spambayes db. Create a file ~/Maildir/train_spambayes and add this:

#!/bin/sh
# Script to copy mail missed spam and ham into correct folders
# and run sb_mboxtrain.py to train spambayes
# Training will be done only on missed spam and ham

# Files we saved or bounced may be in the new directory
# We want them in cur for training
mv /home/hypexr/Maildir/.MissedSpam/new/* \
	/home/hypexr/Maildir/.MissedSpam/cur 2>/dev/null
mv /home/hypexr/Maildir/.MissedHam/new/* \
	/home/hypexr/Maildir/.MissedHam/cur 2>/dev/null
/usr/bin/sb_mboxtrain.py -d /home/hypexr/.hammie.db \
	-g /home/hypexr/Maildir/.MissedHam \
	-s /home/hypexr/Maildir/.MissedSpam

Make the script executable:

$ chmod +x ~/Maildir/train_spambayes

Set up our database that will be consulted for new email and trained on new spam/ham:

$ /usr/bin/sb_filter.py -d $HOME/.hammie.db -n

You will want cron installed so that it can launch the script above, which trains Spambayes, every night at 3:21 am. Edit your cron jobs with the command crontab -e and add:

21 3 * * * ~/Maildir/train_spambayes

Procmail Configuration for Spambayes

These are basic procmail recipes that will sort the spam from ham. From Spambayes on Unix or Linux. Add these lines to your mail recipient’s ~/.procmailrc:

SHELL=/bin/sh
MAILDIR=$HOME/Maildir
DEFAULT=$HOME/Maildir/
CAUGHT_SPAM=$MAILDIR/CaughtSpam/
UNSURE=$MAILDIR/Unsure/

#Spambayes process
:0fw:hamlock
| /usr/bin/sb_filter.py -d /home/hypexr/.hammie.db

:0
* ^X-Spambayes-Classification: spam
${CAUGHT_SPAM}

:0
* ^X-Spambayes-Classification: unsure
${UNSURE}

# Catches everything else.
# Anything left over goes into the DEFAULT folder
:0:
${DEFAULT}

So incoming mail is not filtered through procmail’s rules where Spambayes’s sb_filter decides whether it is spam or ham. The mail is now sitting in the correct directories and ready to be accessed by whatever reader/front-end you would like. Next I going to describe setting up Mutt and SquirrelMail for viewing your mail. Mutt will be the most powerful option combining speed and sleekness but SquirrelMail will be able to match functionality.

Mutt Configuration

First we need to set the MAIL variable so that Mutt will know where to look for your mail. In your ~/.bashrc (create if does not exist) add the line:

MAIL=’~/Maildir’

Exit out of your shell and log back in so that the variable will be set or run export MAIL=’~/Maildir’. On some systems mutt will not use $MAIL to determine the location of your mail box. When trying to start mutt if it reports a message that no mail directory exists you can try setting the MAILDIR variable or just create the directory that it wants to use and make it a valid Maildir: mkdir -p ~/.maildir/{cur,new,tmp} where .maildir is the folder that mutt wants to use. If you use a directory other than ~/Maildir remember to substitute that wherever I reference Maildir in this document.

Now for the fun part. Lets make everything look pretty and integrated in Mutt. If you start up Mutt right now you will see any mail that your system has received since you have completed the steps above. If no mail shows up in Mutt and there should be mail you need to first check if Spambayes has filtered it into your .Unsure or .CaughtSpam directories. Type ‘c‘ to change directories and enter ~/Maildir/Unsure and check out the mail thats been classified as spam: ‘c‘ ~/Maildir/CaughtSpam. If you still have not found the mail that you are expecting to have received make sure that you have configured Postfix, Fetchmail, and Procmail correctly. Also, try sending mail directly to your_user_accunt@example.com and see if it shows up in one of the directories above.

Now mail is flowing into your system and we can check how Spambayes has classified it by checking the different mail folders. The miss classified mail just needs to be moved into the correct folders for the training.

To manually move the misclassified spam mail:

1. From you Maildir inbox tag each message that has arrived that is spam. Do this by pressing ‘t‘ and an asterisk will appear by the messages. After all of the messages have been tagged, press ‘; m‘ and enter ~/Maildir/MissedSpam and the spam messages in your inbox will be moved to the MissedSpam mailbox for training by spambayes.
2. When looking at the mail that it has classified as spam in ‘~/Maildir/CaughtSpam’, tag all of the messages it considered spam that were ham and press ‘; C‘ and enter ~/Maildir/MissedHam. Now tag them again and press ‘; m‘ and enter ~/Maildir to move them into your inbox, where they should have been placed in the first place.

Of course all of this manual stuff is a huge pain so lets automate all of it in .muttrc.

.muttrc Configuration

This .muttrc file is going to save you a quazillion.. thats right a quiazalliona hours in productivity in dealing with our current setup. Here it is:

# Emacs is my editor of choice.  I have mostly read that people like
#  using emacs-client here and using an emacs server so that the
#  editing opens up in an existing emacs session but I prefer
#  loading up a fresh instance. The default editor is some 'vi'
set editor="emacs"

# Setting these macros is going to save us all of that time I was
#  talking about.  Now instead of having to type all of those
#  characters to move the mail around we will can just tag the messages
#  and, for the instance of spam in your inbox, hit 'S'.

# Move mail to correct directories
macro   index   S       ";s~/Maildir/.MissedSpam\r\r$\r"
macro   pager   S       ";s~/Maildir/.MissedSpam\r\r$\r"
macro   index   H       ";C~/Maildir/.MissedHam\r\r;s~/Maildir\r\r$\r"
macro   pager   H       ";C~/Maildir/.MissedHam\r\r;s~/Maildir\r\r$\r"
macro   index   M       ";s~/Maildir/\r"
macro   pager   M       ";s~/Maildir/\r"

# This one lets me quickly move job search related emails to the
#  correct directory.  Want to give me a job? :)
macro   index   B       ";s~/Maildir/.Job\r\r$\r"
macro   pager   B       ";s~/Maildir/.Job\r\r$\r

# This is how we are going to move around to the different
#  mailboxes.  Hitting 'alt-1' will take us to our inbox
#  'alt-2' and we are looking at the mail Spambayes classified
#  as spam.  etc.
#  The extra mailboxes I have in here are for mailing lists I am on.
#  I will show the procmail recipe that automatically puts
#  the mailing list mail into its correct mailbox further down.

# Move to mailboxes quickly
macro   index   <Esc>1  "c~/Maildir\r"
macro   pager   <Esc>1  "c~/Maildir\r"
macro   index   <Esc>2  "c~/Maildir/.CaughtSpam\r"
macro   pager   <Esc>2  "c~/Maildir/.CaughtSpam\r"
macro   index   <Esc>3  "c~/Maildir/.Unsure\r"
macro   pager   <Esc>3  "c~/Maildir/.Unsure\r"
macro   index   <Esc>4  "c~/Maildir/.Tur\r"
macro   pager   <Esc>4  "c~/Maildir/.Tur\r"
macro   index   <Esc>5  "c~/Maildir/.Sbayes\r"
macro   pager   <Esc>5  "c~/Maildir/.Sbayes\r"
macro   index   <Esc>6  "c~/Maildir/.Wikka\r"
macro   pager   <Esc>6  "c~/Maildir/.Wikka\r"
# Training Maildirs for Spam and Ham
macro   index   <Esc>8  "c~/Maildir/.MissedSpam\r"
macro   pager   <Esc>8  "c~/Maildir/.MissedSpam\r"
macro   index   <Esc>9  "c~/Maildir/.MissedHam\r"
macro   pager   <Esc>9  "c~/Maildir/.MissedHam\r"
# Job E-mails
macro   index   <Esc>0  "c~/Maildir/.Job\r"
macro   pager   <Esc>0  "c~/Maildir/.Job\r"

# Setting these colors makes it easy to tell which emails
#  Spambayes has missed.
# Turn spam red and unsure green
color index red default "~h '^X-Spambayes-Classification: spam'"
color index green default "~h '^X-Spambayes-Classification: unsure'"
# Turn gmail mail brightblue
#  (My regular expression is not correct here.  I think that it
#   is looking anywhere in the header for my gmail address
#   instead of what I wanted..  Just in the 'To:'.
#   It still works for the most part)
color index brightblue default "~h '(^)*my_name@gmail.com'

# Lets set more colors to make things look beautiful.  Judging by
#  my title for this section I must have had bad feelings towards
#  colors when I added this.
#Color crap
color index brightwhite default ~N       # color for new messages
color status black yellow
color attachment brightyellow default    # file attachments
color search brightred default           # search matches
color quoted brightyellow default        # quoted text in replies
color quoted1 magenta default            # quoted text in replies
color body cyan default "((ftp|http|https)://|news:)[^ >)\"\t]+"   # URLs
color body cyan default "[-a-z_0-9.+]+@[-a-z_0-9.]+"               # email

# The rest of this stuff is pretty important for functionality and
#  readability.

# Don't ask to move read message
set move=no

# My signature is in this file
set signature="~/.signature"

# Change mail to look like from scott@hypexr.org
#  Stick your email address here.
my_hdr From: Scott Rippee <scott@hypexr.org>

# Header control   h displays header when in pager
# I want to only see the unignored by default
ignore *                                        # weed out all headers
unignored date from: to cc subject organization  # now show me these...

# Thanks to many people whom have written the documents and posts that I
#  have gotten these settings from.

New Mutt Commands

Here are the most useful (as far as dealing with spam) commands that we can now use in mutt:

t       Tag messages to be manipulated
S       Send message/s to spam training folder
H       Send message/s to ham training folder and to inbox
alt-1   Change mailbox to Maildir/ (inbox)
alt-2   Change mailbox to .CaughtSpam
alt-3   Change mailbox to .Unsure
alt-8   Change mailbox to .MissedSpam
alt-9   Change mailbox to .MissedHam

So for example, if a couple of spam messages show up in your inbox you can now tag them and hit ‘S‘ (remember its uppercase s) and have them vanished into .MissedSpam for training.

SquirrelMail Configuration

To configure SquirrelMail go into your html_base_directory/squirrelmail/config and run ./conf.pl. Edit the options to fit your needs. If you want to test your config cruise over to http://localhost/squirrelmail/src/configtest.php.

Now go to http://localhost/squirrelmail/ and login using your user account. If you have any email in your in box you should see it after you log in. Now click on the Folders option that is along the top, select all of the mailbox folders that we created earlier (they are listed here because we prefixed them with a period), and click on subscribe. Hit refresh and the mailboxes should be listed in the side panel. Now you can use SquirrelMail for almost all of you mail needs including tagging messages and moving them to the correct folders for Spambayes training and checking on the mail that Spambayes has classified as spam and unsure. The only functionality from our mutt setup that can’t be duplicated in SquirrelMail is copying missed ham into its training folder and then moving it into your inbox. This is due to the lack of copying ability. So you will have to go into the ham training folder to see some of the mail that should be in your inbox.

!!! IMPORTANT SECURITY NOTE!!! – Using SquirrelMail like this is not secure! Sending the plain text passwords that you use for your Unix account can be accessed by by a 3rd party, which would them access to your account. You should set up a secure server with SSL and have Courier-imap use passwords that differ from the users system password using Courier Authlib, which I will demonstrate in the next session. Excellent documents exist all over the net for setting up your web server with ssl.

Courier Authentication

Now is a good time to address a bit of essential security. As of now when a user connects via the SquirrelMail or IMAP interface the passwords are being sent in plain text and if these are sniffed that users system account is compromised. Without adding secure connections we can minimize the threat by using alternate passwords for imap connections. The snoop would then have access to your mail, but not be able to log in to the system.

Edit your authdaemonrc file. If your on Arch Linux this will be in /etc/authlib. Under authmodulelist I commented out:

authmodulelist="authpam authpwd authuserdb authshadow …"

And replaced it with:

authmodulelist="authuserdb"

I believe that adding authuserdb to the front of the list may work, but I am only using this method of authentication so it is all I need in my list.

Lets create the user account entry:

$ userdb "some_user_name" set home=/home/user_dir \
  mail=/home/user_dir/Maildir \
  uid=users_userid gid=users_groupid
$ userdbpw | userdb "some_user_name" set systempw

You will now be prompted for this pseudo users new password. Next make sure that /etc/authlib/userdb has read/write/execute access for the owner. If not chmod 700 /usr/authlib/userdb. Now create the user database that courier is going to look to for authentication:

$ makeuserdb

Again look to using Postfix/TLS for a tunneling the connection through a secure connection.

Pop-before-smtp Configuration

To successfully configure pop-before-smtp follow the straight forward instructions in contrib/README.QUICKINSTALL. When you are done and everything is working it is important to make sure that you have not some how made your mail server an open relay! This would cause your mail server to be flooded with traffic and be put on the open relay ban lists.

!!! IMPORTANT SECURITY NOTE!!! – I will not use the pop-before-smtp method because it involves sending a password over an insecure connection. Use Postfix through a SSL connection, Postfix/TLS, to accomplish this. Since your local machines are likely already allowed to use your Postfix as a mail relay the only time that you would be using this method is from a non-trusted server where you password can be sniffed. If you still choose to use this method make sure that you have changed the users courier password as described in Courier Authentication.

Additional Simple Procmail Recipes and Mailbox Config

If you are a member of any news letters/mailing lists you can have them automatically sorted to their own mailbox using Procmail. First create the new mail directory in your ~/Maildir directory. Lets call it .Sbayes and pretend that we are on the Spambayes mailing list. So exactly as before with the spam related mailboxes:

mkdir -p ~/Maildir/.Sbayes/{cur,new,tmp}

Now add this new rule to your ~/.procmailrc:

# spambayes-dev@python.org mailing list

:0

* ^X-BeenThere: spambayes@python.org

${MAILDIR}/.Sbayes/

In the .mutrc example above it shows an example of setting alt-5 as the macro to see your Spambayes emails.

A slightly different method can be used to send spam and ham for training. Adding the following lines will allow you to bounce spam messages to spam@localhost (localhost or your mail_domain.com) and ham messages to ham@localhost:

# Mail has been bounced to spam@localhost
:0
* ^Resent-To:.*spam
${MAILDIR}/.MissedSpam

# Mail has been bounced to ham@localhost
:0
* ^Resent-To:.*ham
${MAILDIR}/.MissedHam

If you want to bounce messages to the spam and ham addresses like this, you will need to make sure that the headers stay intact. Do not try using this or a similar technique for forwarding mails to an address because the email’s headers would not be intact and this is a large part of the spam classification. This method works because above we specified spam and ham as aliases for hypexr in the Postfix aliases file.

Conclusion

For Arch Linux users, Add saslauthd authdaemond courier-imap postfix fetchmail to DAEMONS in your /etc/rc.conf file and they will be started automatically when your machine boots. If you are running a web server have the httpd service listed after the services we are adding.

/etc/rc.conf

DAEMONS=(… saslauthd authdaemond postfix courier-imap fetchmail …)

Enjoy your new mail system and centralized mailbox. If you are handling a large number of users for you email system this is just the tip of the iceberg and you are going to mostly be interested in virtual users, security, and a system wide spam classifier. There are many good documents on various setups to accomplish this, see the links section. If you have any suggestions, questions, comments, or corrections send me an email or drop a note in the forum. If this document has been of any use to you, cheers. :)

Resources

Example Files

• .fetchmailrc
• .procmailrc
• .muttrc
• train_spambayes

Links

• Arch Linux
• Postfix
• Procmail
• Fetchmail
• SpamBayes
• Mutt
• SquirrelMail
• Courier-imap
• Courier-authlib

Virtual users and mailboxes using MySQL and setting up SSL for SquirrelMail

• Arch Wiki – Postfix Howto
• Gentoo – Virtual Mailhosting System with Postfix Guide

Other Useful links

• Arch Linux Postfix HowTo
• Jkx@Home’s Courier Authentication
• Postfix with TLS and SASL – geared towards FreeBSD
• Setting up Postfix + MySQL + Courier-IMAP + Mailman + vmail + Procmail + SpamAssassin – geared towards Fedora/Red Hat
• Postfix Howtos and FAQs
• The Mutt E-Mail Client Manual
• Timo’s procmail tips and recipes

Contributions

Thanks to Constantinos Laitsas for reworking the fetchmail rc file so that it will start as a daemon correctly

Thanks to Robert Schweikert for corrections to the train_spambayes script and the path to sb_filter.py

Thanks to Alex Satrapa for tips on example domain naming conventions

Thanks to Austin Duncan for tips on postfix configuration and mutts workings on other distros

 

Source:http://dev.mysql.com/doc/refman/4.1/en/mysqlcheck.html

 

The mysqlcheck client checks, repairs, optimizes, and analyzes tables. mysqlcheck is available as of MySQL 3.23.38.

mysqlcheck is similar in function to myisamchk, but works differently. The main operational difference is that mysqlcheck must be used when the mysqld server is running, whereas myisamchk should be used when it is not. The benefit of using mysqlcheck is that you do not have to stop the server to check or repair your tables.

mysqlcheck uses the SQL statements CHECK TABLE, REPAIR TABLE, ANALYZE TABLE, and OPTIMIZE TABLE in a convenient way for the user. It determines which statements to use for the operation you want to perform, and then sends the statements to the server to be executed. For details about which storage engines each statement works with, see the descriptions for those statements in Chapter 12, SQL Statement Syntax.

The MyISAM storage engine supports all four statements, so mysqlcheck can be used to perform all four operations on MyISAM tables. Other storage engines do not necessarily support all operations. In such cases, an error message is displayed. For example, if test.t is a MEMORY table, an attempt to check it produces this result:

shell> mysqlcheck test t
test.t
note     : The storage engine for the table doesn't support check

Caution

It is best to make a backup of a table before performing a table repair operation; under some circumstances the operation might cause data loss. Possible causes include but are not limited to filesystem errors.

There are three general ways to invoke mysqlcheck:

shell> mysqlcheck [options] db_name [tables]
shell> mysqlcheck [options] --databases db_name1 [db_name2 db_name3...]
shell> mysqlcheck [options] --all-databases

If you do not name any tables following db_name or if you use the --databases or --all-databases option, entire databases are checked.

mysqlcheck has a special feature compared to other client programs. The default behavior of checking tables (--check) can be changed by renaming the binary. If you want to have a tool that repairs tables by default, you should just make a copy of mysqlcheck named mysqlrepair, or make a symbolic link to mysqlcheck named mysqlrepair. If you invoke mysqlrepair, it repairs tables.

The following names can be used to change mysqlcheck default behavior:

mysqlrepair

The default option is --repair

mysqlanalyze

The default option is --analyze

mysqloptimize

The default option is --optimize

mysqlcheck supports the following options:

• --help, -?

  Display a help message and exit.

• --all-databases, -A

  Check all tables in all databases. This is the same as using the --databases option and naming all the databases on the command line.

• --all-in-1, -1

  Instead of issuing a statement for each table, execute a single statement for each database that names all the tables from that database to be processed.

• --analyze, -a

  Analyze the tables.

  MySQL Enterprise. For expert advice on optimizing tables, subscribe to the MySQL Enterprise Monitor. For more information, see http://www.mysql.com/products/enterprise/advisors.html.

• --auto-repair

  If a checked table is corrupted, automatically fix it. Any necessary repairs are done after all tables have been checked.

• --character-sets-dir=path

  The directory where character sets are installed. See Section 9.2, “The Character Set Used for Data and Sorting”.

• --check, -c

  Check the tables for errors. This is the default operation.

• --check-only-changed, -C

  Check only tables that have changed since the last check or that have not been closed properly.

• --compress

  Compress all information sent between the client and the server if both support compression.

• --databases, -B

  Process all tables in the named databases. Normally, mysqlcheck treats the first name argument on the command line as a database name and following names as table names. With this option, it treats all name arguments as database names.

• --debug[=debug_options], -# [debug_options]

  Write a debugging log. A typical debug_options string is often 'd:t:o,file_name'.

• --default-character-set=charset_name

  Use charset_name as the default character set. See Section 9.2, “The Character Set Used for Data and Sorting”.

• --extended, -e

  If you are using this option to check tables, it ensures that they are 100% consistent but takes a long time.

  If you are using this option to repair tables, it runs an extended repair that may not only take a long time to execute, but may produce a lot of garbage rows also!

• --fast, -F

  Check only tables that have not been closed properly.

• --force, -f

  Continue even if an SQL error occurs.

• --host=host_name, -h host_name

  Connect to the MySQL server on the given host.

• --medium-check, -m

  Do a check that is faster than an --extended operation. This finds only 99.99% of all errors, which should be good enough in most cases.

• --optimize, -o

  Optimize the tables.

• --password[=password], -p[password]

  The password to use when connecting to the server. If you use the short option form (-p), you cannot have a space between the option and the password. If you omit the password value following the --password or -p option on the command line, you are prompted for one.

  Specifying a password on the command line should be considered insecure. See Section 5.6.6, “Keeping Passwords Secure”.

• --pipe, -W

  On Windows, connect to the server via a named pipe. This option applies only for connections to a local server, and only if the server supports named-pipe connections.

• --port=port_num, -P port_num

  The TCP/IP port number to use for the connection.

• --protocol={TCP|SOCKET|PIPE|MEMORY}

  The connection protocol to use for connecting to the server. It is useful when the other connection parameters normally would cause a protocol to be used other than the one you want. For details on the allowable values, see Section 4.2.2, “Connecting to the MySQL Server”. This option was added in MySQL 4.1.

• --quick, -q

  If you are using this option to check tables, it prevents the check from scanning the rows to check for incorrect links. This is the fastest check method.

  If you are using this option to repair tables, it tries to repair only the index tree. This is the fastest repair method.

• --repair, -r

  Perform a repair that can fix almost anything except unique keys that are not unique.

• --silent, -s

  Silent mode. Print only error messages.

• --socket=path, -S path

  For connections to localhost, the Unix socket file to use, or, on Windows, the name of the named pipe to use.

• --ssl*

  Options that begin with --ssl specify whether to connect to the server via SSL and indicate where to find SSL keys and certificates. See Section 5.6.7.3, “SSL Command Options”.

• --tables

  Overrides the --databases or -B option. All name arguments following the option are regarded as table names.

• --use-frm

  For repair operations on MyISAM tables, get the table structure from the .frm file so that the table can be repaired even if the .MYI header is corrupted. This option was added in MySQL 4.0.5.

• --user=user_name, -u user_name

  The MySQL user name to use when connecting to the server.

• --verbose, -v

  Verbose mode. Print information about the various stages of program operation.

• --version, -V

  Display version information and exit.

 

Source: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html 

 

If there is database page corruption, you may want to dump your tables from the database with SELECT INTO ... OUTFILE. Usually, most of the data obtained in this way is intact. However, it is possible that the corruption might cause SELECT * FROM tbl_name statements or InnoDB background operations to crash or assert, or even cause InnoDB roll-forward recovery to crash. In such cases, you can use the innodb_force_recovery option to force the InnoDB storage engine to start up while preventing background operations from running, so that you are able to dump your tables. For example, you can add the following line to the [mysqld] section of your option file before restarting the server:

[mysqld]
innodb_force_recovery = 4

innodb_force_recovery is 0 by default (normal startup without forced recovery) The allowable non-zero values for innodb_force_recovery follow. A larger number includes all precautions of smaller numbers. If you are able to dump your tables with an option value of at most 4, then you are relatively safe that only some data on corrupt individual pages is lost. A value of 6 is more drastic because database pages are left in an obsolete state, which in turn may introduce more corruption into B-trees and other database structures.

• 1 (SRV_FORCE_IGNORE_CORRUPT)

  Let the server run even if it detects a corrupt page. Try to make SELECT * FROM tbl_name jump over corrupt index records and pages, which helps in dumping tables.

• 2 (SRV_FORCE_NO_BACKGROUND)

  Prevent the main thread from running. If a crash would occur during the purge operation, this recovery value prevents it.

• 3 (SRV_FORCE_NO_TRX_UNDO)

  Do not run transaction rollbacks after recovery.

• 4 (SRV_FORCE_NO_IBUF_MERGE)

  Prevent insert buffer merge operations. If they would cause a crash, do not do them. Do not calculate table statistics.

• 5 (SRV_FORCE_NO_UNDO_LOG_SCAN)

  Do not look at undo logs when starting the database: InnoDB treats even incomplete transactions as committed.

• 6 (SRV_FORCE_NO_LOG_REDO)

  Do not do the log roll-forward in connection with recovery.

The database must not otherwise be used with any non-zero value of innodb_force_recovery. As a safety measure, InnoDB prevents users from performing INSERT, UPDATE, or DELETE operations when innodb_force_recovery is greater than 0.

You can SELECT from tables to dump them, or DROP or CREATE tables even if forced recovery is used. If you know that a given table is causing a crash on rollback, you can drop it. You can also use this to stop a runaway rollback caused by a failing mass import or ALTER TABLE. You can kill the mysqld process and set innodb_force_recovery to 3 to bring the database up without the rollback, then DROP the table that is causing the runaway rollback.