Saturday, November 8, 2008

Proftpd & MySQL

Recently I re-worked the way that our FTP servers run at work. We use Proftpd in conjunction with MySQL for authentication. The setup required some reading from the Proftpd documentation, so I figured I could help someone out by showing our configuration found in the "proftpd.conf" file and what our database looks like.

First, the MySQL database has three tables: hosts, login_log, and users.


CREATE TABLE `users` (
`id` int(11) NOT NULL auto_increment,
`host_id` varchar(50) NOT NULL default '0',
`username` varchar(100) NOT NULL default '',
`passwd` varchar(100) NOT NULL COMMENT 'use PASSWORD() to encrypt',
`uid` int(5) NOT NULL default '5000',
`gid` int(5) NOT NULL default '5000',
`ftpdir` varchar(255) NOT NULL default '',
`ts_created` timestamp NOT NULL default '0000-00-00 00:00:00',
`ts_modified` timestamp NOT NULL default CURRENT_TIMESTAMP,
`deleted` smallint(1) NOT NULL default '0',
PRIMARY KEY (`id`),
UNIQUE KEY `username` (`username`)
) DEFAULT CHARSET=utf8;

CREATE TABLE `hosts` (
`id` int(11) NOT NULL auto_increment,
`name` varchar(50) NOT NULL default '',
PRIMARY KEY (`id`),
UNIQUE KEY `name` (`name`)
) DEFAULT CHARSET=utf8;

CREATE TABLE `login_log` (
`ip_address` varchar(15) NOT NULL default '',
`dns_name` varchar(150) default NULL,
`timestamp` timestamp NOT NULL default CURRENT_TIMESTAMP,
`username` varchar(100) NOT NULL default ''
) DEFAULT CHARSET=utf8;


The "hosts" table is created with the intent that multiple hosts use this database for FTP authentication. Insert the FQDN of all hosts into the "name" field.

The "users" table holds all the pertinent user information. The "password" field is encrypted using the MySQL "PASSWORD" function. "ftpdir" holds the directory to chroot the user into. "uid" and "gid" are the respective user and group id to assign to files that are created on/uploaded to the server. "host_id" is used to specify which server this user can log in to. It ties back to the "hosts" table.

The "login_log" table is used for logging whenever a user logs into the server. "ip_address" stores the IP address of the person who logged in. "dns_name" is the contents of a reverse DNS lookup on the IP address. "username" is the name of the user that logged in.

Now for the actual "proftpd.conf" configuration.


ServerName "FTP Server"
ServerType standalone
DefaultServer on
Port 21
Umask 007
MaxInstances 30
User nobody
Group nogroup

This group of directives sets up some of the basics for the FTP server. Proftp suports Virtual Hosts in manner like Apache.
"DefaultServer" tells proftpd to use all the settings here as the default server when a connection comes in.
"ServerType" tells the Proftp daemon to listen for connections instead of using inetd.
"Port" sets the port to use.
"Umask" sets the default permissions.
"MaxInstances" sets the maximum simultaneous connections.
"User" and "Group" set the user and group to run the server as.


SQLEngine on
SQLBackend mysql
SQLAuthTypes Backend
SQLConnectInfo mysql_user@mysql_host db_name mypassword
SQLAuthenticate users
SQLUserInfo custom:/select_user
SQLNamedQuery select_user SELECT "username, passwd, uid, gid, ftpdir, '/bin/bash' FROM users WHERE deleted=0 AND username='%U' AND host_id=host_id_for_this_host"


"SQLEngine" tells the server to turn on the abililty to use a SQL backend.
"SQLBackend" should be set to "mysql" to use MySQL as the authentication backend.
"SQLAuthTypes" tells the server what encryption is used for the password in the backend. The value "Backend" tells it to use the "PASSWORD" function for encryption.
"SQLConnectInfo" gives the connection info for the MySQL database.
"SQLAuthenticate" specifies the table to use for authentication.
"SQLNamedQuery" is used to specify a query to be used in some other location. For instance, this one is called "select_user". Looking at the "SQLUserInfo" line it can be seen that we use a custom query to authenticate users. The "select_user" query is used whenever a user attempts to log in. The columns MUST be returned in a very specific order. See the "SQLUserInfo" directive in the Proftp documentation for more information.


SQLDefaultUID 65534
SQLDefaultGID 65534
SQLMinUserUID 20
SQLMinUserGID 20
SQLLog PASS insert_login
SQLNamedQuery insert_login INSERT "'%a', '%h', NOW(), '%U'" login_log


"SQLDefaultUID" and "SQLDefaultGID" are the user id and group id to use if none are specified. These values will probably never be used.
"SQLMinUserUID" and "SQLMinUserGID" are the minimum values allowed in the authentication table for a user. If these minimums aren't met the user will not be able to log in.
"SQLLog" tells Proftp a query to run when the given FTP command is given. In this case, it logs anything that successfully authenticates to the FTP server.
"SQLNamedQuery" again specifies a query to run when called. See "SQLNamedQuery" in the Proftp documentation for more details. To know what the different variables mean in the query, see "LogFormat" in the Proftp documentation.


SystemLog /var/log/proftpd/proftpd.log
DefaultRoot ~
AllowOverwrite on
AllowRetrieveRestart on
AllowStoreRestart on
UseReverseDNS on
IdentLookups off
ListOptions "-a"
TimeoutIdle 3600


"SystemLog" tells where to log all of the Proftpd messages.
"DefaultRoot" jails the user into their ftp directory when set to "~".
"AllowOverwrite" allows files to be overwritten.
"AllowRetrieveRestart" allows a file to restart a download from where it last finished.
"AllowStoreRestart" allows a file to restart an upload from where it last finished.
"UseReverseDNS" tells Proftp to do reverse DNS lookups.
"IdentLookups" tells Proftp whether or not to do Ident lookups.
"ListOptions" tells Proftp what options to pass to an "ls" command by default.
"TimeoutIdle" is the timeout before automatically disconnecting an idle user.



Quite a write-up, but I hope it helps someone.

Sunday, October 26, 2008

SVN & MySQL

Just a week ago, my company migrated to SVN from CVS. I won't go into a discussion of why or what the differences are, but I thought it would be worth writing about our configuration of SVN. Before getting started, I should point you to the free, excellent SVN book found at http://svnbook.red-bean.com/. If you wanted to know about any of the features or configuration options available in Subversion, that's a great place to look.

There were three things that I wanted to accomplish with SVN:

1. Authentication using a MySQL database
2. The ability to manipulate access control at any level in the repository using the MySQL authentication (ie. Only allow access for a user to /client/trunk/project, not to the whole trunk)
3. The connection had to be HTTPS. I don't want our source code going over the internet in plain text.

First, let's cover the basics. Since I'm coming at this from the point of view of a Gentoo installation, I'll cover what's needed for that. The "dev-libs/apr-util" package must be compiled with the "mysql" USE flag enabled. As for apache, I just compile it with all modules enabled. This is easy to accomplish by setting APACHE2_MODULES="*" in /etc/make.conf. As for the MPM to use for Apache, we've been using the "prefork" one. I can't really give an argument for any given MPM since I don't know enough about performance of them to argue a point.

The second step is fairly obvious. You have to compile Apache. It should drag in the apr-util package as a dependency, but if it doesn't for some reason, make sure you compile that package as well. A simple "emerge apache" will do the trick.

The third and final step is to set up the configuration. I'll step through the configuration for our server to give a flavor of what it could look like. You can throw this stuff into a Virtual Host if that is what you want. It's what I did.


#I realize that the document root points to somewhere other than the SVN repositories. This is intentional. It's the only way I could get Subversion to work the way I wanted it to.
DocumentRoot /var/www/localhost/htdocs/

#Enable SSL. It's as simple as turning the engine on and providing the paths to the cert and the key
SSLEngine On
SSLCertificateFile /etc/apache2/ssl/domain.crt
SSLCertificateKeyFile /etc/apache2/ssl/domain.key

#This is the real meat and potatoes of the whole thing.

#This enables the mod_dbd MySQL driver
DBDriver mysql

#This causes mod_dbd to maintain the MySQL connection after every query instead of reconnecting for every MySQL call.
DBDPersist On

#These are the basics for a MySQL connection. Notice that the whole line is surrounded in quotes. See http://httpd.apache.org/docs/2.2/mod/mod_dbd.html for other parameters that can go on this line.
DBDParams "host=mydatabase.server.com user=myuser pass=mypassword dbname=mydatabasename"


# The location "/" causes SVN to interpret anything that follows the "/" as a repository path
<location />
DAV svn #Treat the location as a subversion repository
SVNParentPath /var/svn/ #This treats any directory under "/var/svn/" as the root of a repository
Allow from all #Allow access from anywhere
AuthName "Login" #The title of the authentication window
AuthType Basic #Not quite sure what this does, but it's necessary
AuthBasicAuthoritative Off #Tells Apache to try other authentication methods if this one doesn't work
AuthBasicProvider dbd #Tells Apache to try mod_dbd authentication first
</location>

#A regex that is used to determine the query to use for my current location
<locationmatch ^/client/trunk|tags|branches/project>
# This location requires an authenticated user. AuthDBDUserPWQuery determines how apache decides if the user is valid. The password in MySQL must be stored using the ENCRYPT() function. The supplied password must mach the stored ENCRYPT() password.

Require valid-user
AuthDBDUserPWQuery "SELECT password FROM users WHERE user=%s"
</locationmatch>


That's about it! Not the neatest write-up, but I hope this helps someone. I know I had a heck of a time trying to get it all working correctly. You can add as many locations as you want for different clients and/or subdirectories.

Edit: Something I found out later is that apr-util v1.3.x compiled with MySQL support and PHP compiled with MySQL support don't mix. It will cause Apache to segfault and puke all over the place. Use apr-util v1.2.x to avoid this problem (see https://bugs.gentoo.org/show_bug.cgi?id=240264 for details on the issue).

Thursday, October 9, 2008

Fun With Apache Fix

About a month ago I wrote about a weird problem that I was having with Apache in regards to mod_dbd and the MySQL driver. I never did write about what was causing the problem. After much troubleshooting, I discovered that Apache did not like the mod_dbd driver to be used with PHP if PHP had been compiled with MySQL support. I never could figure out why it was a problem, only that it was the cause of the segmentation fault.

Luckily, my company just recently started using VMware ESX server for virtualization, so I simply made another virtual machine to use for Apache with mod_dbd. SVN uses mod_dbd as a backend for authentication and it works beautifully. Come to think of it, I probably should write up an entry on how to configure SVN to work the way I did. It took me long enough to figure it out, that's for sure.

Sunday, September 28, 2008

101 Things in 1001 Days

My brother-in-law came up with the idea of make a list of 101 things to do in 1001 days. I didn't realize this was so popular, but a quick search on Google comes up with tons of lists. In any case, Aubrie, Matt, and I are all going to be doing this starting October 3, 2008. Prior to popular opinion, Aubrie did not give any of the ideas on here. Here's my list (or at least what I've thought up so far):

1. Get Lasik done on my eyes
2. Play through any video games purchased within the last two years that I haven't finished yet
  • Pirates
  • Portal
  • Far Cry
  • Zelda: Twilight Princess
  • Rayman Ravin' Rabbids 2
  • Unreal Tournament 3
  • Lost Planet: Extreme Conditions
3. Use Linux as my primary OS for 3 months. (in progress 10/20/2008)
4. Build a MythTV box for home.
5. Tear the tree out of our backyard (in progress)
6. Dig out the garden in the backyard
7. Fix the leaking valve in the sprinkler system
8. Clean the BBQ Grill
9. Dig out the flower beds in the front and back yard
10. Build shelves in our garage
11. Repaint the master bedroom
12. Repaint the computer room
13. Put cabinets in the laundry room
14. Put more cabinets in the kitchen
15. Put our TV on a wall mount
16. Get a surround sound system for the Family Room
17. Put new carpet in the bedrooms
18. Buy a new lawn mower
19. Cook dinner for Aubrie 5 nights in one week
20. Be in bed ready to sleep by 10pm every night for 2 weeks
21. Fix the tree in the front yard (done - 10/24/2008)
22. Put shelves up in the Family Room
23. Build shelves in the game closet
24. Figure out some way to finish the crawl space and do it
25. Hang stuff up in the computer room
26. Get Aubrie a new laptop
27. Build a web server for home
28. Vote


More forthcoming.....

Friday, September 26, 2008

Fun with Apache

Over the last week I've been busy getting a new MySQL server and Apache server built on our new VMware ESX server at work. Everything has gone smoothly with the exception of one thing. For the life of me, I cannot get Apache to cooperate. For our new SVN repository, I'm using the mod_dbd module for MySQL authentication, but every time I enable it using "DBDriver mysql" in httpd.conf I randomly start getting errors like the following on start and stop of Apache:

/etc/init.d/apache2: line 43: 16808 Segmentation fault ${APACHE2} ${APACHE2_OPTS} -t > /dev/null 2>&1

The weird thing about it is that it occurs randomly. Sometimes I can stop and restart without any problems at all. The second I remove the DBDriver line, everything works great. What makes it even stranger is that I have a 32 Bit virtual machine on my laptop that uses the module without any problems at all. Our server is running 64 Bit Gentoo, but I have gone to the point of making everything identical (with that one exception). Does anyone have any ideas what could cause this problem?

Saturday, September 13, 2008

Previously on Geek Speak

About 4 months ago I wrote an entry saying I would write up some decent Gentoo packages for a desktop system. While this won't be for a desktop system, I will bring up some good packages for a Gentoo server system. Some of them are obvious, but there are a few that took me a while to find.
  • logrotate - Rotates log files on the server. Comes with some pre-scripted files to rotate, but can also be customized for any log file.
  • metalog - System logger. Can be extended to dump different system events into separate log files.
  • screen - Emulates a terminal. Essentially, it can be used to start programs that need to maintain a session open. I use it all the time for compiling things in the background when I don't want to maintain an SSH connection to a server
  • genlop - Used to find information about currently emerging and already emerged packages. It can give an estimated time to finish an emerge and tell how long an emerge has been going.
  • gentoolkit - This is a must have for Gentoo. It provides various scripts for maintaining a Gentoo install. For instance, there is a program called "revdep-rebuild" which will verify that all emerged binaries link against the correct libraries on the system.
  • iptraf - This is a handy app that can be used for network monitoring.
  • tcpdump - This program can be used to monitor packets coming into a specified network interface.
  • logwatch - Can be used to email a summary of the logs on the system.
  • slocate - Indexes files on the filesystem to allow for quick searching
This is by no means an exhaustive list of helpful packages.

Monday, September 8, 2008

When CVS just won't cooperate...

Last week was pretty hectic for me. It all started Tuesday when our CVS server would freeze on a CVS add. The whole machine would lock up and the only thing that could be done was a hard reboot. Needless to say, this was a less than desirable method of development. It was really weird too. CVS commits and updates of files that already existed worked great, but the CVS add function would freeze the server almost every time. Seeing as how I couldn't find ANYTHING (I literally mean that) on my specific problem on google, I thought I would post my solution.

First I tried simply building a new Gentoo Linux server since that is my preferred Linux distro. I migrated all of the CVS data over to the new server along with all the permissions and ACLs that were assigned to the repositories so that it would be a simple migration process. Unfortunately, this didn't work. The new server exhibited the same behavior as the old one. This pointed me in the direction of three things possibly being wrong
1. The CVS repository was corrupt (I really hoped it wasn't that one)
2. The ACLs were somehow all screwed up
3. Some library or binary in Gentoo had a bug that was causing the problem

Since I was mostly interested in just getting things working again, I decided to go the route of 2 & 3. I spent part of a day building a Fedora Core 9 server and again copied the CVS repository to the new server, but this time I left out all the ACLs. It was somewhat of a pain, but I went through and recreated the ACL structure on the whole repository. The end result was a working stable CVS that I haven't had problems with since I migrated it to Fedora. I still don't know exactly what the problem was, but at least this worked for me. I hope it does for someone else as well.