Completely Fake Blog

Welcome to my blog

FreeBSD web stack

I was lately playing with ZFS on Linux. ZFS is getting increasingly easy on Linux, with Ubuntu Wily (15.10) having zfs packages in default repository. The last part is especially cool surprise. There are some bumps on CentOS 7 though, even though ZFS on Linux Project build packages regularly for this distribution. So far I understood, problem comes rather from CentOS 7 misshaps with modules building rather than from ZOL side.

But still, if you read my previous entry, you may have noticed, setting up ZFS on / is not a simple thing. I still did not get to a working state with Fedora. I think I'll try to play with CentOS in a VM soon, to get a feeling how far away are we from success - or how close. Since this is a must have for boot environments, which I am very fond of, I am watching progress on that front with real interest.

But there is another issue that irks me a bit, coming from illumos land. Observability. While there is a wealth of tracing frameworks for Linux out there, none so far is complete and finished. None so far is also as easy to learn and use as DTrace. When it comes to complete documentation, they all are far behind DTrace Guide also.

There is an operating system, which has pretty good driver coverage while being able to boot off ZFS, has beadm command implemented and a wealthy choice of system packages you'd expect from a server system: FreeBSD.

Honestly, I would prefer, myself, to go with a chosen illumos distribution. I think I may come up with similar post covering OpenIndiana or OmniOS soon. But for those of you, who don't feel comfortable entering lands of pretty exotic operating systems - here's very popular and well know one.

Installing packages, creating the pool

Let us consider a FreeBSD server with operating system installed on single SCSI drive with ZFS as a main filesystem. Let's also consider it have another SCSI drive for operating system pool mirroring and four SAS drives to host data on them.

Before we can start, lets prepare the freshly installed FreeBSD. My personal package manager of choice is pkg and text editor is vim. Issuing command:

root@~# pkg install vim
will do two things for me: install pkg command and follow with installing vim package.

There is one command I wish to show off. Since I've installed FreeBSD on ZFS filesystem, I can freely use beadm command:

root@:~ # pkg install beadm
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	beadm: 1.2.6

The process will require 31 KiB more space.
9 KiB to be downloaded.

Proceed with this action? [y/N]: y
Fetching beadm-1.2.6.txz: 100%    9 KiB   9.5kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing beadm-1.2.6...
[1/1] Extracting beadm-1.2.6: 100%

root@:~ # beadm create clean-install
Created successfully

root@:~ # beadm list
BE            Active Mountpoint  Space Created
default       NR     /          999.1M 2016-02-16 11:48
clean-install -      -          148.0K 2016-02-16 12:08
I hope one day it will be that easy for by favorite Linux distros too. :)

There is a very nice tool for FreeBSD to list block devices on the system: camcontrol. As a side note - I couldn't find a similar tool for Linux. Is there not a one or did I miss it?
The output for camcontrol on my FreeBSD is as follow:

root@:~ # camcontrol devlist
                at scbus0 target 0 lun 0 (pass0,da0)
                at scbus0 target 1 lun 0 (pass1,da1)
                  at scbus0 target 2 lun 0 (pass2,cd0)
                at scbus1 target 0 lun 0 (pass3,da2)
                at scbus1 target 1 lun 0 (pass4,da3)
                at scbus1 target 2 lun 0 (pass5,da4)
                at scbus1 target 3 lun 0 (pass6,da5)
From zpool status I know my system is installed on da0 and da1:
root@:~ # zpool status
  pool: zroot
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    da0p3   ONLINE       0     0     0
	    da1p3   ONLINE       0     0     0

errors: No known data errors
Thus drives we can use are da2 to da5. As a side note, da is a naming scheme for SCSI and SAS.

Lets create a storage pool, named datapool (my personal habit):

root@:~ # zpool create -f datapool mirror da2 da3 mirror da4 da5

root@:~ # zpool list
NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
datapool  3.97G    62K  3.97G         -     0%     0%  1.00x  ONLINE  -
zroot     13.9G   478M  13.4G         -     2%     3%  1.00x  ONLINE  -

root@:~ # zpool status datapool
  pool: datapool
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	datapool    ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da5     ONLINE       0     0     0

errors: No known data errors

I have create a separate filesystem, called data, within the datapool:

root@freebsdzfs:~ # zfs create datapool/data

root@freebsdzfs:~ # zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
datapool                  93.5K  3.84G    19K  /datapool
datapool/data               19K  3.84G    19K  /datapool/data

Installing PostgreSQL

I've installed postgresql 9.4 by running:

root@freebsdzfs:~ # pkg install postgresql94-server postgresql94-client postgresql94-contrib
[...]
echo 'postgresql_enable="YES"' >> /etc/rc.conf
I've omitted pkg's output for brevity. There are some lines on upgrading databases that I cut out. Last line makes sure postgres engine will start at system boot.

I've created a separate ZFS filesystem for the databases. This way I can set block size to be exact same as PostgreSQL's default one, 8 kilobytes, allowing for best performance.

root@freebsdzfs:~ # zfs create -o recordsize=8K datapool/postgres

root@freebsdzfs:~ # zfs get all datapool/postgres
NAME               PROPERTY              VALUE                  SOURCE
[...]
datapool/postgres  recordsize            8K                     local
[...]
Again, other properties have been omitted for brevity.

You will now need to initialize the directory for use by PostgreSQL following their Online Documentation:

root@freebsdzfs:~ # chown -R pgsql:pgsql /datapool/postgres

root@freebsdzfs:~ # su pgsql

$ initdb -D /datapool/postgres/data
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /datapool/postgres/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /datapool/postgres/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    postgres -D /datapool/postgres/data
or
    pg_ctl -D /datapool/postgres/data -l logfile start

$ 
To tell postgres to start using those, you need to add another line to /etc/rc.conf file:
echo 'postgresql_data="/datapool/postgres/data"' >> /etc/rc.conf
You should be now able to create databases and store them in /datapool/postgres directory, living on a mirrored ZFS storage.

Installing Apache HTTP Server

Following similar workflow as above, I've installed Apache web server and configured it to store its files within /datapool/www ZFS filesystem, that I have created specially for that purpose. I've also enabled compression for this dataset (lz4), since web pages should compress nicely, saving me both storage and physical read/write on hard drives.

root@freebsdzfs:~ # pkg install apache24-2.4.18
[...]

root@freebsdzfs:~ # echo 'apache24_enable="yes"' >> /etc/rc.conf

root@freebsdzfs:~ # zfs create -o compress=lz4 datapool/www

Do not forget to change the Document Root path in Apache's configuration file:
root@freebsdzfs:~ # sed -ie 's;/usr/local/www/apache24/data;/datapool/www;' /usr/local/etc/apache24/httpd.conf
Finally, check the configuration and start the service:
root@freebsdzfs:~ # service apache24 configtest

root@freebsdzfs:~ # service apache24 start

Tomcat

Just to have some additional fun, lets install a tomcat container to our FreeBSD server:

root@freebsdzfs:~ # pkg install tomcat8
[...]

Remember about this step:
This OpenJDK implementation requires fdescfs(5) mounted on /dev/fd and
procfs(5) mounted on /proc.

If you have not done it yet, please do the following:

	mount -t fdescfs fdesc /dev/fd
	mount -t procfs proc /proc

To make it permanent, you need the following lines in /etc/fstab:

	fdesc	/dev/fd		fdescfs		rw	0	0
	proc	/proc		procfs		rw	0	0
Start the tomcat server:
root@freebsdzfs:~ # cd /usr/local/apache-tomcat-8.0/bin/

root@freebsdzfs:/usr/local/apache-tomcat-8.0/bin # ./startup.sh 
Using CATALINA_BASE:   /usr/local/apache-tomcat-8.0
Using CATALINA_HOME:   /usr/local/apache-tomcat-8.0
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-8.0/temp
Using JRE_HOME:        /usr/local
Using CLASSPATH:       /usr/local/apache-tomcat-8.0/bin/bootstrap.jar:/usr/local/apache-tomcat-8.0/bin/tomcat-juli.jar
Tomcat started.

Tracing

So why would I go the great lengths to move from known Ubuntu/CentOS/Arch/Debian/Gentoo/... environment to FreeBSD one? Because of mentioned previously DTrace goodness. The Linux tracing frameworks, and there seem to be many of them, are difficult to set up, when compared with illumos/FreeBSD DTrace. It just works.

First thing you may wish to do is follow link to Prefetch Technologies and enable DTrace probes for you Apache.